Welcome to this series of blog posts that covers Redis Streams with the help of a practical example. We will use a sample application to make Twitter data available for search and query in real-time. RediSearch and Redis Streams serve as the backbone of this solution that consists of several co-operating components, each of which will be covered in a dedicated blog post.

• Part 1
• Part 2 — this blog
• Part 3 — coming soon
• Part 4 — coming soon

_The code is available in this GitHub repo — _https://github.com/abhirockzz/redis-streams-in-action

In this part, we look at the service which interacts with the Twitter Streaming API to consume tweets and move them on to the next part in the processing pipeline.

Our end goal is to be able to process tweets and make them available for search and queries via RediSearch. One could write a “do it all” service to consume tweets and directly store them in RediSearch. But, in order to scale to handle the volume of tweets, we need a service to act as a buffer and decouple our producer (the application we will focus in this blog) and consumer (covered in the next blog).

This is exactly what our first component facilitates — it consumes streaming Twitter data and forwards it to Redis Streams. We will deploy it to Azure Container Instances, validate its functionality and also walk-through how it works along with the code.

As you will see in the later parts of this series, this also provides a foundation for scale-out architecture.

All in all, this blog post is short and simple! It lays down the ground work for other parts of the solution which will be covered in subsequent posts. Please don’t worry about the fact that the service is written in Rust (in case you don’t know it already). The logic can be easily ported over to your favorite programming language.

Pre-requisites

Start by getting a free Azure account if you don’t have one already and install the Azure CLI as well.

We will be deploying the tweets consumer application to Azure Container Instances using regular Docker CLI commands. This capability is enabled by integration between Docker and Azure. Just ensure that you have Docker Desktop version 2.3.0.5 or later, for Windows, macOS, or install the Docker ACI Integration CLI for Linux.

To use the Twitter Streaming API, you will also need a Twitter developer account. If you don’t have one already, please follow these instructions on how to set it up.

Deploy the app to Azure Container Instances

To start off, setup the Enterprise tier of Azure Cache for Redis, using this quickstart. Once you finish this step, ensure that you save the following information: the Redis host name and Access key

The tweets consumer application is available as a Docker container — the easiest way is to simply re-use it. If you wish to build you own image, please use the Dockerfile available in the GitHub repo.

You will now see how convenient it is to deploy it to Azure Container Instances, that allows you to run Docker containers on-demand in a managed, serverless Azure environment.

First, create an Azure context to associate Docker with an Azure subscription and resource group so you can create and manage container instances.

#programming #database #cloud #redis