Welcome to this series of blog posts which covers Redis Streams with the help of a practical example. We will use a sample application to make Twitter data available for search and query in real-time. RediSearch and Redis Streams serve as the backbone of this solution that consists of several co-operating components, each of which will we covered in a dedicated blog post.
_The code is available in this GitHub repo — _https://github.com/abhirockzz/redis-streams-in-action
This is the first part which explores the use case, motivations and provides a high level overview of the Redis features used in the solution.
High level architecture
The use case is relatively simple. As an end goal, we want to have a service that allows us to search for tweets based on some criteria such as hashtags, user, location etc. Of course, there are existing solutions for this. The one presented in this blog series is an example scenario and can be applied to similar problems.
Here is a summary of the individual components:
I have used a few Azure services (including Enterprise tier of Azure Cache for Redis that supports Redis modules such as RediSearch
, RedisTimeSeries
and Redis Bloom) to run different parts of the solution, but you can tweak the instructions a little bit and apply them as per your environment e.g. you can use use Docker to run everything locally! Although the individual services have been written in different programming languages, the same concepts apply (in terms of Redis Streams, RediSearch
, scalability etc.) and can be implemented in the language of your choice.
I had written a blog post RediSearch in Action that covered the same use case i.e. how to implement a set of applications for consuming tweets in real-time, index them in RediSearch and query them using a REST API. However, the solution presented here has been implemented with the help of Redis Streams along with other components in order to make the architecture scalable and fault-tolerant. In this specific example, it’s the ability to process large volume of tweets, but the same idea can be extended/applied to other use-cases which deal with high velocity data e.g. IoT, log analytics, etc. Such problems benefit from an architecture where you can horizontally scale out your applications to handle increasing data volumes. Typically, this involves introducing a Messaging system to act a buffer between producers and consumers. Since this is a common requirement and the problem space is well understood, there are lot of established solutions in the distributed messaging world ranging from JMS (Java Messaging Service), Apache Kafka, RabbitMQ, NATS, and of course Redis.
There is something unique about Redis though. From a messaging point of view, Redis is quite flexible since it provides multiple options to support different paradigms, hence serving a wide range of use cases. It’s features include Pub-Sub, Lists (worker queue approach) and Redis Streams. Since this blog series is focuses on Redis Streams, I will provide a quick over view of the other possibilities before moving on.
RPOPLPUSH
(and BRPOPLPUSH
)Introduced in Redis 5.0, Redis Streams provides the best of Pub/Sub and Lists along with reliable messaging, durability for messages replay, Consumer Groups for load balancing, Pending Entry List for monitoring and much more! What makes it different is that fact it is a append-only log
data structure. In a nutshell, producers can add records (using XADD
), consumers can subscribe to new items arriving to the stream (with XREAD
). It supports range queries (XRANGE
etc.) and thanks to consumer groups, a group of apps can distribute the processing load (XREADGROUP
) and its possible to monitor its state (XPENDING
etc).
Since the magic of Redis lies in its powerful command system, let’s go over some of the Redis Streams commands, grouped by functionality for easier understanding:
Add entries
There is only one way you can add messages to a Redis Stream. XADD appends the specified stream entry to the stream at the specified key. If the key does not exist, as a side effect of running this command the key is created with a stream value.
Read entries
-
and +
special IDs mean respectively the minimum ID possible and the maximum ID possible inside a stream)XRANGE
, but with the difference of returning the entries in reverse order (use the end ID first and the start ID later)XREAD
command with support for consumer groups. You can create groups of clients that consume different parts of the messages arriving in a given streamManage Redis Streams
PEL
) of a stream consumer group.XAUTOCLAIM
is equivalent to calling XPENDING
and then XCLAIM
Delete
For a detailed, I would highly recommend reading “Introduction to Redis Streams” (from the official Redis docs).
#cloud-computing #programming #azure #redis