When building an application that performs time-consuming, complex, or resource-intensive tasks, it can be frustrating to wait for these to complete within the front end application. Additionally, complex tasks in the front end can time-out. Redis Queue fixes this by pushing more sophisticated tasks to a worker for processing.

Using Redis with Redis Queue allows you to enter those complex tasks into a queue, so the Redis worker executes these tasks outside of your application’s HTTP server.

In this article, we will build an app that enqueues jobs with Redis queue, performs a function on those jobs and returns the result of the function.

Here is the link to the Github Repository with our code for this project.

What is Redis?

Redis is an open-source, in-memory database, cache, and message broker. Messages handled by Redis are essentially JSONs. Redis is ideal for quickly and easily working with specific data types in a temporary database, and provides very rapid access and delivery of queries.

For us, Redis offers two benefits. First, it pushes complex tasks to another space to be processed. Second, it is easier for the developer to handle complex actions by splitting the task into separate functions, the main application and the queue.

For this application, we’ll be using Redis to hold our queue of JSON messages. Redis can be a stand-alone database, accessible by many computers or systems. In our example, we will be using it as a local memory store to support our application.

What is Redis Queue?

Redis Queue is a python library for queueing jobs for background processing. Since many hosting services will time out on long HTTP requests, it is best to design APIs to close requests as quickly as possible. Redis Queue allows us to do this by pushing tasks to a queue and then to a worker for processing.

Using Redis in conjunction with Redis Queue allows you to request input from the user, return a validation response to the user, and queue up processes in the background. All without the front end user having to wait for those processes to complete. Processes could be anything from Machine Learning models, to duplicating an image to complex simulations.

Anything that takes longer to complete than you would like to have taken place on the front end of your application belongs in the Redis Queue.

Modular App Structure

We will cover using Docker to run the Redis database and initialize the Redis Queue worker. The worker will allow us to process the jobs in our application’s queue.

We’ll go over each of these files in more detail in the following sections.

We’ll discuss the following constituent files of our app,** main.py**,** functions.py**, and redis_resc.py.

There are two ways to initiate the docker container. We will be covering the standard process one step at a time in this article. However, the docker container can be created using docker-compose, which we will cover in a separate article.

#app-development #towards-data-science #software-development #python #editors-pick