If a long-running process is part of your application’s workflow, rather blocking the response, you should handle it in the background, outside the normal request/response flow.
Perhaps your web application requires users to submit a thumbnail (which will probably need to be re-sized) and confirm their email when they register. If your application processed the image and sent a confirmation email directly in the request handler, then the end user would have to wait unnecessarily for them both to finish processing before the page loads or updates. Instead, you’ll want to pass these processes off to a task queue and let a separate worker process deal with it, so you can immediately send a response back to the client. The end user can then do other things on the client-side while the processing takes place. Your application is also free to respond to requests from other users and clients.
To achieve this, we’ll walk you through the process of setting up and configuring Celery and Redis for handling long-running processes in a Flask app. We’ll also use Docker and Docker Compose to tie everything together. Finally, we’ll look at how to test the Celery tasks with unit and integration tests.
Objectives
By the end of this tutorial, you will be able to:
- Integrate Celery into a Flask app and create tasks.
- Containerize Flask, Celery, and Redis with Docker.
- Run processes in the background with a separate worker process.
- Save Celery logs to a file.
- Set up Flower to monitor and administer Celery jobs and workers.
- Test a Celery task with both unit and integration tests.
Background Tasks
Again, to improve user experience, long-running processes should be run outside the normal HTTP request/response flow, in a background process.
Examples:
- Sending confirmation emails
- Scraping and crawling
- Analyzing data
- Processing images
- Generating reports
As you’re building out an app, try to distinguish tasks that should run during the request/response lifecycle, like CRUD operations, from those that should run in the background.
#flask #python #redis #celery #docker
