Gordon  Murray

Gordon Murray

1669018881

How to Configure Celery To Handle Long-running Tasks in A Django App

If a long-running process is part of your application's workflow, rather than blocking the response, you should handle it in the background, outside the normal request/response flow.

Perhaps your web application requires users to submit a thumbnail (which will probably need to be re-sized) and confirm their email when they register. If your application processed the image and sent a confirmation email directly in the request handler, then the end user would have to wait unnecessarily for them both to finish processing before the page loads or updates. Instead, you'll want to pass these processes off to a task queue and let a separate worker process deal with it, so you can immediately send a response back to the client. The end user can then do other things on the client-side while the processing takes place. Your application is also free to respond to requests from other users and clients.

To achieve this, we'll walk you through the process of setting up and configuring Celery and Redis for handling long-running processes in a Django app. We'll also use Docker and Docker Compose to tie everything together. Finally, we'll look at how to test the Celery tasks with unit and integration tests.

Django + Celery Series:

  1. Asynchronous Tasks with Django and Celery (this article!)
  2. Handling Periodic Tasks in Django with Celery and Docker
  3. Automatically Retrying Failed Celery Tasks
  4. Working with Celery and Database Transactions

Objectives

By the end of this tutorial you will be able to:

  1. Integrate Celery into a Django app and create tasks
  2. Containerize Django, Celery, and Redis with Docker
  3. Run processes in the background with a separate worker process
  4. Save Celery logs to a file
  5. Set up Flower to monitor and administer Celery jobs and workers
  6. Test a Celery task with both unit and integration tests

Background Tasks

Again, to improve user experience, long-running processes should be run outside the normal HTTP request/response flow, in a background process.

Examples:

  1. Running machine learning models
  2. Sending confirmation emails
  3. Scraping and crawling
  4. Analyzing data
  5. Processing images
  6. Generating reports

As you're building out an app, try to distinguish tasks that should run during the request/response lifecycle, like CRUD operations, from those that should run in the background.

Workflow

Our goal is to develop a Django application that works in conjunction with Celery to handle long-running processes outside the normal request/response cycle.

  1. The end user kicks off a new task via a POST request to the server-side.
  2. Within the view, a task is added to the queue and the task id is sent back to the client-side.
  3. Using AJAX, the client continues to poll the server to check the status of the task while the task itself is running in the background.

django and celery queue user flow

Project Setup

Clone down the base project from the django-celery repo, and then check out the v1 tag to the master branch:

$ git clone https://github.com/testdrivenio/django-celery --branch v1 --single-branch
$ cd django-celery
$ git checkout v1 -b master

Since we'll need to manage three processes in total (Django, Redis, worker), we'll use Docker to simplify our workflow by wiring them up so that they can all be run from one terminal window with a single command.

From the project root, create the images and spin up the Docker containers:

$ docker-compose up -d --build

Once the build is complete, navigate to http://localhost:1337:

django project

Make sure the tests pass as well:

$ docker-compose exec web python -m pytest

=============================== test session starts ===============================
platform linux -- Python 3.9.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
django: settings: core.settings (from ini)
rootdir: /usr/src/app, configfile: pytest.ini
plugins: django-4.4.0
collected 1 item

tests/test_tasks.py .                                                       [100%]

================================ 1 passed in 0.63s ================================

Take a quick look at the project structure before moving on:

├── .gitignore
├── LICENSE
├── README.md
├── docker-compose.yml
└── project
    ├── Dockerfile
    ├── core
    │   ├── __init__.py
    │   ├── asgi.py
    │   ├── settings.py
    │   ├── urls.py
    │   └── wsgi.py
    ├── entrypoint.sh
    ├── manage.py
    ├── pytest.ini
    ├── requirements.txt
    ├── static
    │   ├── bulma.min.css
    │   ├── jquery-3.4.1.min.js
    │   ├── main.css
    │   └── main.js
    ├── tasks
    │   ├── __init__.py
    │   ├── apps.py
    │   ├── migrations
    │   │   └── __init__.py
    │   ├── templates
    │   │   └── home.html
    │   └── views.py
    └── tests
        ├── __init__.py
        └── test_tasks.py

Want to learn how to build this project? Check out the Dockerizing Django with Postgres, Gunicorn, and Nginx blog post.

Trigger a Task

An event handler in project/static/main.js is set up that listens for a button click. On click, an AJAX POST request is sent to the server with the appropriate task type: 1, 2, or 3.

$('.button').on('click', function() {
  $.ajax({
    url: '/tasks/',
    data: { type: $(this).data('type') },
    method: 'POST',
  })
  .done((res) => {
    getStatus(res.task_id);
  })
  .fail((err) => {
    console.log(err);
  });
});

On the server-side, a view is already configured to handle the request in project/tasks/views.py:

@csrf_exempt
def run_task(request):
    if request.POST:
        task_type = request.POST.get("type")
        return JsonResponse({"task_type": task_type}, status=202)

Now comes the fun part: wiring up Celery!

Celery Setup

Start by adding both Celery and Redis to the project/requirements.txt file:

celery==4.4.7
Django==3.2.4
redis==3.5.3

pytest==6.2.4
pytest-django==4.4.0

Celery uses a message broker -- RabbitMQ, Redis, or AWS Simple Queue Service (SQS) -- to facilitate communication between the Celery worker and the web application. Messages are added to the broker, which are then processed by the worker(s). Once done, the results are added to the backend.

Redis will be used as both the broker and backend. Add both Redis and a Celery worker to the docker-compose.yml file like so:

version: '3.8'

services:
  web:
    build: ./project
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - ./project:/usr/src/app/
    ports:
      - 1337:8000
    environment:
      - DEBUG=1
      - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
      - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
      - CELERY_BROKER=redis://redis:6379/0
      - CELERY_BACKEND=redis://redis:6379/0
    depends_on:
      - redis

  celery:
    build: ./project
    command: celery worker --app=core --loglevel=info
    volumes:
      - ./project:/usr/src/app
    environment:
      - DEBUG=1
      - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
      - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
      - CELERY_BROKER=redis://redis:6379/0
      - CELERY_BACKEND=redis://redis:6379/0
    depends_on:
      - web
      - redis

  redis:
    image: redis:6-alpine

Take note of celery worker --app=core --loglevel=info:

  1. celery worker is used to start a Celery worker
  2. --app=core runs the core Celery Application (which we'll define shortly)
  3. --loglevel=info sets the logging level to info

Within the project's settings module, add the following at the bottom to tell Celery to use Redis as the broker and backend:

CELERY_BROKER_URL = os.environ.get("CELERY_BROKER", "redis://redis:6379/0")
CELERY_RESULT_BACKEND = os.environ.get("CELERY_BROKER", "redis://redis:6379/0")

Next, create a new file called sample_tasks.py in "project/tasks":

# project/tasks/sample_tasks.py

import time

from celery import shared_task


@shared_task
def create_task(task_type):
    time.sleep(int(task_type) * 10)
    return True

Here, using the shared_task decorator, we defined a new Celery task function called create_task.

Keep in mind that the task itself will not be executed from the Django process; it will be executed by the Celery worker.

Now, add a celery.py file to "project/core":

import os

from celery import Celery


os.environ.setdefault("DJANGO_SETTINGS_MODULE", "core.settings")
app = Celery("core")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()

What's happening here?

  1. First, we set a default value for the DJANGO_SETTINGS_MODULE environment variable so that Celery will know how to find the Django project.
  2. Next, we created a new Celery instance, with the name core, and assigned the value to a variable called app.
  3. We then loaded the celery configuration values from the settings object from django.conf. We used namespace="CELERY" to prevent clashes with other Django settings. All config settings for Celery must be prefixed with CELERY_, in other words.
  4. Finally, app.autodiscover_tasks() tells Celery to look for Celery tasks from applications defined in settings.INSTALLED_APPS.

Update project/core/__init__.py so that the Celery app is automatically imported when Django starts:

from .celery import app as celery_app


__all__ = ("celery_app",)

Trigger a Task

Update the view to kick off the task and respond with the id:

@csrf_exempt
def run_task(request):
    if request.POST:
        task_type = request.POST.get("type")
        task = create_task.delay(int(task_type))
        return JsonResponse({"task_id": task.id}, status=202)

Don't forget to import the task:

from tasks.sample_tasks import create_task

Build the images and spin up the new containers:

$ docker-compose up -d --build

To trigger a new task, run:

$ curl -F type=0 http://localhost:1337/tasks/

You should see something like:

{
  "task_id": "6f025ed9-09be-4cbb-be10-1dce919797de"
}

Task Status

Turn back to the event handler on the client-side:

$('.button').on('click', function() {
  $.ajax({
    url: '/tasks/',
    data: { type: $(this).data('type') },
    method: 'POST',
  })
  .done((res) => {
    getStatus(res.task_id);
  })
  .fail((err) => {
    console.log(err);
  });
});

When the response comes back from the original AJAX request, we then continue to call getStatus() with the task id every second:

function getStatus(taskID) {
  $.ajax({
    url: `/tasks/${taskID}/`,
    method: 'GET'
  })
  .done((res) => {
    const html = `
      <tr>
        <td>${res.task_id}</td>
        <td>${res.task_status}</td>
        <td>${res.task_result}</td>
      </tr>`
    $('#tasks').prepend(html);

    const taskStatus = res.task_status;

    if (taskStatus === 'SUCCESS' || taskStatus === 'FAILURE') return false;
    setTimeout(function() {
      getStatus(res.task_id);
    }, 1000);
  })
  .fail((err) => {
    console.log(err)
  });
}

If the response is successful, a new row is added to the table on the DOM.

Update the get_status view to return the status:

@csrf_exempt
def get_status(request, task_id):
    task_result = AsyncResult(task_id)
    result = {
        "task_id": task_id,
        "task_status": task_result.status,
        "task_result": task_result.result
    }
    return JsonResponse(result, status=200)

Import AsyncResult:

from celery.result import AsyncResult

Update the containers:

$ docker-compose up -d --build

Trigger a new task:

$ curl -F type=1 http://localhost:1337/tasks/

Then, grab the task_id from the response and call the updated endpoint to view the status:

$ curl http://localhost:1337/tasks/25278457-0957-4b0b-b1da-2600525f812f/

{
    "task_id": "25278457-0957-4b0b-b1da-2600525f812f",
    "task_status": "SUCCESS",
    "task_result": true
}

Test it out in the browser as well:

django, celery, docker

Celery Logs

Update the celery service, in docker-compose.yml, so that Celery logs are dumped to a log file:

celery:
  build: ./project
  command: celery worker --app=core --loglevel=info --logfile=logs/celery.log
  volumes:
    - ./project:/usr/src/app
  environment:
    - DEBUG=1
    - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
    - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
    - CELERY_BROKER=redis://redis:6379/0
    - CELERY_BACKEND=redis://redis:6379/0
  depends_on:
    - web
    - redis

Add a new directory to "project" called "logs. Then, add a new file called celery.log to that newly created directory.

Update:

$ docker-compose up -d --build

You should see the log file fill up locally since we set up a volume:

[2021-06-20 00:00:31,333: INFO/MainProcess] Connected to redis://redis:6379/0
[2021-06-20 00:00:31,343: INFO/MainProcess] mingle: searching for neighbors
[2021-06-20 00:00:32,375: INFO/MainProcess] mingle: all alone
[2021-06-20 00:00:32,392: WARNING/MainProcess]
    /usr/local/lib/python3.9/site-packages/celery/fixups/django.py:205:
    UserWarning: Using settings.DEBUG leads to a memory
    leak, never use this setting in production environments!
    warnings.warn('''Using settings.DEBUG leads to a memory
[2021-06-20 00:00:32,394: INFO/MainProcess] celery@eb0f99f2fad9 ready.

[2021-06-20 00:00:44,488: INFO/MainProcess]
    Received task: tasks.sample_tasks.create_task[570bb712-bf2c-4406-b205-07f2e83e113d]
[2021-06-20 00:00:54,479: INFO/ForkPoolWorker-7]
    Task tasks.sample_tasks.create_task[570bb712-bf2c-4406-b205-07f2e83e113d]
    succeeded in 10.023200300012832s: True

Flower Dashboard

Flower is a lightweight, real-time, web-based monitoring tool for Celery. You can monitor currently running tasks, increase or decrease the worker pool, view graphs and a number of statistics, to name a few.

Add it to requirements.txt:

celery==4.4.7
Django==3.2.4
flower==0.9.7
redis==3.5.3

pytest==6.2.4
pytest-django==4.4.0

Then, add a new service to docker-compose.yml:

dashboard:
  build: ./project
  command:  flower -A core --port=5555 --broker=redis://redis:6379/0
  ports:
    - 5555:5555
  environment:
    - DEBUG=1
    - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
    - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
    - CELERY_BROKER=redis://redis:6379/0
    - CELERY_BACKEND=redis://redis:6379/0
  depends_on:
    - web
    - redis
    - celery

Test it out:

$ docker-compose up -d --build

Navigate to http://localhost:5555 to view the dashboard. You should see one worker ready to go:

flower dashboard

Kick off a few more tasks to fully test the dashboard:

flower dashboard

Try adding a few more workers to see how that affects things:

$ docker-compose up -d --build --scale celery=3

Tests

Let's start with the most basic test:

def test_task():
    assert sample_tasks.create_task.run(1)
    assert sample_tasks.create_task.run(2)
    assert sample_tasks.create_task.run(3)

Add the above test case to project/tests/test_tasks.py, and then add the following import:

from tasks import sample_tasks

Run that test individually:

$ docker-compose exec web python -m pytest -k "test_task and not test_home"

It should take about one minute to run:

=============================== test session starts ===============================
platform linux -- Python 3.9.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
django: settings: core.settings (from ini)
rootdir: /usr/src/app, configfile: pytest.ini
plugins: django-4.4.0, celery-4.4.7
collected 2 items / 1 deselected / 1 selected

tests/test_tasks.py .                                                       [100%]

=================== 1 passed, 1 deselected in 60.69s (0:01:00) ====================

It's worth noting that in the above asserts, we used the .run method (rather than .delay) to run the task directly without a Celery worker.

Want to mock the .run method to speed things up?

@patch("tasks.sample_tasks.create_task.run")
def test_mock_task(mock_run):
    assert sample_tasks.create_task.run(1)
    sample_tasks.create_task.run.assert_called_once_with(1)

    assert sample_tasks.create_task.run(2)
    assert sample_tasks.create_task.run.call_count == 2

    assert sample_tasks.create_task.run(3)
    assert sample_tasks.create_task.run.call_count == 3

Import:

from unittest.mock import patch

Test:

$ docker-compose exec web python -m pytest -k "test_mock_task"

=============================== test session starts ===============================
platform linux -- Python 3.9.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
django: settings: core.settings (from ini)
rootdir: /usr/src/app, configfile: pytest.ini
plugins: django-4.4.0, celery-4.4.7
collected 3 items / 2 deselected / 1 selected

tests/test_tasks.py .                                                       [100%]

========================= 1 passed, 2 deselected in 0.64s =========================

Much quicker!

How about a full integration test?

def test_task_status(client):
    response = client.post(reverse("run_task"), {"type": 0})
    content = json.loads(response.content)
    task_id = content["task_id"]
    assert response.status_code == 202
    assert task_id

    response = client.get(reverse("get_status", args=[task_id]))
    content = json.loads(response.content)
    assert content == {"task_id": task_id, "task_status": "PENDING", "task_result": None}
    assert response.status_code == 200

    while content["task_status"] == "PENDING":
        response = client.get(reverse("get_status", args=[task_id]))
        content = json.loads(response.content)
    assert content == {"task_id": task_id, "task_status": "SUCCESS", "task_result": True}

Keep in mind that this test uses the same broker and backend used in development. You may want to instantiate a new Celery app for testing:

app = celery.Celery('tests', broker=CELERY_TEST_BROKER, backend=CELERY_TEST_BACKEND)

Add the import:

import json

Ensure the test passes.

Conclusion

This has been a basic guide on how to configure Celery to run long-running tasks in a Django app. You should let the queue handle any processes that could block or slow down the user-facing code.

Celery can also be used to execute repeatable tasks and break up complex, resource-intensive tasks so that the computational workload can be distributed across a number of machines to reduce (1) the time to completion and (2) the load on the machine handling client requests.

Finally, if you're curious about how to use WebSockets (via Django Channels) to check the status of a Celery task, instead of using AJAX polling, check out the The Definitive Guide to Celery and Django course.

Grab the code from the repo.

Django + Celery Series:

  1. Asynchronous Tasks with Django and Celery (this article!)
  2. Handling Periodic Tasks in Django with Celery and Docker
  3. Automatically Retrying Failed Celery Tasks
  4. Working with Celery and Database Transactions

Original article source at: https://testdriven.io/

#django #configure #celery 

How to Configure Celery To Handle Long-running Tasks in A Django App

How to Configure Celery To Handle Long-running Tasks in A FastAPI App

If a long-running process is part of your application's workflow, rather than blocking the response, you should handle it in the background, outside the normal request/response flow.

Perhaps your web application requires users to submit a thumbnail (which will probably need to be re-sized) and confirm their email when they register. If your application processed the image and sent a confirmation email directly in the request handler, then the end user would have to wait unnecessarily for them both to finish processing before the page loads or updates. Instead, you'll want to pass these processes off to a task queue and let a separate worker process deal with it, so you can immediately send a response back to the client. The end user can then do other things on the client-side while the processing takes place. Your application is also free to respond to requests from other users and clients.

To achieve this, we'll walk you through the process of setting up and configuring Celery and Redis for handling long-running processes in a FastAPI app. We'll also use Docker and Docker Compose to tie everything together. Finally, we'll look at how to test the Celery tasks with unit and integration tests.

Objectives

By the end of this tutorial, you will be able to:

  1. Integrate Celery into a FastAPI app and create tasks.
  2. Containerize FastAPI, Celery, and Redis with Docker.
  3. Run processes in the background with a separate worker process.
  4. Save Celery logs to a file.
  5. Set up Flower to monitor and administer Celery jobs and workers.
  6. Test a Celery task with both unit and integration tests.

Background Tasks

Again, to improve user experience, long-running processes should be run outside the normal HTTP request/response flow, in a background process.

Examples:

  1. Running machine learning models
  2. Sending confirmation emails
  3. Scraping and crawling
  4. Analyzing data
  5. Processing images
  6. Generating reports

As you're building out an app, try to distinguish tasks that should run during the request/response lifecycle, like CRUD operations, from those that should run in the background.

It's worth noting that you can leverage FastAPI's BackgroundTasks class, which comes directly from Starlette, to run tasks in the background.

For example:

from fastapi import BackgroundTasks


def send_email(email, message):
    pass


@app.get("/")
async def ping(background_tasks: BackgroundTasks):
    background_tasks.add_task(send_email, "email@address.com", "Hi!")
    return {"message": "pong!"}

So, when should you use Celery instead of BackgroundTasks?

  1. CPU intensive tasks: Celery should be used for tasks that perform heavy background computations since BackgroundTasks runs in the same event loop that serves your app's requests.
  2. Task queue: If you require a task queue to manage the tasks and workers, you should use Celery. Often you'll want to retrieve the status of a job and then perform some action based on the status -- i.e., send an error email, kick off a different background task, or retry the task. Celery manages all this for you.

Workflow

Our goal is to develop a FastAPI application that works in conjunction with Celery to handle long-running processes outside the normal request/response cycle.

  1. The end user kicks off a new task via a POST request to the server-side.
  2. Within the route handler, a task is added to the queue and the task ID is sent back to the client-side.
  3. Using AJAX, the client continues to poll the server to check the status of the task while the task itself is running in the background.

fastapi and celery queue user flow

Project Setup

Clone down the base project from the fastapi-celery repo, and then check out the v1 tag to the master branch:

$ git clone https://github.com/testdrivenio/fastapi-celery --branch v1 --single-branch
$ cd fastapi-celery
$ git checkout v1 -b master

Since we'll need to manage three processes in total (FastAPI, Redis, Celery worker), we'll use Docker to simplify our workflow by wiring them up so that they can all be run from one terminal window with a single command.

From the project root, create the images and spin up the Docker containers:

$ docker-compose up -d --build

Once the build is complete, navigate to http://localhost:8004:

fastapi project

Make sure the tests pass as well:

$ docker-compose exec web python -m pytest

================================== test session starts ===================================
platform linux -- Python 3.9.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /usr/src/app
collected 1 item

tests/test_tasks.py .                                                               [100%]

=================================== 1 passed in 0.06s ====================================

Take a quick look at the project structure before moving on:

├── .gitignore
├── LICENSE
├── README.md
├── docker-compose.yml
└── project
    ├── Dockerfile
    ├── main.py
    ├── requirements.txt
    ├── static
    │   ├── main.css
    │   └── main.js
    ├── templates
    │   ├── _base.html
    │   ├── footer.html
    │   └── home.html
    └── tests
        ├── __init__.py
        ├── conftest.py
        └── test_tasks.py

Trigger a Task

An onclick event handler in project/templates/home.html is set up that listens for a button click:

<div class="btn-group" role="group" aria-label="Basic example">
  <button type="button" class="btn btn-primary" onclick="handleClick(1)">Short</a>
  <button type="button" class="btn btn-primary" onclick="handleClick(2)">Medium</a>
  <button type="button" class="btn btn-primary" onclick="handleClick(3)">Long</a>
</div>

onclick calls handleClick found in project/static/main.js, which sends an AJAX POST request to the server with the appropriate task type: 1, 2, or 3.

function handleClick(type) {
  fetch('/tasks', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({ type: type }),
  })
  .then(response => response.json())
  .then(res => getStatus(res.data.task_id));
}

On the server-side, a route is already configured to handle the request in project/main.py:

@app.post("/tasks", status_code=201)
def run_task(payload = Body(...)):
    task_type = payload["type"]
    return JSONResponse(task_type)

Now comes the fun part -- wiring up Celery!

Celery Setup

Start by adding both Celery and Redis to the requirements.txt file:

aiofiles==0.6.0
celery==4.4.7
fastapi==0.64.0
Jinja2==2.11.3
pytest==6.2.4
redis==3.5.3
requests==2.25.1
uvicorn==0.13.4

This tutorial uses Celery v4.4.7 since Flower does not support Celery 5.

Celery uses a message broker -- RabbitMQ, Redis, or AWS Simple Queue Service (SQS) -- to facilitate communication between the Celery worker and the web application. Messages are added to the broker, which are then processed by the worker(s). Once done, the results are added to the backend.

Redis will be used as both the broker and backend. Add both Redis and a Celery worker to the docker-compose.yml file like so:

version: '3.8'

services:

  web:
    build: ./project
    ports:
      - 8004:8000
    command: uvicorn main:app --host 0.0.0.0 --reload
    volumes:
      - ./project:/usr/src/app
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/0
    depends_on:
      - redis

  worker:
    build: ./project
    command: celery worker --app=worker.celery --loglevel=info
    volumes:
      - ./project:/usr/src/app
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/0
    depends_on:
      - web
      - redis

  redis:
    image: redis:6-alpine

Take note of celery worker --app=worker.celery --loglevel=info:

  1. celery worker is used to start a Celery worker
  2. --app=worker.celery runs the Celery Application (which we'll define shortly)
  3. --loglevel=info sets the logging level to info

Next, create a new file called worker.py in "project":

import os
import time

from celery import Celery


celery = Celery(__name__)
celery.conf.broker_url = os.environ.get("CELERY_BROKER_URL", "redis://localhost:6379")
celery.conf.result_backend = os.environ.get("CELERY_RESULT_BACKEND", "redis://localhost:6379")


@celery.task(name="create_task")
def create_task(task_type):
    time.sleep(int(task_type) * 10)
    return True

Here, we created a new Celery instance, and using the task decorator, we defined a new Celery task function called create_task.

Keep in mind that the task itself will be executed by the Celery worker.

Trigger a Task

Update the route handler to kick off the task and respond with the task ID:

@app.post("/tasks", status_code=201)
def run_task(payload = Body(...)):
    task_type = payload["type"]
    task = create_task.delay(int(task_type))
    return JSONResponse({"task_id": task.id})

Don't forget to import the task:

from worker import create_task

Build the images and spin up the new containers:

$ docker-compose up -d --build

To trigger a new task, run:

$ curl http://localhost:8004/tasks -H "Content-Type: application/json" --data '{"type": 0}'

You should see something like:

{
  "task_id": "14049663-6257-4a1f-81e5-563c714e90af"
}

Task Status

Turn back to the handleClick function on the client-side:

function handleClick(type) {
  fetch('/tasks', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({ type: type }),
  })
  .then(response => response.json())
  .then(res => getStatus(res.data.task_id));
}

When the response comes back from the original AJAX request, we then continue to call getStatus() with the task ID every second:

function getStatus(taskID) {
  fetch(`/tasks/${taskID}`, {
    method: 'GET',
    headers: {
      'Content-Type': 'application/json'
    },
  })
  .then(response => response.json())
  .then(res => {
    const html = `
      <tr>
        <td>${taskID}</td>
        <td>${res.data.task_status}</td>
        <td>${res.data.task_result}</td>
      </tr>`;
    document.getElementById('tasks').prepend(html);
    const newRow = document.getElementById('table').insertRow();
    newRow.innerHTML = html;
    const taskStatus = res.data.task_status;
    if (taskStatus === 'finished' || taskStatus === 'failed') return false;
    setTimeout(function() {
      getStatus(res.data.task_id);
    }, 1000);
  })
  .catch(err => console.log(err));
}

If the response is successful, a new row is added to the table on the DOM.

Update the get_status route handler to return the status:

@app.get("/tasks/{task_id}")
def get_status(task_id):
    task_result = AsyncResult(task_id)
    result = {
        "task_id": task_id,
        "task_status": task_result.status,
        "task_result": task_result.result
    }
    return JSONResponse(result)

Import AsyncResult:

from celery.result import AsyncResult

Update the containers:

$ docker-compose up -d --build

Trigger a new task:

$ curl http://localhost:8004/tasks -H "Content-Type: application/json" --data '{"type": 1}'

Then, grab the task_id from the response and call the updated endpoint to view the status:

$ curl http://localhost:8004/tasks/f3ae36f1-58b8-4c2b-bf5b-739c80e9d7ff

{
  "task_id": "455234e0-f0ea-4a39-bbe9-e3947e248503",
  "task_result": true,
  "task_status": "SUCCESS"
}

Test it out in the browser as well:

fastapi, celery, docker

Celery Logs

Update the worker service, in docker-compose.yml, so that Celery logs are dumped to a log file:

worker:
  build: ./project
  command: celery worker --app=worker.celery --loglevel=info --logfile=logs/celery.log
  volumes:
    - ./project:/usr/src/app
  environment:
    - CELERY_BROKER_URL=redis://redis:6379/0
    - CELERY_RESULT_BACKEND=redis://redis:6379/0
  depends_on:
    - web
    - redis

Add a new directory to "project" called "logs. Then, add a new file called celery.log to that newly created directory.

Update:

$ docker-compose up -d --build

You should see the log file fill up locally since we set up a volume:

[2021-05-08 15:32:24,407: INFO/MainProcess] Connected to redis://redis:6379/0
[2021-05-08 15:32:24,415: INFO/MainProcess] mingle: searching for neighbors
[2021-05-08 15:32:25,434: INFO/MainProcess] mingle: all alone
[2021-05-08 15:32:25,448: INFO/MainProcess] celery@365a3b836a91 ready.
[2021-05-08 15:32:29,834: INFO/MainProcess]
    Received task: create_task[013df48c-4548-4a2b-9b22-7267da215361]
[2021-05-08 15:32:39,825: INFO/ForkPoolWorker-7]
    Task create_task[013df48c-4548-4a2b-9b22-7267da215361]
    succeeded in 10.02114040000015s: True

Flower Dashboard

Flower is a lightweight, real-time, web-based monitoring tool for Celery. You can monitor currently running tasks, increase or decrease the worker pool, view graphs and a number of statistics, to name a few.

Add it to requirements.txt:

aiofiles==0.6.0
celery==4.4.7
fastapi==0.64.0
flower==0.9.7
Jinja2==2.11.3
pytest==6.2.4
redis==3.5.3
requests==2.25.1
uvicorn==0.13.4

Then, add a new service to docker-compose.yml:

dashboard:
  build: ./project
  command:  flower --app=worker.celery --port=5555 --broker=redis://redis:6379/0
  ports:
    - 5556:5555
  environment:
    - CELERY_BROKER_URL=redis://redis:6379/0
    - CELERY_RESULT_BACKEND=redis://redis:6379/0
  depends_on:
    - web
    - redis
    - worker

Test it out:

$ docker-compose up -d --build

Navigate to http://localhost:5556 to view the dashboard. You should see one worker ready to go:

flower dashboard

Kick off a few more tasks to fully test the dashboard:

flower dashboard

Try adding a few more workers to see how that affects things:

$ docker-compose up -d --build --scale worker=3

Tests

Let's start with the most basic test:

def test_task():
    assert create_task.run(1)
    assert create_task.run(2)
    assert create_task.run(3)

Add the above test case to project/tests/test_tasks.py, and then add the following import:

from worker import create_task

Run that test individually:

$ docker-compose exec web python -m pytest -k "test_task and not test_home"

It should take about one minute to run:

================================== test session starts ===================================
platform linux -- Python 3.9.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /usr/src/app
plugins: celery-4.4.7
collected 2 items / 1 deselected / 1 selected

tests/test_tasks.py .                                                               [100%]

====================== 1 passed, 1 deselected in 60.05s (0:01:00)  ========================

It's worth noting that in the above asserts, we used the .run method (rather than .delay) to run the task directly without a Celery worker.

Want to mock the .run method to speed things up?

@patch("worker.create_task.run")
def test_mock_task(mock_run):
    assert create_task.run(1)
    create_task.run.assert_called_once_with(1)

    assert create_task.run(2)
    assert create_task.run.call_count == 2

    assert create_task.run(3)
    assert create_task.run.call_count == 3

Import:

from unittest.mock import patch, call

Test:

$ docker-compose exec web python -m pytest -k "test_mock_task"

================================== test session starts ===================================
platform linux -- Python 3.9.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /usr/src/app
plugins: celery-4.4.7
collected 3 items / 2 deselected / 1 selected

tests/test_tasks.py .                                                               [100%]

============================ 1 passed, 2 deselected in 0.13s =============================

Much quicker!

How about a full integration test?

def test_task_status(test_app):
    response = test_app.post(
        "/tasks",
        data=json.dumps({"type": 1})
    )
    content = response.json()
    task_id = content["task_id"]
    assert task_id

    response = test_app.get(f"tasks/{task_id}")
    content = response.json()
    assert content == {"task_id": task_id, "task_status": "PENDING", "task_result": None}
    assert response.status_code == 200

    while content["task_status"] == "PENDING":
        response = test_app.get(f"tasks/{task_id}")
        content = response.json()
    assert content == {"task_id": task_id, "task_status": "SUCCESS", "task_result": True}

Keep in mind that this test uses the same broker and backend used in development. You may want to instantiate a new Celery app for testing.

Add the import:

import json

Ensure the test passes.

Conclusion

This has been a basic guide on how to configure Celery to run long-running tasks in a FastAPI app. You should let the queue handle any processes that could block or slow down the user-facing code.

Celery can also be used to execute repeatable tasks and break up complex, resource-intensive tasks so that the computational workload can be distributed across a number of machines to reduce (1) the time to completion and (2) the load on the machine handling client requests.

Grab the code from the repo.

Original article source at: https://testdriven.io/

#docker #fastapi #celery 

How to Configure Celery To Handle Long-running Tasks in A FastAPI App
Nigel  Uys

Nigel Uys

1669003860

How to Manage Periodic Tasks with Django, Celery, and Docker

As you build and scale a Django app you'll inevitably need to run certain tasks periodically and automatically in the background.

Some examples:

  • Generating periodic reports
  • Clearing cache
  • Sending batch e-mail notifications
  • Running nightly maintenance jobs

This is one of the few pieces of functionality required for building and scaling a web app that isn't part of the Django core. Fortunately, Celery provides a powerful solution, which is fairly easy to implement called Celery Beat.

In the following article, we'll show you how to set up Django, Celery, and Redis with Docker in order to run a custom Django Admin command periodically with Celery Beat.

Django + Celery Series:

  1. Asynchronous Tasks with Django and Celery
  2. Handling Periodic Tasks in Django with Celery and Docker (this article!)
  3. Automatically Retrying Failed Celery Tasks
  4. Working with Celery and Database Transactions

Objectives

By the end of this tutorial, you should be able to:

  1. Containerize Django, Celery, and Redis with Docker
  2. Integrate Celery into a Django app and create tasks
  3. Write a custom Django Admin command
  4. Schedule a custom Django Admin command to run periodically via Celery Beat

Project Setup

Clone down the base project from the django-celery-beat repo, and then check out the base branch:

$ git clone https://github.com/testdrivenio/django-celery-beat --branch base --single-branch
$ cd django-celery-beat

Since we'll need to manage four processes in total (Django, Redis, worker, and scheduler), we'll use Docker to simplify our workflow by wiring them up so that they can all be run from one terminal window with a single command.

From the project root, create the images and spin up the Docker containers:

$ docker-compose up -d --build

Next, apply the migrations:

$ docker-compose exec web python manage.py migrate

Once the build is complete, navigate to http://localhost:1337 to ensure the app works as expected. You should see the following text:

Orders

No orders found!

Take a quick look at the project structure before moving on:

├── .gitignore
├── docker-compose.yml
└── project
    ├── Dockerfile
    ├── core
    │   ├── __init__.py
    │   ├── asgi.py
    │   ├── settings.py
    │   ├── urls.py
    │   └── wsgi.py
    ├── entrypoint.sh
    ├── manage.py
    ├── orders
    │   ├── __init__.py
    │   ├── admin.py
    │   ├── apps.py
    │   ├── migrations
    │   │   ├── 0001_initial.py
    │   │   └── __init__.py
    │   ├── models.py
    │   ├── tests.py
    │   ├── urls.py
    │   └── views.py
    ├── products.json
    ├── requirements.txt
    └── templates
        └── orders
            └── order_list.html

Want to learn how to build this project? Check out the Dockerizing Django with Postgres, Gunicorn, and Nginx blog post.

Celery and Redis

Now, we need to add containers for Celery, Celery Beat, and Redis.

We'll begin by adding the dependencies to the requirements.txt file:

Django==3.2.4
celery==5.1.2
redis==3.5.3

Next, add the following to the end of the docker-compose.yml file:

redis:
  image: redis:alpine
celery:
  build: ./project
  command: celery -A core worker -l info
  volumes:
    - ./project/:/usr/src/app/
  environment:
    - DEBUG=1
    - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
    - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
  depends_on:
    - redis
celery-beat:
  build: ./project
  command: celery -A core beat -l info
  volumes:
    - ./project/:/usr/src/app/
  environment:
    - DEBUG=1
    - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
    - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
  depends_on:
    - redis

We also need to update the web service's depends_on section:

web:
  build: ./project
  command: python manage.py runserver 0.0.0.0:8000
  volumes:
    - ./project/:/usr/src/app/
  ports:
    - 1337:8000
  environment:
    - DEBUG=1
    - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
    - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
  depends_on:
    - redis # NEW

The full docker-compose.yml file should now look like this:

version: '3.8'

services:
  web:
    build: ./project
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - ./project/:/usr/src/app/
    ports:
      - 1337:8000
    environment:
      - DEBUG=1
      - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
      - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
    depends_on:
      - redis
  redis:
    image: redis:alpine
  celery:
    build: ./project
    command: celery -A core worker -l info
    volumes:
      - ./project/:/usr/src/app/
    environment:
      - DEBUG=1
      - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
      - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
    depends_on:
      - redis
  celery-beat:
    build: ./project
    command: celery -A core beat -l info
    volumes:
      - ./project/:/usr/src/app/
    environment:
      - DEBUG=1
      - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
      - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
    depends_on:
      - redis

Before building the new containers we need to configure Celery in our Django app.

Celery Configuration

Setup

In the "core" directory, create a celery.py file and add the following code:

import os

from celery import Celery


os.environ.setdefault("DJANGO_SETTINGS_MODULE", "core.settings")

app = Celery("core")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()

What's happening here?

  1. First, we set a default value for the DJANGO_SETTINGS_MODULE environment variable so that the Celery will know how to find the Django project.
  2. Next, we created a new Celery instance, with the name core, and assigned the value to a variable called app.
  3. We then loaded the celery configuration values from the settings object from django.conf. We used namespace="CELERY" to prevent clashes with other Django settings. All config settings for Celery must be prefixed with CELERY_, in other words.
  4. Finally, app.autodiscover_tasks() tells Celery to look for Celery tasks from applications defined in settings.INSTALLED_APPS.

Add the following code to core/__init__.py:

from .celery import app as celery_app

__all__ = ("celery_app",)

Lastly, update the core/settings.py file with the following Celery settings so that it can connect to Redis:

CELERY_BROKER_URL = "redis://redis:6379"
CELERY_RESULT_BACKEND = "redis://redis:6379"

Build the new containers to ensure that everything works:

$ docker-compose up -d --build

Take a look at the logs for each service to see that they are ready, without errors:

$ docker-compose logs 'web'
$ docker-compose logs 'celery'
$ docker-compose logs 'celery-beat'
$ docker-compose logs 'redis'

If all went well, we now have four containers, each with different services.

Now we're ready to create a sample task to see that it works as it should.

Create a Task

Create a new file called core/tasks.py and add the following code for a sample task that just logs to the console:

from celery import shared_task
from celery.utils.log import get_task_logger


logger = get_task_logger(__name__)


@shared_task
def sample_task():
    logger.info("The sample task just ran.")

Schedule the Task

At the end of your settings.py file, add the following code to schedule sample_task to run once per minute, using Celery Beat:

CELERY_BEAT_SCHEDULE = {
    "sample_task": {
        "task": "core.tasks.sample_task",
        "schedule": crontab(minute="*/1"),
    },
}

Here, we defined a periodic task using the CELERY_BEAT_SCHEDULE setting. We gave the task a name, sample_task, and then declared two settings:

  1. task declares which task to run.
  2. schedule sets the interval on which the task should run. This can be an integer, a timedelta, or a crontab. We used a crontab pattern for our task to tell it to run once every minute. You can find more info on Celery's scheduling here.

Make sure to add the imports:

from celery.schedules import crontab

import core.tasks

Restart the container to pull in the new settings:

$ docker-compose up -d --build

Once done, take a look at the celery logs in the container:

$ docker-compose logs -f 'celery'

You should see something similar to:

celery_1  |  -------------- [queues]
celery_1  |                 .> celery           exchange=celery(direct) key=celery
celery_1  |
celery_1  |
celery_1  | [tasks]
celery_1  |   . core.tasks.sample_task

We can see that Celery picked up our sample task, core.tasks.sample_task.

Every minute you should see a row in the log that ends with "The sample task just ran.":

celery_1  | [2021-07-01 03:06:00,003: INFO/MainProcess]
              Task core.tasks.sample_task[b8041b6c-bf9b-47ce-ab00-c37c1e837bc7] received
celery_1  | [2021-07-01 03:06:00,004: INFO/ForkPoolWorker-8]
              core.tasks.sample_task[b8041b6c-bf9b-47ce-ab00-c37c1e837bc7]:
              The sample task just ran.

Custom Django Admin Command

Django provides a number of built-in django-admin commands, like:

  • migrate
  • startproject
  • startapp
  • dumpdata
  • makemigrations

Along with the built-in commands, Django also gives us the option to create our own custom commands:

Custom management commands are especially useful for running standalone scripts or for scripts that are periodically executed from the UNIX crontab or from Windows scheduled tasks control panel.

So, we'll first configure a new command and then use Celery Beat to run it automatically.

Start by creating a new file called orders/management/commands/my_custom_command.py. Then, add the minimal required code for it to run:

from django.core.management.base import BaseCommand, CommandError


class Command(BaseCommand):
    help = "A description of the command"

    def handle(self, *args, **options):
        pass

The BaseCommand has a few methods that can be overridden, but the only method that's required is handle. handle is the entry point for custom commands. In other words, when we run the command, this method is called.

To test, we'd normally just add a quick print statement. However, it's recommended to use stdout.write instead per the Django documentation:

When you are using management commands and wish to provide console output, you should write to self.stdout and self.stderr, instead of printing to stdout and stderr directly. By using these proxies, it becomes much easier to test your custom command. Note also that you don’t need to end messages with a newline character, it will be added automatically, unless you specify the ending parameter.

So, add a self.stdout.write command:

from django.core.management.base import BaseCommand, CommandError


class Command(BaseCommand):
    help = "A description of the command"

    def handle(self, *args, **options):
        self.stdout.write("My sample command just ran.") # NEW

To test, from the command line, run:

$ docker-compose exec web python manage.py my_custom_command

You should see:

My sample command just ran.

With that, let's tie everything together!

Schedule a Custom Command with Celery Beat

Now that we've spun up the containers, tested that we can schedule a task to run periodically, and wrote a custom Django Admin sample command, it's time to configure Celery Beat to run the custom command periodically.

Setup

In the project we have a very basic app called orders. It contains two models, Product and Order. Let's create a custom command that sends an email report of the confirmed orders from the day.

To begin with, we'll add a few products and orders to the database via the fixture included in this project:

$ docker-compose exec web python manage.py loaddata products.json

Next, add some sample orders via the Django Admin interface. To do so, first create a superuser:

$ docker-compose exec web python manage.py createsuperuser

Fill in username, email, and password when prompted. Then navigate to http://127.0.0.1:1337/admin in your web browser. Log in with the superuser you just created and create a couple of orders. Make sure at least one has a confirmed_date of today.

Let's create a new custom command for our e-mail report.

Create a file called orders/management/commands/email_report.py:

from datetime import timedelta, time, datetime

from django.core.mail import mail_admins
from django.core.management import BaseCommand
from django.utils import timezone
from django.utils.timezone import make_aware

from orders.models import Order

today = timezone.now()
tomorrow = today + timedelta(1)
today_start = make_aware(datetime.combine(today, time()))
today_end = make_aware(datetime.combine(tomorrow, time()))


class Command(BaseCommand):
    help = "Send Today's Orders Report to Admins"

    def handle(self, *args, **options):
        orders = Order.objects.filter(confirmed_date__range=(today_start, today_end))

        if orders:
            message = ""

            for order in orders:
                message += f"{order} \n"

            subject = (
                f"Order Report for {today_start.strftime('%Y-%m-%d')} "
                f"to {today_end.strftime('%Y-%m-%d')}"
            )

            mail_admins(subject=subject, message=message, html_message=None)

            self.stdout.write("E-mail Report was sent.")
        else:
            self.stdout.write("No orders confirmed today.")

In the code, we queried the database for orders with a confirmed_date of today, combined the orders into a single message for the email body, and used Django's built in mail_admins command to send the emails to the admins.

Add a dummy admin email and set the EMAIL_BACKEND to use the Console backend, so the email is sent to stdout, in the settings file:

EMAIL_BACKEND = "django.core.mail.backends.console.EmailBackend"
DEFAULT_FROM_EMAIL = "noreply@email.com"
ADMINS = [("testuser", "test.user@email.com"), ]

It should now be possible to run our new command from the terminal.

$ docker-compose exec web python manage.py email_report

And the output should look similar to this:

Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Subject: [Django] Order Report for 2021-07-01 to 2021-07-02
From: root@localhost
To: test.user@email.com
Date: Thu, 01 Jul 2021 12:15:50 -0000
Message-ID: <162514175053.40.5705892371538583115@5140844ebb15>

Order: 3947963f-1860-44d1-9b9a-4648fed04581 - product: Coffee
Order: ff449e6e-3dfd-48a8-9d5c-79a145d08253 - product: Rice

-------------------------------------------------------------------------------
E-mail Report was sent.

Celery Beat

We now need to create a periodic task to run this command daily.

Add a new task to core/tasks.py:

from celery import shared_task
from celery.utils.log import get_task_logger
from django.core.management import call_command # NEW


logger = get_task_logger(__name__)


@shared_task
def sample_task():
    logger.info("The sample task just ran.")


# NEW
@shared_task
def send_email_report():
    call_command("email_report", )

So, first we added a call_command import, which is used for programmatically calling django-admin commands. In the new task, we then used the call_command with the name of our custom command as an argument.

To schedule this task, open the core/settings.py file, and update the CELERY_BEAT_SCHEDULE setting to include the new task:

CELERY_BEAT_SCHEDULE = {
    "sample_task": {
        "task": "core.tasks.sample_task",
        "schedule": crontab(minute="*/1"),
    },
    "send_email_report": {
        "task": "core.tasks.send_email_report",
        "schedule": crontab(hour="*/1"),
    },
}

Here we added a new entry to the CELERY_BEAT_SCHEDULE called send_email_report. As we did for our previous task, we declared which task it should run -- e.g., core.tasks.send_email_report -- and used a crontab pattern to set the recurrence.

Restart the containers to make sure the new settings become active:

$ docker-compose up -d --build

Open the logs associated with the celery service:

$ docker-compose logs -f 'celery'

You should see the send_email_report listed:

celery_1  |  -------------- [queues]
celery_1  |                 .> celery           exchange=celery(direct) key=celery
celery_1  |
celery_1  |
celery_1  | [tasks]
celery_1  |   . core.tasks.sample_task
celery_1  |   . core.tasks.send_email_report

A minute or so later you should see that the e-mail report is sent:

celery_1  | [2021-07-01 12:22:00,071: WARNING/ForkPoolWorker-8] Content-Type: text/plain; charset="utf-8"
celery_1  | MIME-Version: 1.0
celery_1  | Content-Transfer-Encoding: 7bit
celery_1  | Subject: [Django] Order Report for 2021-07-01 to 2021-07-02
celery_1  | From: root@localhost
celery_1  | To: test.user@email.com
celery_1  | Date: Thu, 01 Jul 2021 12:22:00 -0000
celery_1  | Message-ID: <162514212006.17.6080459299558356876@55b9883c5414>
celery_1  |
celery_1  | Order: 3947963f-1860-44d1-9b9a-4648fed04581 - product: Coffee
celery_1  | Order: ff449e6e-3dfd-48a8-9d5c-79a145d08253 - product: Rice
celery_1  |
celery_1  |
celery_1  | [2021-07-01 12:22:00,071: WARNING/ForkPoolWorker-8] -------------------------------------------------------------------------------
celery_1  | [2021-07-01 12:22:00,071: WARNING/ForkPoolWorker-8]
celery_1  |
celery_1  | [2021-07-01 12:22:00,071: WARNING/ForkPoolWorker-8] E-mail Report was sent.

Conclusion

In this article we guided you through setting up Docker containers for Celery, Celery Beat, and Redis. We then showed how to create a custom Django Admin command and a periodic task with Celery Beat to run that command automatically.

Looking for more?

  1. Set up Flower to monitor and administer Celery jobs and workers
  2. Test a Celery task with both unit and integration tests

Grab the code from the repo.

Django + Celery Series:

  1. Asynchronous Tasks with Django and Celery
  2. Handling Periodic Tasks in Django with Celery and Docker (this article!)
  3. Automatically Retrying Failed Celery Tasks
  4. Working with Celery and Database Transactions

Original article source at: https://testdriven.io/

#django #celery #docker 

How to Manage Periodic Tasks with Django, Celery, and Docker
Nigel  Uys

Nigel Uys

1668750200

How to Automatically Retrying Failed Celery Tasks

In this article, we'll look at how to automatically retry failed Celery tasks.

Objectives

After reading, you should be able to:

  1. Retry a failed Celery task with both the retry method and a decorator argument
  2. Use exponential backoff when retrying a failed task
  3. Use a class-based task to reuse retry arguments

Celery Task

You can find the source code for this article on GitHub.

Let's assume we have a Celery task like this:

@shared_task
def task_process_notification():
    if not random.choice([0, 1]):
        # mimic random error
        raise Exception()

    requests.post('https://httpbin.org/delay/5')

In the real world this may call an internal or external third-party service. Regardless of the service, assume it's very unreliable, especially at peak periods. How can we handle failures?

It's worth noting that many Celery beginners get confused as to why some articles use app.task while others use shared_task. Well, shared_task lets you define Celery tasks without having to import the Celery instance, so it can make your task code more reusable.

Solution 1: Use a Try/Except Block

We can use a try/except block to catch the exception and raise retry:

@shared_task(bind=True)
def task_process_notification(self):
    try:
        if not random.choice([0, 1]):
            # mimic random error
            raise Exception()

        requests.post('https://httpbin.org/delay/5')
    except Exception as e:
        logger.error('exception raised, it would be retry after 5 seconds')
        raise self.retry(exc=e, countdown=5)

Notes:

  1. Since we set bind to True, this is a bound task, so the first argument to the task will always be the current task instance (self). Because of this, we can call self.retry to retry the failed task.
  2. Please remember to raise the exception returned by the self.retry method to make it work.
  3. By setting the countdown argument to 5, the task will retry after a 5 second delay.

Let's run the code below in the Python shell:

>>> from polls.tasks import task_process_notification
>>> task_process_notification.delay()

You should see output like this in your Celery worker terminal output:

Task polls.tasks.task_process_notification[06e1f985-90d4-4453-9870-fab57c5885c4] retry: Retry in 5s: Exception()
Task polls.tasks.task_process_notification[06e1f985-90d4-4453-9870-fab57c5885c4] retry: Retry in 5s: Exception()
Task polls.tasks.task_process_notification[06e1f985-90d4-4453-9870-fab57c5885c4] succeeded in 3.3638455480104312s: None

As you can see, the Celery task failed twice and succeeded the third time.

Solution 2: Task Retry Decorator

Celery 4.0 added built-in support for retrying, so you can let the exception bubble up and specify in the decorator how to handle it:

@shared_task(bind=True, autoretry_for=(Exception,), retry_kwargs={'max_retries': 7, 'countdown': 5})
def task_process_notification(self):
    if not random.choice([0, 1]):
        # mimic random error
        raise Exception()

    requests.post('https://httpbin.org/delay/5')

Notes:

  1. autoretry_for takes a list/tuple of exception types that you'd like to retry for.
  2. retry_kwargs takes a dictionary of additional options for specifying how autoretries are executed. In the above example, the task will retry after a 5 second delay (via countdown) and it allows for a maximum of 7 retry attempts (via max_retries). Celery will stop retrying after 7 failed attempts and raise an exception.

Exponential Backoff

If your Celery task needs to send a request to a third-party service, it's a good idea to use exponential backoff to avoid overwhelming the service.

Celery supports this by default:

@shared_task(bind=True, autoretry_for=(Exception,), retry_backoff=True, retry_kwargs={'max_retries': 5})
def task_process_notification(self):
    if not random.choice([0, 1]):
        # mimic random error
        raise Exception()

    requests.post('https://httpbin.org/delay/5')

In this example, the first retry should run after 1s, the following after 2s, the third one after 4s, the fourth after 8s, and so forth:

[02:09:59,014: INFO/ForkPoolWorker-8] Task polls.tasks.task_process_notification[fbe041b6-e6c1-453d-9cc9-cb99236df6ff] retry: Retry in 1s: Exception()
[02:10:00,210: INFO/ForkPoolWorker-2] Task polls.tasks.task_process_notification[fbe041b6-e6c1-453d-9cc9-cb99236df6ff] retry: Retry in 2s: Exception()
[02:10:02,291: INFO/ForkPoolWorker-4] Task polls.tasks.task_process_notification[fbe041b6-e6c1-453d-9cc9-cb99236df6ff] retry: Retry in 4s: Exception()

You can also set retry_backoff to a number for use as a delay factor:

@shared_task(bind=True, autoretry_for=(Exception,), retry_backoff=5, retry_kwargs={'max_retries': 5})
def task_process_notification(self):
    if not random.choice([0, 1]):
        # mimic random error
        raise Exception()

    requests.post('https://httpbin.org/delay/5')

Example:

[02:21:45,887: INFO/ForkPoolWorker-8] Task polls.tasks.task_process_notification[6a0b2682-74f5-410b-af1e-352069238f3d] retry: Retry in 5s: Exception()
[02:21:55,170: INFO/ForkPoolWorker-2] Task polls.tasks.task_process_notification[6a0b2682-74f5-410b-af1e-352069238f3d] retry: Retry in 10s: Exception()
[02:22:15,706: INFO/ForkPoolWorker-4] Task polls.tasks.task_process_notification[6a0b2682-74f5-410b-af1e-352069238f3d] retry: Retry in 20s: Exception()
[02:22:55,450: INFO/ForkPoolWorker-6] Task polls.tasks.task_process_notification[6a0b2682-74f5-410b-af1e-352069238f3d] retry: Retry in 40s: Exception()

By default, the exponential backoff will also introduce random jitter to avoid having all the tasks run at the same moment.

Randomness

When you build a custom retry strategy for your Celery task (which needs to send a request to another service), you should add some randomness to the delay calculation to prevent all tasks from being executed simultaneously resulting in a thundering herd.

Celery has you covered here as well with retry_jitter:

@shared_task(bind=True, autoretry_for=(Exception,), retry_backoff=5, retry_jitter=True, retry_kwargs={'max_retries': 5})
def task_process_notification(self):
    if not random.choice([0, 1]):
        # mimic random error
        raise Exception()

    requests.post('https://httpbin.org/delay/5')

This option is set to True by default, which helps prevent the thundering herd problem when you use Celery's built-in retry_backoff.

Task Base Class

If you find yourself writing the same retry arguments in your Celery task decorators, you can (as of Celery 4.4) define retry arguments in a base class, which you can then use as a base class in your Celery tasks:

class BaseTaskWithRetry(celery.Task):
    autoretry_for = (Exception, KeyError)
    retry_kwargs = {'max_retries': 5}
    retry_backoff = True


@shared_task(bind=True, base=BaseTaskWithRetry)
def task_process_notification(self):
    raise Exception()

So if you run the task in the Python shell, you would see the following:

[03:12:29,002: INFO/ForkPoolWorker-8] Task polls.tasks.task_process_notification[3231ef9b-00c7-4ab1-bf0b-2fdea6fa8348] retry: Retry in 1s: Exception()
[03:12:30,445: INFO/ForkPoolWorker-8] Task polls.tasks.task_process_notification[3231ef9b-00c7-4ab1-bf0b-2fdea6fa8348] retry: Retry in 2s: Exception()
[03:12:33,080: INFO/ForkPoolWorker-8] Task polls.tasks.task_process_notification[3231ef9b-00c7-4ab1-bf0b-2fdea6fa8348] retry: Retry in 3s: Exception()

Conclusion

In this Celery article, we looked at how to automatically retry failed celery tasks.

Again, the source code for this article can be found on GitHub.

Thanks for your reading. If you have any questions, please feel free to contact me.

Django + Celery Series:

  1. Asynchronous Tasks with Django and Celery
  2. Handling Periodic Tasks in Django with Celery and Docker
  3. Automatically Retrying Failed Celery Tasks (this article!)
  4. Working with Celery and Database Transactions

Original article source at: https://testdriven.io/

#django #celery 

How to Automatically Retrying Failed Celery Tasks
Rupert  Beatty

Rupert Beatty

1668674880

How to Asynchronous Tasks with Flask and Celery

If a long-running process is part of your application's workflow, rather than blocking the response, you should handle it in the background, outside the normal request/response flow.

Perhaps your web application requires users to submit a thumbnail (which will probably need to be re-sized) and confirm their email when they register. If your application processed the image and sent a confirmation email directly in the request handler, then the end user would have to wait unnecessarily for them both to finish processing before the page loads or updates. Instead, you'll want to pass these processes off to a task queue and let a separate worker process deal with it, so you can immediately send a response back to the client. The end user can then do other things on the client-side while the processing takes place. Your application is also free to respond to requests from other users and clients.

To achieve this, we'll walk you through the process of setting up and configuring Celery and Redis for handling long-running processes in a Flask app. We'll also use Docker and Docker Compose to tie everything together. Finally, we'll look at how to test the Celery tasks with unit and integration tests.

Redis Queue is a viable solution as well. Check out Asynchronous Tasks with Flask and Redis Queue for more.

Objectives

By the end of this tutorial, you will be able to:

  1. Integrate Celery into a Flask app and create tasks.
  2. Containerize Flask, Celery, and Redis with Docker.
  3. Run processes in the background with a separate worker process.
  4. Save Celery logs to a file.
  5. Set up Flower to monitor and administer Celery jobs and workers.
  6. Test a Celery task with both unit and integration tests.

Background Tasks

Again, to improve user experience, long-running processes should be run outside the normal HTTP request/response flow, in a background process.

Examples:

  1. Running machine learning models
  2. Sending confirmation emails
  3. Scraping and crawling
  4. Analyzing data
  5. Processing images
  6. Generating reports

As you're building out an app, try to distinguish tasks that should run during the request/response lifecycle, like CRUD operations, from those that should run in the background.

Workflow

Our goal is to develop a Flask application that works in conjunction with Celery to handle long-running processes outside the normal request/response cycle.

  1. The end user kicks off a new task via a POST request to the server-side.
  2. Within the route handler, a task is added to the queue and the task ID is sent back to the client-side.
  3. Using AJAX, the client continues to poll the server to check the status of the task while the task itself is running in the background.

flask and celery queue user flow

Project Setup

Clone down the base project from the flask-celery repo, and then check out the v1 tag to the master branch:

$ git clone https://github.com/testdrivenio/flask-celery --branch v1 --single-branch
$ cd flask-celery
$ git checkout v1 -b master

Since we'll need to manage three processes in total (Flask, Redis, Celery worker), we'll use Docker to simplify our workflow by wiring them up so that they can all be run from one terminal window with a single command.

From the project root, create the images and spin up the Docker containers:

$ docker-compose up -d --build

Once the build is complete, navigate to http://localhost:5004:

flask project

Make sure the tests pass as well:

$ docker-compose exec web python -m pytest

================================== test session starts ===================================
platform linux -- Python 3.10.2, pytest-7.0.1, pluggy-1.0.0
rootdir: /usr/src/app
collected 1 item

project/tests/test_tasks.py .                                                       [100%]

=================================== 1 passed in 0.34s ====================================

Take a quick look at the project structure before moving on:

├── .gitignore
├── Dockerfile
├── LICENSE
├── README.md
├── docker-compose.yml
├── manage.py
├── project
│   ├── __init__.py
│   ├── client
│   │   ├── static
│   │   │   ├── main.css
│   │   │   └── main.js
│   │   └── templates
│   │       ├── _base.html
│   │       ├── footer.html
│   │       └── main
│   │           └── home.html
│   ├── server
│   │   ├── __init__.py
│   │   ├── config.py
│   │   └── main
│   │       ├── __init__.py
│   │       └── views.py
│   └── tests
│       ├── __init__.py
│       ├── conftest.py
│       └── test_tasks.py
└── requirements.txt

Want to learn how to build this project? Check out the Dockerizing Flask with Postgres, Gunicorn, and Nginx article.

Trigger a Task

An onclick event handler in project/client/templates/main/home.html is set up that listens for a button click:

<div class="btn-group" role="group" aria-label="Basic example">
  <button type="button" class="btn btn-primary" onclick="handleClick(1)">Short</button>
  <button type="button" class="btn btn-primary" onclick="handleClick(2)">Medium</button>
  <button type="button" class="btn btn-primary" onclick="handleClick(3)">Long</button>
</div>

onclick calls handleClick found in project/client/static/main.js, which sends an AJAX POST request to the server with the appropriate task type: 1, 2, or 3.

function handleClick(type) {
  fetch('/tasks', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({ type: type }),
  })
  .then(response => response.json())
  .then(data => getStatus(data.task_id));
}

On the server-side, a route is already configured to handle the request in project/server/main/views.py:

@main_blueprint.route("/tasks", methods=["POST"])
def run_task():
    content = request.json
    task_type = content["type"]
    return jsonify(task_type), 202

Now comes the fun part -- wiring up Celery!

Celery Setup

Start by adding both Celery and Redis to the requirements.txt file:

celery==5.2.3
Flask==2.0.3
Flask-WTF==1.0.0
pytest==7.0.1
redis==4.1.4

Celery uses a message broker -- RabbitMQ, Redis, or AWS Simple Queue Service (SQS) -- to facilitate communication between the Celery worker and the web application. Messages are added to the broker, which are then processed by the worker(s). Once done, the results are added to the backend.

Redis will be used as both the broker and backend. Add both Redis and a Celery worker to the docker-compose.yml file like so:

version: '3.8'

services:

  web:
    build: .
    image: web
    container_name: web
    ports:
      - 5004:5000
    command: python manage.py run -h 0.0.0.0
    volumes:
      - .:/usr/src/app
    environment:
      - FLASK_DEBUG=1
      - APP_SETTINGS=project.server.config.DevelopmentConfig
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/0
    depends_on:
      - redis

  worker:
    build: .
    command: celery --app project.server.tasks.celery worker --loglevel=info
    volumes:
      - .:/usr/src/app
    environment:
      - FLASK_DEBUG=1
      - APP_SETTINGS=project.server.config.DevelopmentConfig
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/0
    depends_on:
      - web
      - redis

  redis:
    image: redis:6-alpine

Take note of celery --app project.server.tasks.celery worker --loglevel=info:

  1. celery worker is used to start a Celery worker
  2. --app=project.server.tasks.celery runs the Celery Application (which we'll define shortly)
  3. --loglevel=info sets the logging level to info

Next, create a new file called tasks.py in "project/server":

import os
import time

from celery import Celery


celery = Celery(__name__)
celery.conf.broker_url = os.environ.get("CELERY_BROKER_URL", "redis://localhost:6379")
celery.conf.result_backend = os.environ.get("CELERY_RESULT_BACKEND", "redis://localhost:6379")


@celery.task(name="create_task")
def create_task(task_type):
    time.sleep(int(task_type) * 10)
    return True

Here, we created a new Celery instance, and using the task decorator, we defined a new Celery task function called create_task.

Keep in mind that the task itself will be executed by the Celery worker.

Trigger a Task

Update the route handler to kick off the task and respond with the task ID:

@main_blueprint.route("/tasks", methods=["POST"])
def run_task():
    content = request.json
    task_type = content["type"]
    task = create_task.delay(int(task_type))
    return jsonify({"task_id": task.id}), 202

Don't forget to import the task:

from project.server.tasks import create_task

Build the images and spin up the new containers:

$ docker-compose up -d --build

To trigger a new task, run:

$ curl http://localhost:5004/tasks -H "Content-Type: application/json" --data '{"type": 0}'

You should see something like:

{
  "task_id": "14049663-6257-4a1f-81e5-563c714e90af"
}

Task Status

Turn back to the handleClick function on the client-side:

function handleClick(type) {
  fetch('/tasks', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({ type: type }),
  })
  .then(response => response.json())
  .then(data => getStatus(data.task_id));
}

When the response comes back from the original AJAX request, we then continue to call getStatus() with the task ID every second:

function getStatus(taskID) {
  fetch(`/tasks/${taskID}`, {
    method: 'GET',
    headers: {
      'Content-Type': 'application/json'
    },
  })
  .then(response => response.json())
  .then(res => {
    const html = `
      <tr>
        <td>${taskID}</td>
        <td>${res.task_status}</td>
        <td>${res.task_result}</td>
      </tr>`;
    const newRow = document.getElementById('tasks').insertRow(0);
    newRow.innerHTML = html;

    const taskStatus = res.task_status;
    if (taskStatus === 'SUCCESS' || taskStatus === 'FAILURE') return false;
    setTimeout(function() {
      getStatus(res.task_id);
    }, 1000);
  })
  .catch(err => console.log(err));
}

If the response is successful, a new row is added to the table on the DOM.

Update the get_status route handler to return the status:

@main_blueprint.route("/tasks/<task_id>", methods=["GET"])
def get_status(task_id):
    task_result = AsyncResult(task_id)
    result = {
        "task_id": task_id,
        "task_status": task_result.status,
        "task_result": task_result.result
    }
    return jsonify(result), 200

Import AsyncResult:

from celery.result import AsyncResult

Update the containers:

$ docker-compose up -d --build

Trigger a new task:

$ curl http://localhost:5004/tasks -H "Content-Type: application/json" --data '{"type": 1}'

Then, grab the task_id from the response and call the updated endpoint to view the status:

$ curl http://localhost:5004/tasks/f3ae36f1-58b8-4c2b-bf5b-739c80e9d7ff

{
  "task_id": "455234e0-f0ea-4a39-bbe9-e3947e248503",
  "task_result": true,
  "task_status": "SUCCESS"
}

Test it out in the browser as well:

flask, celery, docker

Celery Logs

Update the worker service, in docker-compose.yml, so that Celery logs are dumped to a log file:

worker:
  build: .
  command: celery --app project.server.tasks.celery worker --loglevel=info --logfile=project/logs/celery.log
  volumes:
    - .:/usr/src/app
  environment:
    - FLASK_DEBUG=1
    - APP_SETTINGS=project.server.config.DevelopmentConfig
    - CELERY_BROKER_URL=redis://redis:6379/0
    - CELERY_RESULT_BACKEND=redis://redis:6379/0
  depends_on:
    - web
    - redis

Add a new directory to "project" called "logs. Then, add a new file called celery.log to that newly created directory.

Update:

$ docker-compose up -d --build

You should see the log file fill up locally since we set up a volume:

[2022-02-16 21:01:09,961: INFO/MainProcess] Connected to redis://redis:6379/0
[2022-02-16 21:01:09,965: INFO/MainProcess] mingle: searching for neighbors
[2022-02-16 21:01:10,977: INFO/MainProcess] mingle: all alone
[2022-02-16 21:01:10,994: INFO/MainProcess] celery@f9921f0e0b83 ready.
[2022-02-16 21:01:23,349: INFO/MainProcess]
    Task create_task[ceb6cffc-e426-4970-a5df-5a1fac4478cc] received
[2022-02-16 21:01:33,378: INFO/ForkPoolWorker-7]
    Task create_task[ceb6cffc-e426-4970-a5df-5a1fac4478cc]
    succeeded in 10.025073800003156s: True

Flower Dashboard

Flower is a lightweight, real-time, web-based monitoring tool for Celery. You can monitor currently running tasks, increase or decrease the worker pool, view graphs and a number of statistics, to name a few.

Add it to requirements.txt:

celery==5.2.3
Flask==2.0.3
Flask-WTF==1.0.0
flower==1.0.0
pytest==7.0.1
redis==4.1.4

Then, add a new service to docker-compose.yml:

dashboard:
  build: .
  command: celery --app project.server.tasks.celery flower --port=5555 --broker=redis://redis:6379/0
  ports:
    - 5556:5555
  environment:
    - FLASK_DEBUG=1
    - APP_SETTINGS=project.server.config.DevelopmentConfig
    - CELERY_BROKER_URL=redis://redis:6379/0
    - CELERY_RESULT_BACKEND=redis://redis:6379/0
  depends_on:
    - web
    - redis
    - worker

Test it out:

$ docker-compose up -d --build

Navigate to http://localhost:5556 to view the dashboard. You should see one worker ready to go:

flower dashboard

Kick off a few more tasks to fully test the dashboard:

flower dashboard

Try adding a few more workers to see how that affects things:

$ docker-compose up -d --build --scale worker=3

Tests

Let's start with the most basic test:

def test_task():
    assert create_task.run(1)
    assert create_task.run(2)
    assert create_task.run(3)

Add the above test case to project/tests/test_tasks.py, and then add the following import:

from project.server.tasks import create_task

Run that test individually:

$ docker-compose exec web python -m pytest -k "test_task and not test_home"

It should take about one minute to run:

================================== test session starts ===================================
platform linux -- Python 3.10.2, pytest-7.0.1, pluggy-1.0.0
rootdir: /usr/src/app
collected 2 items / 1 deselected / 1 selected

project/tests/test_tasks.py .                                                       [100%]

====================== 1 passed, 1 deselected in 60.28s (0:01:00) ========================

It's worth noting that in the above asserts, we used the .run method (rather than .delay) to run the task directly without a Celery worker.

Want to mock the .run method to speed things up?

@patch("project.server.tasks.create_task.run")
def test_mock_task(mock_run):
    assert create_task.run(1)
    create_task.run.assert_called_once_with(1)

    assert create_task.run(2)
    assert create_task.run.call_count == 2

    assert create_task.run(3)
    assert create_task.run.call_count == 3

Import:

from unittest.mock import patch, call

Test:

$ docker-compose exec web python -m pytest -k "test_mock_task"

================================== test session starts ===================================
platform linux -- Python 3.10.2, pytest-7.0.1, pluggy-1.0.0
rootdir: /usr/src/app
collected 3 items / 2 deselected / 1 selected

project/tests/test_tasks.py .                                                       [100%]

============================ 1 passed, 2 deselected in 0.37s =============================

Much quicker!

How about a full integration test?

def test_task_status(test_app):
    client = test_app.test_client()

    resp = client.post(
        "/tasks",
        data=json.dumps({"type": 0}),
        content_type='application/json'
    )
    content = json.loads(resp.data.decode())
    task_id = content["task_id"]
    assert resp.status_code == 202
    assert task_id

    resp = client.get(f"tasks/{task_id}")
    content = json.loads(resp.data.decode())
    assert content == {"task_id": task_id, "task_status": "PENDING", "task_result": None}
    assert resp.status_code == 200

    while content["task_status"] == "PENDING":
        resp = client.get(f"tasks/{task_id}")
        content = json.loads(resp.data.decode())
    assert content == {"task_id": task_id, "task_status": "SUCCESS", "task_result": True}

Keep in mind that this test uses the same broker and backend used in development. You may want to instantiate a new Celery app for testing.

Add the import:

import json

Ensure the test passes.

Conclusion

This has been a basic guide on how to configure Celery to run long-running tasks in a Flask app. You should let the queue handle any processes that could block or slow down the user-facing code.

Celery can also be used to execute repeatable tasks and break up complex, resource-intensive tasks so that the computational workload can be distributed across a number of machines to reduce (1) the time to completion and (2) the load on the machine handling client requests.

Finally, if you're curious about how to use WebSockets to check the status of a Celery task, instead of using AJAX polling, check out the The Definitive Guide to Celery and Flask course.

Grab the code from the repo.

Original article source at: https://testdriven.io/

#flask #celery #docker 

How to Asynchronous Tasks with Flask and Celery
Vicenta  Hauck

Vicenta Hauck

1660964220

How to Set Up Django, Celery, and Redis with Docker

As you build and scale a Django app you'll inevitably need to run certain tasks periodically and automatically in the background.

In the following article, we'll show you how to set up Django, Celery, and Redis with Docker in order to run a custom Django Admin command periodically with Celery Beat.

Source: https://testdriven.io

#django #celery #docker 

How to Set Up Django, Celery, and Redis with Docker
Hoang Tran

Hoang Tran

1660956960

Cách Thiết Lập Django, Celery Và Redis Bằng Docker

Khi bạn xây dựng và mở rộng một ứng dụng Django, chắc chắn bạn sẽ cần chạy một số tác vụ nhất định theo định kỳ và tự động trong nền.

Vài ví dụ:

  • Tạo báo cáo định kỳ
  • Xóa bộ nhớ cache
  • Gửi thông báo e-mail hàng loạt
  • Thực hiện công việc bảo trì hàng đêm

Đây là một trong số ít các chức năng cần thiết để xây dựng và mở rộng ứng dụng web không phải là một phần của lõi Django. May mắn thay, Celery cung cấp một giải pháp mạnh mẽ, khá dễ thực hiện được gọi là Celery Beat.

Trong bài viết sau, chúng tôi sẽ hướng dẫn bạn cách thiết lập Django, Celery và Redis với Docker để chạy lệnh quản trị Django tùy chỉnh định kỳ với Celery Beat.

Mục tiêu

Đến cuối hướng dẫn này, bạn sẽ có thể:

  1. Chứa Django, Celery và Redis bằng Docker
  2. Tích hợp Celery vào ứng dụng Django và tạo nhiệm vụ
  3. Viết lệnh Quản trị Django tùy chỉnh
  4. Lên lịch để lệnh Quản trị Django tùy chỉnh chạy định kỳ qua Celery Beat

Thiết lập dự án

Sao chép dự án cơ sở từ repo django-celery-beat , sau đó kiểm tra nhánh cơ sở :

$ git clone https://github.com/testdrivenio/django-celery-beat --branch base --single-branch
$ cd django-celery-beat

Vì chúng tôi sẽ cần quản lý tổng cộng bốn quy trình (Django, Redis, worker và lập lịch), chúng tôi sẽ sử dụng Docker để đơn giản hóa quy trình làm việc của mình bằng cách kết nối chúng để tất cả chúng có thể được chạy từ một cửa sổ đầu cuối bằng một lệnh duy nhất .

Từ thư mục gốc của dự án, tạo hình ảnh và xoay các vùng chứa Docker:

$ docker-compose up -d --build

Tiếp theo, áp dụng các di chuyển:

$ docker-compose exec web python manage.py migrate

Khi quá trình xây dựng hoàn tất, hãy điều hướng đến http: // localhost: 1337 để đảm bảo ứng dụng hoạt động như mong đợi. Bạn sẽ thấy văn bản sau:

Orders

No orders found!

Hãy xem nhanh cấu trúc dự án trước khi chuyển sang:

├── .gitignore
├── docker-compose.yml
└── project
    ├── Dockerfile
    ├── core
    │   ├── __init__.py
    │   ├── asgi.py
    │   ├── settings.py
    │   ├── urls.py
    │   └── wsgi.py
    ├── entrypoint.sh
    ├── manage.py
    ├── orders
    │   ├── __init__.py
    │   ├── admin.py
    │   ├── apps.py
    │   ├── migrations
    │   │   ├── 0001_initial.py
    │   │   └── __init__.py
    │   ├── models.py
    │   ├── tests.py
    │   ├── urls.py
    │   └── views.py
    ├── products.json
    ├── requirements.txt
    └── templates
        └── orders
            └── order_list.html

Bạn muốn tìm hiểu cách xây dựng dự án này? Xem bài đăng trên blog Dockerizing Django với Postgres, Gunicorn và Nginx .

Celery and Redis

Bây giờ, chúng ta cần thêm các vùng chứa cho Celery, Celery Beat và Redis.

Chúng ta sẽ bắt đầu bằng cách thêm các phần phụ thuộc vào tệp tin request.txt :

Django==3.2.4
celery==5.1.2
redis==3.5.3

Tiếp theo, thêm phần sau vào cuối tệp docker-compost.yml :

redis:
  image: redis:alpine
celery:
  build: ./project
  command: celery -A core worker -l info
  volumes:
    - ./project/:/usr/src/app/
  environment:
    - DEBUG=1
    - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
    - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
  depends_on:
    - redis
celery-beat:
  build: ./project
  command: celery -A core beat -l info
  volumes:
    - ./project/:/usr/src/app/
  environment:
    - DEBUG=1
    - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
    - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
  depends_on:
    - redis

Chúng tôi cũng cần cập nhật phần của dịch vụ web depends_on:

web:
  build: ./project
  command: python manage.py runserver 0.0.0.0:8000
  volumes:
    - ./project/:/usr/src/app/
  ports:
    - 1337:8000
  environment:
    - DEBUG=1
    - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
    - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
  depends_on:
    - redis # NEW

Tệp docker-compo.yml đầy đủ bây giờ sẽ trông giống như sau:

version: '3.8'

services:
  web:
    build: ./project
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - ./project/:/usr/src/app/
    ports:
      - 1337:8000
    environment:
      - DEBUG=1
      - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
      - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
    depends_on:
      - redis
  redis:
    image: redis:alpine
  celery:
    build: ./project
    command: celery -A core worker -l info
    volumes:
      - ./project/:/usr/src/app/
    environment:
      - DEBUG=1
      - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
      - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
    depends_on:
      - redis
  celery-beat:
    build: ./project
    command: celery -A core beat -l info
    volumes:
      - ./project/:/usr/src/app/
    environment:
      - DEBUG=1
      - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
      - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
    depends_on:
      - redis

Trước khi xây dựng các vùng chứa mới, chúng ta cần định cấu hình Cần tây trong ứng dụng Django của chúng ta.

Cấu hình cần tây

Thành lập

Trong thư mục "lõi", tạo tệp celery.py và thêm mã sau:

import os

from celery import Celery


os.environ.setdefault("DJANGO_SETTINGS_MODULE", "core.settings")

app = Celery("core")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()

Điều gì đang xảy ra ở đây?

  1. Đầu tiên, chúng tôi đặt một giá trị mặc định cho DJANGO_SETTINGS_MODULEbiến môi trường để Celery biết cách tìm dự án Django.
  2. Tiếp theo, chúng tôi tạo một cá thể Celery mới, với tên corevà gán giá trị cho một biến được gọi app.
  3. Sau đó, chúng tôi tải các giá trị cấu hình cần tây từ đối tượng cài đặt từ django.conf. Chúng tôi đã sử dụng namespace="CELERY"để ngăn chặn xung đột với các cài đặt Django khác. Nói cách khác, tất cả các cài đặt cấu hình cho Celery phải có tiền tố CELERY_.
  4. Cuối cùng, app.autodiscover_tasks()yêu cầu Celery tìm kiếm các tác vụ Celery từ các ứng dụng được xác định trong settings.INSTALLED_APPS.

Thêm mã sau vào core / __ init__.py :

from .celery import app as celery_app

__all__ = ("celery_app",)

Cuối cùng, hãy cập nhật tệp core / settings.py với cài đặt Celery sau để nó có thể kết nối với Redis:

CELERY_BROKER_URL = "redis://redis:6379"
CELERY_RESULT_BACKEND = "redis://redis:6379"

Xây dựng các vùng chứa mới để đảm bảo rằng mọi thứ hoạt động:

$ docker-compose up -d --build

Hãy xem nhật ký của từng dịch vụ để biết rằng chúng đã sẵn sàng, không có lỗi:

$ docker-compose logs 'web'
$ docker-compose logs 'celery'
$ docker-compose logs 'celery-beat'
$ docker-compose logs 'redis'

Nếu mọi việc suôn sẻ, chúng tôi hiện có bốn container, mỗi container có các dịch vụ khác nhau.

Bây giờ chúng tôi đã sẵn sàng tạo một nhiệm vụ mẫu để xem nó hoạt động như bình thường.

Tạo một công việc

Tạo một tệp mới có tên là core / task.py và thêm mã sau cho một tác vụ mẫu chỉ ghi vào bảng điều khiển:

from celery import shared_task
from celery.utils.log import get_task_logger


logger = get_task_logger(__name__)


@shared_task
def sample_task():
    logger.info("The sample task just ran.")

Lên lịch công việc

Ở cuối tệp settings.py của bạn , hãy thêm mã sau để lập lịch sample_taskchạy một lần mỗi phút, sử dụng Celery Beat:

CELERY_BEAT_SCHEDULE = {
    "sample_task": {
        "task": "core.tasks.sample_task",
        "schedule": crontab(minute="*/1"),
    },
}

Ở đây, chúng tôi đã xác định một nhiệm vụ định kỳ bằng cách sử dụng cài đặt CELERY_BEAT_SCHEDULE . Chúng tôi đã đặt tên cho nhiệm vụ sample_taskvà sau đó khai báo hai cài đặt:

  1. taskkhai báo tác vụ nào sẽ chạy.
  2. scheduleđặt khoảng thời gian mà tác vụ sẽ chạy. Đây có thể là một số nguyên, một bộ đếm thời gian hoặc một crontab. Chúng tôi đã sử dụng một mẫu crontab cho nhiệm vụ của chúng tôi để yêu cầu nó chạy một lần mỗi phút. Bạn có thể tìm thêm thông tin về lịch trình của Celery tại đây .

Đảm bảo thêm các mục nhập:

from celery.schedules import crontab

import core.tasks

Khởi động lại vùng chứa để lấy cài đặt mới:

$ docker-compose up -d --build

Sau khi hoàn tất, hãy xem nhật ký cần tây trong thùng chứa:

$ docker-compose logs -f 'celery'

Bạn sẽ thấy một cái gì đó tương tự như:

celery_1  |  -------------- [queues]
celery_1  |                 .> celery           exchange=celery(direct) key=celery
celery_1  |
celery_1  |
celery_1  | [tasks]
celery_1  |   . core.tasks.sample_task

Chúng ta có thể thấy rằng Celery đã chọn nhiệm vụ mẫu của chúng ta , core.tasks.sample_task.

Mỗi phút, bạn sẽ thấy một hàng trong nhật ký kết thúc bằng "Nhiệm vụ mẫu vừa chạy.":

celery_1  | [2021-07-01 03:06:00,003: INFO/MainProcess]
              Task core.tasks.sample_task[b8041b6c-bf9b-47ce-ab00-c37c1e837bc7] received
celery_1  | [2021-07-01 03:06:00,004: INFO/ForkPoolWorker-8]
              core.tasks.sample_task[b8041b6c-bf9b-47ce-ab00-c37c1e837bc7]:
              The sample task just ran.

Lệnh quản trị Django tùy chỉnh

Django cung cấp một số lệnh tích hợp django-admin, như:

  • migrate
  • startproject
  • startapp
  • dumpdata
  • makemigrations

Cùng với các lệnh tích hợp, Django cũng cung cấp cho chúng ta tùy chọn để tạo các lệnh tùy chỉnh của riêng mình :

Các lệnh quản lý tùy chỉnh đặc biệt hữu ích để chạy các tập lệnh độc lập hoặc cho các tập lệnh được thực thi định kỳ từ UNIX crontab hoặc từ bảng điều khiển tác vụ đã lên lịch của Windows.

Vì vậy, trước tiên chúng ta sẽ cấu hình một lệnh mới và sau đó sử dụng Celery Beat để chạy nó tự động.

Bắt đầu bằng cách tạo một tệp mới có tên là đơn đặt hàng / quản lý / lệnh / my_custom_command.py . Sau đó, thêm mã bắt buộc tối thiểu để nó chạy:

from django.core.management.base import BaseCommand, CommandError


class Command(BaseCommand):
    help = "A description of the command"

    def handle(self, *args, **options):
        pass

BaseCommandmột số phương thức có thể được ghi đè, nhưng phương thức duy nhất được yêu cầu là handle. handlelà điểm nhập cho các lệnh tùy chỉnh. Nói cách khác, khi chúng ta chạy lệnh, phương thức này được gọi.

Để kiểm tra, chúng tôi thường chỉ thêm một câu lệnh in nhanh. Tuy nhiên, bạn nên sử dụng stdout.writethay thế theo tài liệu Django:

Khi bạn đang sử dụng các lệnh quản lý và muốn cung cấp đầu ra bảng điều khiển, bạn nên viết vào self.stdout và self.stderr, thay vì in trực tiếp ra stdout và stderr. Bằng cách sử dụng các proxy này, việc kiểm tra lệnh tùy chỉnh của bạn trở nên dễ dàng hơn nhiều. Cũng lưu ý rằng bạn không cần phải kết thúc thư bằng một ký tự dòng mới, nó sẽ được thêm tự động, trừ khi bạn chỉ định tham số kết thúc.

Vì vậy, hãy thêm một self.stdout.writelệnh:

from django.core.management.base import BaseCommand, CommandError


class Command(BaseCommand):
    help = "A description of the command"

    def handle(self, *args, **options):
        self.stdout.write("My sample command just ran.") # NEW

Để kiểm tra, từ dòng lệnh, hãy chạy:

$ docker-compose exec web python manage.py my_custom_command

Bạn nên thấy:

My sample command just ran.

Cùng với đó, hãy gắn kết mọi thứ lại với nhau!

Lên lịch một lệnh tùy chỉnh với Celery Beat

Bây giờ chúng tôi đã xoay vòng các vùng chứa, đã kiểm tra rằng chúng tôi có thể lên lịch tác vụ để chạy định kỳ và viết lệnh mẫu Django Admin tùy chỉnh, đã đến lúc định cấu hình Celery Beat để chạy lệnh tùy chỉnh theo định kỳ.

Thành lập

Trong dự án, chúng tôi có một ứng dụng rất cơ bản được gọi là đơn đặt hàng. Nó chứa hai mô hình, ProductOrder. Hãy tạo một lệnh tùy chỉnh để gửi một báo cáo email về các đơn đặt hàng đã được xác nhận trong ngày.

Để bắt đầu, chúng tôi sẽ thêm một vài sản phẩm và đơn đặt hàng vào cơ sở dữ liệu thông qua vật cố định có trong dự án này:

$ docker-compose exec web python manage.py loaddata products.json

Tiếp theo, thêm một số đơn đặt hàng mẫu qua giao diện Django Admin. Để làm như vậy, trước tiên hãy tạo một superuser:

$ docker-compose exec web python manage.py createsuperuser

Điền tên người dùng, email và mật khẩu khi được nhắc. Sau đó, điều hướng đến http://127.0.0.1:1337/admin trong trình duyệt web của bạn. Đăng nhập với superuser mà bạn vừa tạo và tạo một vài đơn đặt hàng. Hãy chắc chắn rằng ít nhất một người có confirmed_datengày hôm nay.

Hãy tạo một lệnh tùy chỉnh mới cho báo cáo e-mail của chúng ta.

Tạo một tệp có tên là đơn đặt hàng / quản lý / lệnh / email_report.py :

from datetime import timedelta, time, datetime

from django.core.mail import mail_admins
from django.core.management import BaseCommand
from django.utils import timezone
from django.utils.timezone import make_aware

from orders.models import Order

today = timezone.now()
tomorrow = today + timedelta(1)
today_start = make_aware(datetime.combine(today, time()))
today_end = make_aware(datetime.combine(tomorrow, time()))


class Command(BaseCommand):
    help = "Send Today's Orders Report to Admins"

    def handle(self, *args, **options):
        orders = Order.objects.filter(confirmed_date__range=(today_start, today_end))

        if orders:
            message = ""

            for order in orders:
                message += f"{order} \n"

            subject = (
                f"Order Report for {today_start.strftime('%Y-%m-%d')} "
                f"to {today_end.strftime('%Y-%m-%d')}"
            )

            mail_admins(subject=subject, message=message, html_message=None)

            self.stdout.write("E-mail Report was sent.")
        else:
            self.stdout.write("No orders confirmed today.")

Trong mã, chúng tôi truy vấn cơ sở dữ liệu cho các đơn đặt hàng có confirmed_datengày hôm nay, kết hợp các đơn đặt hàng thành một thông báo duy nhất cho nội dung email và sử dụng mail_adminslệnh tích hợp của Django để gửi email đến quản trị viên.

Thêm một email quản trị giả và đặt EMAIL_BACKENDđể sử dụng phần phụ trợ Console , để email được gửi đến stdout, trong tệp cài đặt:

EMAIL_BACKEND = "django.core.mail.backends.console.EmailBackend"
DEFAULT_FROM_EMAIL = "noreply@email.com"
ADMINS = [("testuser", "test.user@email.com"), ]

Bây giờ có thể chạy lệnh mới của chúng tôi từ thiết bị đầu cuối.

$ docker-compose exec web python manage.py email_report

Và đầu ra sẽ giống như sau:

Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Subject: [Django] Order Report for 2021-07-01 to 2021-07-02
From: root@localhost
To: test.user@email.com
Date: Thu, 01 Jul 2021 12:15:50 -0000
Message-ID: <162514175053.40.5705892371538583115@5140844ebb15>

Order: 3947963f-1860-44d1-9b9a-4648fed04581 - product: Coffee
Order: ff449e6e-3dfd-48a8-9d5c-79a145d08253 - product: Rice

-------------------------------------------------------------------------------
E-mail Report was sent.

Celery Beat

Bây giờ chúng ta cần tạo một nhiệm vụ định kỳ để chạy lệnh này hàng ngày.

Thêm một nhiệm vụ mới vào core / task.py :

from celery import shared_task
from celery.utils.log import get_task_logger
from django.core.management import call_command # NEW


logger = get_task_logger(__name__)


@shared_task
def sample_task():
    logger.info("The sample task just ran.")


# NEW
@shared_task
def send_email_report():
    call_command("email_report", )

Vì vậy, trước tiên, chúng tôi đã thêm một call_commandnhập, được sử dụng để gọi các lệnh django-admin theo chương trình. Trong tác vụ mới, sau đó chúng tôi sử dụng call_commandtên lệnh tùy chỉnh của chúng tôi làm đối số.

Để lên lịch cho tác vụ này, hãy mở tệp core / settings.py và cập nhật CELERY_BEAT_SCHEDULEcài đặt để bao gồm tác vụ mới:

CELERY_BEAT_SCHEDULE = {
    "sample_task": {
        "task": "core.tasks.sample_task",
        "schedule": crontab(minute="*/1"),
    },
    "send_email_report": {
        "task": "core.tasks.send_email_report",
        "schedule": crontab(hour="*/1"),
    },
}

Ở đây chúng tôi đã thêm một mục mới vào CELERY_BEAT_SCHEDULEđược gọi send_email_report. Như chúng ta đã làm cho tác vụ trước đó, chúng tôi đã khai báo tác vụ nào nó sẽ chạy - ví dụ, core.tasks.send_email_report- và sử dụng một mẫu crontab để đặt thời gian lặp lại.

Khởi động lại vùng chứa để đảm bảo cài đặt mới đang hoạt động:

$ docker-compose up -d --build

Mở nhật ký được liên kết với celerydịch vụ:

$ docker-compose logs -f 'celery'

Bạn sẽ thấy send_email_reportdanh sách:

celery_1  |  -------------- [queues]
celery_1  |                 .> celery           exchange=celery(direct) key=celery
celery_1  |
celery_1  |
celery_1  | [tasks]
celery_1  |   . core.tasks.sample_task
celery_1  |   . core.tasks.send_email_report

Một phút sau, bạn sẽ thấy rằng báo cáo e-mail đã được gửi:

celery_1  | [2021-07-01 12:22:00,071: WARNING/ForkPoolWorker-8] Content-Type: text/plain; charset="utf-8"
celery_1  | MIME-Version: 1.0
celery_1  | Content-Transfer-Encoding: 7bit
celery_1  | Subject: [Django] Order Report for 2021-07-01 to 2021-07-02
celery_1  | From: root@localhost
celery_1  | To: test.user@email.com
celery_1  | Date: Thu, 01 Jul 2021 12:22:00 -0000
celery_1  | Message-ID: <162514212006.17.6080459299558356876@55b9883c5414>
celery_1  |
celery_1  | Order: 3947963f-1860-44d1-9b9a-4648fed04581 - product: Coffee
celery_1  | Order: ff449e6e-3dfd-48a8-9d5c-79a145d08253 - product: Rice
celery_1  |
celery_1  |
celery_1  | [2021-07-01 12:22:00,071: WARNING/ForkPoolWorker-8] -------------------------------------------------------------------------------
celery_1  | [2021-07-01 12:22:00,071: WARNING/ForkPoolWorker-8]
celery_1  |
celery_1  | [2021-07-01 12:22:00,071: WARNING/ForkPoolWorker-8] E-mail Report was sent.

Sự kết luận

Trong bài viết này, chúng tôi đã hướng dẫn bạn cách thiết lập vùng chứa Docker cho Celery, Celery Beat và Redis. Sau đó, chúng tôi đã hướng dẫn cách tạo lệnh Quản trị Django tùy chỉnh và một nhiệm vụ định kỳ với Celery Beat để chạy lệnh đó tự động.

Tìm kiếm thêm?

  1. Thiết lập Flower để giám sát và điều hành các công việc và công nhân của Celery
  2. Kiểm tra nhiệm vụ Cần tây với cả kiểm tra đơn vị và kiểm tra tích hợp

Lấy mã từ repo .

Nguồn:  https://testdriven.io

#django #celery #docker #redis 

Cách Thiết Lập Django, Celery Và Redis Bằng Docker

Как настроить Django, Celery и Redis с помощью Docker

При создании и масштабировании приложения Django вам неизбежно потребуется периодически и автоматически запускать определенные задачи в фоновом режиме.

Некоторые примеры:

  • Создание периодических отчетов
  • Очистка кеша
  • Отправка пакетных уведомлений по электронной почте
  • Выполнение ночных работ по техническому обслуживанию

Это одна из немногих функций, необходимых для создания и масштабирования веб-приложения, которое не является частью ядра Django. К счастью, Celery предоставляет мощное и достаточно простое в реализации решение под названием Celery Beat.

В следующей статье мы покажем вам, как настроить Django, Celery и Redis с помощью Docker, чтобы периодически запускать пользовательскую команду администратора Django с помощью Celery Beat.

Цели

К концу этого урока вы должны уметь:

  1. Контейнеризируйте Django, Celery и Redis с помощью Docker
  2. Интегрируйте Celery в приложение Django и создавайте задачи
  3. Напишите пользовательскую команду администратора Django
  4. Запланируйте периодическое выполнение пользовательской команды администратора Django через Celery Beat.

Настройка проекта

Клонируйте базовый проект из репозитория django-celery-beat , а затем проверьте базовую ветку:

$ git clone https://github.com/testdrivenio/django-celery-beat --branch base --single-branch
$ cd django-celery-beat

Поскольку нам нужно будет управлять четырьмя процессами в общей сложности (Django, Redis, worker и планировщик), мы будем использовать Docker, чтобы упростить наш рабочий процесс, подключив их так, чтобы их можно было запускать из одного окна терминала с помощью одной команды. .

В корне проекта создайте образы и разверните контейнеры Docker:

$ docker-compose up -d --build

Затем примените миграции:

$ docker-compose exec web python manage.py migrate

После завершения сборки перейдите по адресу http://localhost:1337 , чтобы убедиться, что приложение работает должным образом. Вы должны увидеть следующий текст:

Orders

No orders found!

Прежде чем двигаться дальше, взгляните на структуру проекта:

├── .gitignore
├── docker-compose.yml
└── project
    ├── Dockerfile
    ├── core
    │   ├── __init__.py
    │   ├── asgi.py
    │   ├── settings.py
    │   ├── urls.py
    │   └── wsgi.py
    ├── entrypoint.sh
    ├── manage.py
    ├── orders
    │   ├── __init__.py
    │   ├── admin.py
    │   ├── apps.py
    │   ├── migrations
    │   │   ├── 0001_initial.py
    │   │   └── __init__.py
    │   ├── models.py
    │   ├── tests.py
    │   ├── urls.py
    │   └── views.py
    ├── products.json
    ├── requirements.txt
    └── templates
        └── orders
            └── order_list.html

Хотите узнать, как построить этот проект? Ознакомьтесь с сообщением в блоге Dockerizing Django с Postgres, Gunicorn и Nginx .

Сельдерей и Редис

Теперь нам нужно добавить контейнеры для Celery, Celery Beat и Redis.

Мы начнем с добавления зависимостей в файл requirements.txt :

Django==3.2.4
celery==5.1.2
redis==3.5.3

Затем добавьте следующее в конец файла docker-compose.yml :

redis:
  image: redis:alpine
celery:
  build: ./project
  command: celery -A core worker -l info
  volumes:
    - ./project/:/usr/src/app/
  environment:
    - DEBUG=1
    - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
    - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
  depends_on:
    - redis
celery-beat:
  build: ./project
  command: celery -A core beat -l info
  volumes:
    - ./project/:/usr/src/app/
  environment:
    - DEBUG=1
    - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
    - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
  depends_on:
    - redis

Нам также необходимо обновить depends_onраздел веб-сервиса:

web:
  build: ./project
  command: python manage.py runserver 0.0.0.0:8000
  volumes:
    - ./project/:/usr/src/app/
  ports:
    - 1337:8000
  environment:
    - DEBUG=1
    - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
    - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
  depends_on:
    - redis # NEW

Полный файл docker-compose.yml теперь должен выглядеть так:

version: '3.8'

services:
  web:
    build: ./project
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - ./project/:/usr/src/app/
    ports:
      - 1337:8000
    environment:
      - DEBUG=1
      - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
      - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
    depends_on:
      - redis
  redis:
    image: redis:alpine
  celery:
    build: ./project
    command: celery -A core worker -l info
    volumes:
      - ./project/:/usr/src/app/
    environment:
      - DEBUG=1
      - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
      - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
    depends_on:
      - redis
  celery-beat:
    build: ./project
    command: celery -A core beat -l info
    volumes:
      - ./project/:/usr/src/app/
    environment:
      - DEBUG=1
      - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
      - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
    depends_on:
      - redis

Перед созданием новых контейнеров нам нужно настроить Celery в нашем приложении Django.

Конфигурация сельдерея

Настраивать

В каталоге «core» создайте файл celery.py и добавьте следующий код:

import os

from celery import Celery


os.environ.setdefault("DJANGO_SETTINGS_MODULE", "core.settings")

app = Celery("core")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()

Что тут происходит?

  1. Во-первых, мы устанавливаем значение по умолчанию для DJANGO_SETTINGS_MODULEпеременной среды, чтобы Celery знал, как найти проект Django.
  2. Затем мы создали новый экземпляр Celery с именем coreи присвоили значение переменной с именем app.
  3. Затем мы загрузили значения конфигурации сельдерея из объекта настроек из файла django.conf. Раньше мы namespace="CELERY"предотвращали конфликты с другими настройками Django. CELERY_Другими словами, все параметры конфигурации для Celery должны иметь префикс .
  4. Наконец, app.autodiscover_tasks()указывает Celery искать задачи Celery в приложениях, определенных в файлах settings.INSTALLED_APPS.

Добавьте следующий код в core/__init__.py :

from .celery import app as celery_app

__all__ = ("celery_app",)

Наконец, обновите файл core/settings.py со следующими настройками Celery, чтобы он мог подключаться к Redis:

CELERY_BROKER_URL = "redis://redis:6379"
CELERY_RESULT_BACKEND = "redis://redis:6379"

Создайте новые контейнеры, чтобы убедиться, что все работает:

$ docker-compose up -d --build

Взгляните на журналы для каждой службы, чтобы увидеть, что они готовы, без ошибок:

$ docker-compose logs 'web'
$ docker-compose logs 'celery'
$ docker-compose logs 'celery-beat'
$ docker-compose logs 'redis'

Если все прошло хорошо, у нас теперь есть четыре контейнера, каждый с разными сервисами.

Теперь мы готовы создать пример задачи, чтобы убедиться, что она работает должным образом.

Создать задачу

Создайте новый файл с именем core/tasks.py и добавьте следующий код для примера задачи, которая просто регистрируется в консоли:

from celery import shared_task
from celery.utils.log import get_task_logger


logger = get_task_logger(__name__)


@shared_task
def sample_task():
    logger.info("The sample task just ran.")

Расписание задачи

В конце файла settings.pysample_task добавьте следующий код, чтобы запланировать запуск раз в минуту с помощью Celery Beat:

CELERY_BEAT_SCHEDULE = {
    "sample_task": {
        "task": "core.tasks.sample_task",
        "schedule": crontab(minute="*/1"),
    },
}

Здесь мы определили периодическую задачу, используя настройку CELERY_BEAT_SCHEDULE . Мы дали задаче имя, sample_taskа затем объявили две настройки:

  1. taskобъявляет, какую задачу выполнять.
  2. scheduleустанавливает интервал, в течение которого задача должна выполняться. Это может быть целое число, timedelta или crontab. Мы использовали шаблон crontab для нашей задачи, чтобы заставить ее запускаться каждую минуту. Вы можете найти больше информации о расписании Celery здесь .

Обязательно добавьте импорт:

from celery.schedules import crontab

import core.tasks

Перезапустите контейнер, чтобы получить новые настройки:

$ docker-compose up -d --build

После этого взгляните на журналы сельдерея в контейнере:

$ docker-compose logs -f 'celery'

Вы должны увидеть что-то похожее на:

celery_1  |  -------------- [queues]
celery_1  |                 .> celery           exchange=celery(direct) key=celery
celery_1  |
celery_1  |
celery_1  | [tasks]
celery_1  |   . core.tasks.sample_task

Мы видим, что Celery подхватил нашу тестовую задачу, core.tasks.sample_task.

Каждую минуту вы должны видеть в журнале строку, заканчивающуюся словами «Только что запущен пример задачи»:

celery_1  | [2021-07-01 03:06:00,003: INFO/MainProcess]
              Task core.tasks.sample_task[b8041b6c-bf9b-47ce-ab00-c37c1e837bc7] received
celery_1  | [2021-07-01 03:06:00,004: INFO/ForkPoolWorker-8]
              core.tasks.sample_task[b8041b6c-bf9b-47ce-ab00-c37c1e837bc7]:
              The sample task just ran.

Пользовательская команда администратора Django

Django предоставляет ряд встроенных django-adminкоманд, таких как:

  • migrate
  • startproject
  • startapp
  • dumpdata
  • makemigrations

Наряду со встроенными командами Django также дает нам возможность создавать собственные пользовательские команды :

Пользовательские команды управления особенно полезны для запуска автономных сценариев или сценариев, которые периодически выполняются из crontab UNIX или из панели управления запланированными задачами Windows.

Итак, мы сначала настроим новую команду, а затем воспользуемся Celery Beat для ее автоматического запуска.

Начните с создания нового файла с именем orders/management/commands/my_custom_command.py . Затем добавьте минимальный необходимый код для его запуска:

from django.core.management.base import BaseCommand, CommandError


class Command(BaseCommand):
    help = "A description of the command"

    def handle(self, *args, **options):
        pass

Есть BaseCommandнесколько методов , которые можно переопределить, но требуется только один метод handle. handleявляется точкой входа для пользовательских команд. Другими словами, когда мы запускаем команду, вызывается этот метод.

Для проверки мы обычно просто добавляем оператор быстрой печати. Однако вместо этого рекомендуется использовать stdout.writeдокументацию Django:

Когда вы используете команды управления и хотите предоставить консольный вывод, вы должны писать в self.stdout и self.stderr, а не печатать напрямую в stdout и stderr. Используя эти прокси, становится намного проще протестировать пользовательскую команду. Также обратите внимание, что вам не нужно заканчивать сообщения символом новой строки, он будет добавлен автоматически, если вы не укажете конечный параметр.

Итак, добавляем self.stdout.writeкоманду:

from django.core.management.base import BaseCommand, CommandError


class Command(BaseCommand):
    help = "A description of the command"

    def handle(self, *args, **options):
        self.stdout.write("My sample command just ran.") # NEW

Для проверки из командной строки запустите:

$ docker-compose exec web python manage.py my_custom_command

Тебе следует увидеть:

My sample command just ran.

С этим, давайте свяжем все вместе!

Запланируйте пользовательскую команду с помощью Celery Beat

Теперь, когда мы развернули контейнеры, проверили, что мы можем запланировать периодическое выполнение задачи, и написали пример пользовательской команды администратора Django, пришло время настроить Celery Beat для периодического запуска пользовательской команды.

Настраивать

В проекте у нас есть очень простое приложение под названием заказы. Он содержит две модели Productи Order. Давайте создадим пользовательскую команду, которая отправляет по электронной почте отчет о подтвержденных заказах за день.

Для начала добавим в базу несколько товаров и заказов через фикстуру, включенную в этот проект:

$ docker-compose exec web python manage.py loaddata products.json

Затем добавьте несколько образцов заказов через интерфейс администратора Django. Для этого сначала создайте суперпользователя:

$ docker-compose exec web python manage.py createsuperuser

При появлении запроса введите имя пользователя, адрес электронной почты и пароль. Затем перейдите по адресу http://127.0.0.1:1337/admin в веб-браузере. Войдите в систему с помощью только что созданного суперпользователя и создайте пару заказов. Убедитесь, что хотя бы один confirmed_dateиз них имеет сегодня.

Давайте создадим новую пользовательскую команду для нашего отчета по электронной почте.

Создайте файл с именем orders/management/commands/email_report.py :

from datetime import timedelta, time, datetime

from django.core.mail import mail_admins
from django.core.management import BaseCommand
from django.utils import timezone
from django.utils.timezone import make_aware

from orders.models import Order

today = timezone.now()
tomorrow = today + timedelta(1)
today_start = make_aware(datetime.combine(today, time()))
today_end = make_aware(datetime.combine(tomorrow, time()))


class Command(BaseCommand):
    help = "Send Today's Orders Report to Admins"

    def handle(self, *args, **options):
        orders = Order.objects.filter(confirmed_date__range=(today_start, today_end))

        if orders:
            message = ""

            for order in orders:
                message += f"{order} \n"

            subject = (
                f"Order Report for {today_start.strftime('%Y-%m-%d')} "
                f"to {today_end.strftime('%Y-%m-%d')}"
            )

            mail_admins(subject=subject, message=message, html_message=None)

            self.stdout.write("E-mail Report was sent.")
        else:
            self.stdout.write("No orders confirmed today.")

В коде мы запросили в базе данных заказы с confirmed_dateсегодняшним днем, объединили заказы в одно сообщение для тела электронной почты и использовали встроенную mail_adminsкоманду Django для отправки электронных писем администраторам.

Добавьте фиктивный адрес электронной почты администратора и установите EMAIL_BACKENDдля использования серверную часть консоли , чтобы электронная почта отправлялась на стандартный вывод в файле настроек:

EMAIL_BACKEND = "django.core.mail.backends.console.EmailBackend"
DEFAULT_FROM_EMAIL = "noreply@email.com"
ADMINS = [("testuser", "test.user@email.com"), ]

Теперь можно запустить нашу новую команду из терминала.

$ docker-compose exec web python manage.py email_report

И вывод должен выглядеть примерно так:

Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Subject: [Django] Order Report for 2021-07-01 to 2021-07-02
From: root@localhost
To: test.user@email.com
Date: Thu, 01 Jul 2021 12:15:50 -0000
Message-ID: <162514175053.40.5705892371538583115@5140844ebb15>

Order: 3947963f-1860-44d1-9b9a-4648fed04581 - product: Coffee
Order: ff449e6e-3dfd-48a8-9d5c-79a145d08253 - product: Rice

-------------------------------------------------------------------------------
E-mail Report was sent.

Сельдерей Бит

Теперь нам нужно создать периодическую задачу для ежедневного запуска этой команды.

Добавьте новую задачу в core/tasks.py :

from celery import shared_task
from celery.utils.log import get_task_logger
from django.core.management import call_command # NEW


logger = get_task_logger(__name__)


@shared_task
def sample_task():
    logger.info("The sample task just ran.")


# NEW
@shared_task
def send_email_report():
    call_command("email_report", )

Итак, сначала мы добавили call_commandимпорт, который используется для программного вызова команд django-admin. Затем в новой задаче мы использовали call_commandимя нашей пользовательской команды в качестве аргумента.

Чтобы запланировать эту задачу, откройте файл core/settings.py и обновите CELERY_BEAT_SCHEDULEпараметр, чтобы включить новую задачу:

CELERY_BEAT_SCHEDULE = {
    "sample_task": {
        "task": "core.tasks.sample_task",
        "schedule": crontab(minute="*/1"),
    },
    "send_email_report": {
        "task": "core.tasks.send_email_report",
        "schedule": crontab(hour="*/1"),
    },
}

Здесь мы добавили новую запись в CELERY_BEAT_SCHEDULEвызываемый файл send_email_report. Как и для предыдущей задачи, мы объявили, какую задачу она должна запускать, например, core.tasks.send_email_report--, и использовали шаблон crontab для установки повторения.

Перезапустите контейнеры, чтобы новые настройки стали активными:

$ docker-compose up -d --build

Откройте журналы, связанные с celeryсервисом:

$ docker-compose logs -f 'celery'

Вы должны увидеть в send_email_reportсписке:

celery_1  |  -------------- [queues]
celery_1  |                 .> celery           exchange=celery(direct) key=celery
celery_1  |
celery_1  |
celery_1  | [tasks]
celery_1  |   . core.tasks.sample_task
celery_1  |   . core.tasks.send_email_report

Через минуту или около того вы должны увидеть, что отчет по электронной почте отправлен:

celery_1  | [2021-07-01 12:22:00,071: WARNING/ForkPoolWorker-8] Content-Type: text/plain; charset="utf-8"
celery_1  | MIME-Version: 1.0
celery_1  | Content-Transfer-Encoding: 7bit
celery_1  | Subject: [Django] Order Report for 2021-07-01 to 2021-07-02
celery_1  | From: root@localhost
celery_1  | To: test.user@email.com
celery_1  | Date: Thu, 01 Jul 2021 12:22:00 -0000
celery_1  | Message-ID: <162514212006.17.6080459299558356876@55b9883c5414>
celery_1  |
celery_1  | Order: 3947963f-1860-44d1-9b9a-4648fed04581 - product: Coffee
celery_1  | Order: ff449e6e-3dfd-48a8-9d5c-79a145d08253 - product: Rice
celery_1  |
celery_1  |
celery_1  | [2021-07-01 12:22:00,071: WARNING/ForkPoolWorker-8] -------------------------------------------------------------------------------
celery_1  | [2021-07-01 12:22:00,071: WARNING/ForkPoolWorker-8]
celery_1  |
celery_1  | [2021-07-01 12:22:00,071: WARNING/ForkPoolWorker-8] E-mail Report was sent.

Вывод

В этой статье мы провели вас через настройку контейнеров Docker для Celery, Celery Beat и Redis. Затем мы показали, как создать пользовательскую команду администратора Django и периодическую задачу с Celery Beat для автоматического запуска этой команды.

Ищете больше?

  1. Настройте Flower для мониторинга и управления заданиями и работниками Celery .
  2. Протестируйте задачу Celery как с помощью модульных, так и с интеграционными тестами .

Возьмите код из репозитория .

Источник:  https://testdriven.io

#django #celery #docker #redis 

鈴木  治

鈴木 治

1660942320

如何使用 Docker 設置 Django、Celery 和 Redis

在構建和擴展 Django 應用程序時,您不可避免地需要在後台定期自動運行某些任務。

一些例子:

  • 生成定期報告
  • 清除緩存
  • 發送批量電子郵件通知
  • 運行夜間維護作業

這是構建和擴展不屬於 Django 核心的 Web 應用程序所需的少數功能之一。幸運的是,Celery 提供了一個強大的解決方案,它很容易實現,稱為 Celery Beat。

在下面的文章中,我們將向您展示如何使用 Docker 設置 Django、Celery 和 Redis,以便使用 Celery Beat 定期運行自定義 Django Admin 命令。

目標

在本教程結束時,您應該能夠:

  1. 使用 Docker 將 Django、Celery 和 Redis 容器化
  2. 將 Celery 集成到 Django 應用程序並創建任務
  3. 編寫自定義 Django Admin 命令
  4. 安排一個自定義的 Django Admin 命令通過 Celery Beat 定期運行

項目設置

django-celery-beat存儲庫中克隆基礎項目,然後查看基礎分支:

$ git clone https://github.com/testdrivenio/django-celery-beat --branch base --single-branch
$ cd django-celery-beat

由於我們總共需要管理四個進程(Django、Redis、worker 和調度程序),我們將使用 Docker 通過將它們連接起來來簡化我們的工作流程,以便它們都可以通過一個命令從一個終端窗口運行.

從項目根目錄,創建鏡像並啟動 Docker 容器:

$ docker-compose up -d --build

接下來,應用遷移:

$ docker-compose exec web python manage.py migrate

構建完成後,導航到http://localhost:1337以確保應用程序按預期運行。您應該看到以下文本:

Orders

No orders found!

在繼續之前快速瀏覽一下項目結構:

├── .gitignore
├── docker-compose.yml
└── project
    ├── Dockerfile
    ├── core
    │   ├── __init__.py
    │   ├── asgi.py
    │   ├── settings.py
    │   ├── urls.py
    │   └── wsgi.py
    ├── entrypoint.sh
    ├── manage.py
    ├── orders
    │   ├── __init__.py
    │   ├── admin.py
    │   ├── apps.py
    │   ├── migrations
    │   │   ├── 0001_initial.py
    │   │   └── __init__.py
    │   ├── models.py
    │   ├── tests.py
    │   ├── urls.py
    │   └── views.py
    ├── products.json
    ├── requirements.txt
    └── templates
        └── orders
            └── order_list.html

想學習如何構建這個項目?查看Dockerizing Django with Postgres、Gunicorn 和 Nginx博客文章。

芹菜和 Redis

現在,我們需要為 Celery、Celery Beat 和 Redis 添加容器。

我們首先將依賴項添加到requirements.txt文件中:

Django==3.2.4
celery==5.1.2
redis==3.5.3

接下來,將以下內容添加到docker-compose.yml文件的末尾:

redis:
  image: redis:alpine
celery:
  build: ./project
  command: celery -A core worker -l info
  volumes:
    - ./project/:/usr/src/app/
  environment:
    - DEBUG=1
    - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
    - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
  depends_on:
    - redis
celery-beat:
  build: ./project
  command: celery -A core beat -l info
  volumes:
    - ./project/:/usr/src/app/
  environment:
    - DEBUG=1
    - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
    - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
  depends_on:
    - redis

我們還需要更新 Web 服務的depends_on部分:

web:
  build: ./project
  command: python manage.py runserver 0.0.0.0:8000
  volumes:
    - ./project/:/usr/src/app/
  ports:
    - 1337:8000
  environment:
    - DEBUG=1
    - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
    - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
  depends_on:
    - redis # NEW

完整的 docker-compose.yml文件現在應該如下所示:

version: '3.8'

services:
  web:
    build: ./project
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - ./project/:/usr/src/app/
    ports:
      - 1337:8000
    environment:
      - DEBUG=1
      - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
      - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
    depends_on:
      - redis
  redis:
    image: redis:alpine
  celery:
    build: ./project
    command: celery -A core worker -l info
    volumes:
      - ./project/:/usr/src/app/
    environment:
      - DEBUG=1
      - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
      - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
    depends_on:
      - redis
  celery-beat:
    build: ./project
    command: celery -A core beat -l info
    volumes:
      - ./project/:/usr/src/app/
    environment:
      - DEBUG=1
      - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
      - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
    depends_on:
      - redis

在構建新容器之前,我們需要在 Django 應用程序中配置 Celery。

芹菜配置

設置

在“core”目錄中,創建一個celery.py文件並添加以下代碼:

import os

from celery import Celery


os.environ.setdefault("DJANGO_SETTINGS_MODULE", "core.settings")

app = Celery("core")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()

這裡發生了什麼事?

  1. 首先,我們為環境變量設置一個默認值,DJANGO_SETTINGS_MODULE以便 Celery 知道如何找到 Django 項目。
  2. 接下來,我們創建了一個名為 的新 Celery 實例,core並將值分配給名為 的變量app
  3. 然後我們從設置對像中加載 celery 配置值django.conf。我們曾經namespace="CELERY"防止與其他 Django 設置發生衝突。CELERY_換句話說,Celery 的所有配置設置都必須以 為前綴。
  4. 最後,app.autodiscover_tasks()告訴 Celery 從settings.INSTALLED_APPS.

將以下代碼添加到core/__init__.py

from .celery import app as celery_app

__all__ = ("celery_app",)

最後,使用以下 Celery 設置更新core/settings.py文件,以便它可以連接到 Redis:

CELERY_BROKER_URL = "redis://redis:6379"
CELERY_RESULT_BACKEND = "redis://redis:6379"

構建新容器以確保一切正常:

$ docker-compose up -d --build

查看每個服務的日誌以查看它們是否已準備就緒,沒有錯誤:

$ docker-compose logs 'web'
$ docker-compose logs 'celery'
$ docker-compose logs 'celery-beat'
$ docker-compose logs 'redis'

如果一切順利,我們現在有四個容器,每個容器都有不同的服務。

現在我們已經準備好創建一個示例任務來查看它是否可以正常工作。

創建任務

創建一個名為core/tasks.py的新文件,並為僅記錄到控制台的示例任務添加以下代碼:

from celery import shared_task
from celery.utils.log import get_task_logger


logger = get_task_logger(__name__)


@shared_task
def sample_task():
    logger.info("The sample task just ran.")

安排任務

settings.py文件的末尾,使用 Celery Beat 添加以下代碼以安排sample_task每分鐘運行一次:

CELERY_BEAT_SCHEDULE = {
    "sample_task": {
        "task": "core.tasks.sample_task",
        "schedule": crontab(minute="*/1"),
    },
}

在這裡,我們使用CELERY_BEAT_SCHEDULE設置定義了一個週期性任務。我們為任務命名,sample_task然後聲明了兩個設置:

  1. task聲明要運行的任務。
  2. schedule設置任務應該運行的時間間隔。這可以是整數、timedelta 或 crontab。我們為我們的任務使用了一個 crontab 模式來告訴它每分鐘運行一次。您可以在此處找到有關 Celery 日程安排的更多信息。

確保添加導入:

from celery.schedules import crontab

import core.tasks

重新啟動容器以引入新設置:

$ docker-compose up -d --build

完成後,查看容器中的 celery 日誌:

$ docker-compose logs -f 'celery'

您應該會看到類似於以下內容的內容:

celery_1  |  -------------- [queues]
celery_1  |                 .> celery           exchange=celery(direct) key=celery
celery_1  |
celery_1  |
celery_1  | [tasks]
celery_1  |   . core.tasks.sample_task

我們可以看到 Celery 拿起了我們的示例任務,core.tasks.sample_task.

每分鐘您都應該在日誌中看到以“剛剛運行的示例任務”結尾的一行:

celery_1  | [2021-07-01 03:06:00,003: INFO/MainProcess]
              Task core.tasks.sample_task[b8041b6c-bf9b-47ce-ab00-c37c1e837bc7] received
celery_1  | [2021-07-01 03:06:00,004: INFO/ForkPoolWorker-8]
              core.tasks.sample_task[b8041b6c-bf9b-47ce-ab00-c37c1e837bc7]:
              The sample task just ran.

自定義 Django 管理命令

Django 提供了許多內置django-admin命令,例如:

  • migrate
  • startproject
  • startapp
  • dumpdata
  • makemigrations

除了內置命令外,Django 還為我們提供了創建自己的自定義命令的選項:

自定義管理命令對於運行獨立腳本或從 UNIX crontab 或 Windows 計劃任務控制面板定期執行的腳本特別有用。

因此,我們將首先配置一個新命令,然後使用 Celery Beat 自動運行它。

首先創建一個名為orders/management/commands/my_custom_command.py的新文件。然後,添加運行所需的最少代碼:

from django.core.management.base import BaseCommand, CommandError


class Command(BaseCommand):
    help = "A description of the command"

    def handle(self, *args, **options):
        pass

BaseCommand一些方法可以被覆蓋,但唯一需要的方法是handle. handle是自定義命令的入口點。換句話說,當我們運行命令時,就會調用這個方法。

為了測試,我們通常只添加一個快速打印語句。但是,建議stdout.write按照 Django 文檔使用:

當您使用管理命令並希望提供控制台輸出時,您應該寫入 self.stdout 和 self.stderr,而不是直接打印到 stdout 和 stderr。通過使用這些代理,測試您的自定義命令變得更加容易。另請注意,您不需要以換行符結束消息,它將自動添加,除非您指定結束參數。

所以,添加一個self.stdout.write命令:

from django.core.management.base import BaseCommand, CommandError


class Command(BaseCommand):
    help = "A description of the command"

    def handle(self, *args, **options):
        self.stdout.write("My sample command just ran.") # NEW

要進行測試,請從命令行運行:

$ docker-compose exec web python manage.py my_custom_command

你應該看到:

My sample command just ran.

有了這個,讓我們把一切聯繫在一起!

使用 Celery Beat 安排自定義命令

現在我們已經啟動了容器,測試了我們可以安排任務定期運行,並編寫了自定義 Django Admin 示例命令,是時候配置 Celery Beat 以定期運行自定義命令了。

設置

在項目中,我們有一個非常基本的應用程序,稱為訂單。它包含兩個模型,ProductOrder。讓我們創建一個自定義命令,發送當天確認訂單的電子郵件報告。

首先,我們將通過此項目中包含的夾具將一些產品和訂單添加到數據庫中:

$ docker-compose exec web python manage.py loaddata products.json

接下來,通過 Django Admin 界面添加一些示例訂單。為此,首先創建一個超級用戶:

$ docker-compose exec web python manage.py createsuperuser

出現提示時填寫用戶名、電子郵件和密碼。然後在 Web 瀏覽器中導航到http://127.0.0.1:1337/admin 。使用您剛剛創建的超級用戶登錄並創建幾個訂單。確保至少有一個confirmed_date今天。

讓我們為我們的電子郵件報告創建一個新的自定義命令。

創建一個名為orders/management/commands/email_report.py的文件:

from datetime import timedelta, time, datetime

from django.core.mail import mail_admins
from django.core.management import BaseCommand
from django.utils import timezone
from django.utils.timezone import make_aware

from orders.models import Order

today = timezone.now()
tomorrow = today + timedelta(1)
today_start = make_aware(datetime.combine(today, time()))
today_end = make_aware(datetime.combine(tomorrow, time()))


class Command(BaseCommand):
    help = "Send Today's Orders Report to Admins"

    def handle(self, *args, **options):
        orders = Order.objects.filter(confirmed_date__range=(today_start, today_end))

        if orders:
            message = ""

            for order in orders:
                message += f"{order} \n"

            subject = (
                f"Order Report for {today_start.strftime('%Y-%m-%d')} "
                f"to {today_end.strftime('%Y-%m-%d')}"
            )

            mail_admins(subject=subject, message=message, html_message=None)

            self.stdout.write("E-mail Report was sent.")
        else:
            self.stdout.write("No orders confirmed today.")

在代碼中,我們使用confirmed_date今天的 a 查詢數據庫中的訂單,將訂單組合成單個消息作為電子郵件正文,並使用 Django 的內置mail_admins命令將電子郵件發送給管理員。

添加一個虛擬管理員電子郵件並將其設置EMAIL_BACKEND為使用控制台後端,以便在設置文件中將電子郵件發送到標準輸出:

EMAIL_BACKEND = "django.core.mail.backends.console.EmailBackend"
DEFAULT_FROM_EMAIL = "noreply@email.com"
ADMINS = [("testuser", "test.user@email.com"), ]

現在應該可以從終端運行我們的新命令了。

$ docker-compose exec web python manage.py email_report

輸出應該類似於:

Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Subject: [Django] Order Report for 2021-07-01 to 2021-07-02
From: root@localhost
To: test.user@email.com
Date: Thu, 01 Jul 2021 12:15:50 -0000
Message-ID: <162514175053.40.5705892371538583115@5140844ebb15>

Order: 3947963f-1860-44d1-9b9a-4648fed04581 - product: Coffee
Order: ff449e6e-3dfd-48a8-9d5c-79a145d08253 - product: Rice

-------------------------------------------------------------------------------
E-mail Report was sent.

芹菜節拍

我們現在需要創建一個定期任務來每天運行這個命令。

向core/tasks.py添加一個新任務:

from celery import shared_task
from celery.utils.log import get_task_logger
from django.core.management import call_command # NEW


logger = get_task_logger(__name__)


@shared_task
def sample_task():
    logger.info("The sample task just ran.")


# NEW
@shared_task
def send_email_report():
    call_command("email_report", )

因此,首先我們添加了一個call_command導入,用於以編程方式調用 django-admin 命令。在新任務中,我們使用call_command自定義命令的名稱作為參數。

要安排此任務,請打開core/settings.py文件,並更新CELERY_BEAT_SCHEDULE設置以包含新任務:

CELERY_BEAT_SCHEDULE = {
    "sample_task": {
        "task": "core.tasks.sample_task",
        "schedule": crontab(minute="*/1"),
    },
    "send_email_report": {
        "task": "core.tasks.send_email_report",
        "schedule": crontab(hour="*/1"),
    },
}

在這裡,我們向被CELERY_BEAT_SCHEDULE調用的send_email_report. 正如我們對之前的任務所做的那樣,我們聲明了它應該運行哪個任務——例如core.tasks.send_email_report——並使用 crontab 模式來設置循環。

重新啟動容器以確保新設置生效:

$ docker-compose up -d --build

打開與服務關聯的日誌celery

$ docker-compose logs -f 'celery'

您應該看到send_email_report列出的:

celery_1  |  -------------- [queues]
celery_1  |                 .> celery           exchange=celery(direct) key=celery
celery_1  |
celery_1  |
celery_1  | [tasks]
celery_1  |   . core.tasks.sample_task
celery_1  |   . core.tasks.send_email_report

大約一分鐘後,您應該會看到電子郵件報告已發送:

celery_1  | [2021-07-01 12:22:00,071: WARNING/ForkPoolWorker-8] Content-Type: text/plain; charset="utf-8"
celery_1  | MIME-Version: 1.0
celery_1  | Content-Transfer-Encoding: 7bit
celery_1  | Subject: [Django] Order Report for 2021-07-01 to 2021-07-02
celery_1  | From: root@localhost
celery_1  | To: test.user@email.com
celery_1  | Date: Thu, 01 Jul 2021 12:22:00 -0000
celery_1  | Message-ID: <162514212006.17.6080459299558356876@55b9883c5414>
celery_1  |
celery_1  | Order: 3947963f-1860-44d1-9b9a-4648fed04581 - product: Coffee
celery_1  | Order: ff449e6e-3dfd-48a8-9d5c-79a145d08253 - product: Rice
celery_1  |
celery_1  |
celery_1  | [2021-07-01 12:22:00,071: WARNING/ForkPoolWorker-8] -------------------------------------------------------------------------------
celery_1  | [2021-07-01 12:22:00,071: WARNING/ForkPoolWorker-8]
celery_1  |
celery_1  | [2021-07-01 12:22:00,071: WARNING/ForkPoolWorker-8] E-mail Report was sent.

結論

在本文中,我們指導您為 Celery、Celery Beat 和 Redis 設置 Docker 容器。然後,我們展示瞭如何使用 Celery Beat 創建自定義 Django Admin 命令和定期任務以自動運行該命令。

尋找更多?

  1. 設置Flower以監控和管理 Celery 工作和工人
  2. 使用單元測試和集成測試來測試 Celery 任務

repo中獲取代碼。

來源:  https ://testdriven.io

#django #celery #docker #redis 

如何使用 Docker 設置 Django、Celery 和 Redis
Jarrod  Douglas

Jarrod Douglas

1660934940

Comment Configurer Django, Celery et Redis avec Docker

Lorsque vous créez et mettez à l'échelle une application Django, vous devrez inévitablement exécuter certaines tâches périodiquement et automatiquement en arrière-plan.

Quelques exemples:

  • Génération de rapports périodiques
  • Vider le cache
  • Envoi de notifications par e-mail par lots
  • Exécution de tâches de maintenance nocturnes

C'est l'une des rares fonctionnalités requises pour créer et faire évoluer une application Web qui ne fait pas partie du cœur de Django. Heureusement, Celery fournit une solution puissante et assez facile à mettre en œuvre appelée Celery Beat.

Dans l'article suivant, nous allons vous montrer comment configurer Django, Celery et Redis avec Docker afin d'exécuter régulièrement une commande Django Admin personnalisée avec Celery Beat.

Objectifs

À la fin de ce didacticiel, vous devriez être en mesure de :

  1. Conteneurisez Django, Celery et Redis avec Docker
  2. Intégrez Celery dans une application Django et créez des tâches
  3. Écrire une commande Django Admin personnalisée
  4. Planifiez une commande Django Admin personnalisée à exécuter périodiquement via Celery Beat

Configuration du projet

Clonez le projet de base à partir du référentiel django-celery-beat , puis consultez la branche de base :

$ git clone https://github.com/testdrivenio/django-celery-beat --branch base --single-branch
$ cd django-celery-beat

Étant donné que nous devrons gérer quatre processus au total (Django, Redis, travailleur et planificateur), nous utiliserons Docker pour simplifier notre flux de travail en les connectant afin qu'ils puissent tous être exécutés à partir d'une fenêtre de terminal avec une seule commande .

À partir de la racine du projet, créez les images et lancez les conteneurs Docker :

$ docker-compose up -d --build

Ensuite, appliquez les migrations :

$ docker-compose exec web python manage.py migrate

Une fois la construction terminée, accédez à http://localhost:1337 pour vous assurer que l'application fonctionne comme prévu. Vous devriez voir le texte suivant :

Orders

No orders found!

Jetez un coup d'œil à la structure du projet avant de continuer :

├── .gitignore
├── docker-compose.yml
└── project
    ├── Dockerfile
    ├── core
    │   ├── __init__.py
    │   ├── asgi.py
    │   ├── settings.py
    │   ├── urls.py
    │   └── wsgi.py
    ├── entrypoint.sh
    ├── manage.py
    ├── orders
    │   ├── __init__.py
    │   ├── admin.py
    │   ├── apps.py
    │   ├── migrations
    │   │   ├── 0001_initial.py
    │   │   └── __init__.py
    │   ├── models.py
    │   ├── tests.py
    │   ├── urls.py
    │   └── views.py
    ├── products.json
    ├── requirements.txt
    └── templates
        └── orders
            └── order_list.html

Vous voulez apprendre à construire ce projet ? Consultez l' article de blog Dockerizing Django avec Postgres, Gunicorn et Nginx .

Céleri et Redis

Maintenant, nous devons ajouter des conteneurs pour Celery, Celery Beat et Redis.

Nous allons commencer par ajouter les dépendances au fichier requirements.txt :

Django==3.2.4
celery==5.1.2
redis==3.5.3

Ensuite, ajoutez ce qui suit à la fin du fichier docker-compose.yml :

redis:
  image: redis:alpine
celery:
  build: ./project
  command: celery -A core worker -l info
  volumes:
    - ./project/:/usr/src/app/
  environment:
    - DEBUG=1
    - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
    - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
  depends_on:
    - redis
celery-beat:
  build: ./project
  command: celery -A core beat -l info
  volumes:
    - ./project/:/usr/src/app/
  environment:
    - DEBUG=1
    - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
    - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
  depends_on:
    - redis

Nous devons également mettre à jour la section du service Webdepends_on :

web:
  build: ./project
  command: python manage.py runserver 0.0.0.0:8000
  volumes:
    - ./project/:/usr/src/app/
  ports:
    - 1337:8000
  environment:
    - DEBUG=1
    - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
    - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
  depends_on:
    - redis # NEW

Le fichier docker-compose.yml complet devrait maintenant ressembler à ceci :

version: '3.8'

services:
  web:
    build: ./project
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - ./project/:/usr/src/app/
    ports:
      - 1337:8000
    environment:
      - DEBUG=1
      - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
      - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
    depends_on:
      - redis
  redis:
    image: redis:alpine
  celery:
    build: ./project
    command: celery -A core worker -l info
    volumes:
      - ./project/:/usr/src/app/
    environment:
      - DEBUG=1
      - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
      - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
    depends_on:
      - redis
  celery-beat:
    build: ./project
    command: celery -A core beat -l info
    volumes:
      - ./project/:/usr/src/app/
    environment:
      - DEBUG=1
      - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
      - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
    depends_on:
      - redis

Avant de créer les nouveaux conteneurs, nous devons configurer Celery dans notre application Django.

Configuration Céleri

Installer

Dans le répertoire "core", créez un fichier celery.py et ajoutez le code suivant :

import os

from celery import Celery


os.environ.setdefault("DJANGO_SETTINGS_MODULE", "core.settings")

app = Celery("core")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()

Qu'est-ce qu'il se passe ici?

  1. Tout d'abord, nous définissons une valeur par défaut pour la DJANGO_SETTINGS_MODULEvariable d'environnement afin que Celery sache comment trouver le projet Django.
  2. Ensuite, nous avons créé une nouvelle instance Celery, avec le nom core, et attribué la valeur à une variable appelée app.
  3. Nous avons ensuite chargé les valeurs de configuration du céleri à partir de l'objet de paramètres de django.conf. Nous avions l'habitude namespace="CELERY"d'empêcher les conflits avec d'autres paramètres de Django. CELERY_En d'autres termes, tous les paramètres de configuration de Celery doivent être précédés de .
  4. Enfin, app.autodiscover_tasks()indique à Celery de rechercher des tâches Celery à partir d'applications définies dans settings.INSTALLED_APPS.

Ajoutez le code suivant à core/__init__.py :

from .celery import app as celery_app

__all__ = ("celery_app",)

Enfin, mettez à jour le fichier core/settings.py avec les paramètres Celery suivants afin qu'il puisse se connecter à Redis :

CELERY_BROKER_URL = "redis://redis:6379"
CELERY_RESULT_BACKEND = "redis://redis:6379"

Construisez les nouveaux conteneurs pour vous assurer que tout fonctionne :

$ docker-compose up -d --build

Jetez un œil aux journaux de chaque service pour voir qu'ils sont prêts, sans erreur :

$ docker-compose logs 'web'
$ docker-compose logs 'celery'
$ docker-compose logs 'celery-beat'
$ docker-compose logs 'redis'

Si tout s'est bien passé, nous avons maintenant quatre conteneurs, chacun avec des services différents.

Nous sommes maintenant prêts à créer un exemple de tâche pour voir si cela fonctionne comme il se doit.

Créer une tâche

Créez un nouveau fichier appelé core/tasks.py et ajoutez le code suivant pour un exemple de tâche qui se connecte simplement à la console :

from celery import shared_task
from celery.utils.log import get_task_logger


logger = get_task_logger(__name__)


@shared_task
def sample_task():
    logger.info("The sample task just ran.")

Planifier la tâche

À la fin de votre fichier settings.py , ajoutez le code suivant pour planifier sample_taskune exécution une fois par minute, en utilisant Celery Beat :

CELERY_BEAT_SCHEDULE = {
    "sample_task": {
        "task": "core.tasks.sample_task",
        "schedule": crontab(minute="*/1"),
    },
}

Ici, nous avons défini une tâche périodique à l'aide du paramètre CELERY_BEAT_SCHEDULE . Nous avons donné un nom à la tâche, sample_task, puis avons déclaré deux paramètres :

  1. taskdéclare quelle tâche exécuter.
  2. scheduledéfinit l'intervalle d'exécution de la tâche. Il peut s'agir d'un entier, d'un timedelta ou d'un crontab. Nous avons utilisé un modèle crontab pour notre tâche pour lui dire de s'exécuter une fois par minute. Vous pouvez trouver plus d'informations sur la programmation de Celery ici .

Assurez-vous d'ajouter les importations :

from celery.schedules import crontab

import core.tasks

Redémarrez le conteneur pour extraire les nouveaux paramètres :

$ docker-compose up -d --build

Une fois cela fait, jetez un œil aux bûches de céleri dans le conteneur :

$ docker-compose logs -f 'celery'

Vous devriez voir quelque chose de similaire à :

celery_1  |  -------------- [queues]
celery_1  |                 .> celery           exchange=celery(direct) key=celery
celery_1  |
celery_1  |
celery_1  | [tasks]
celery_1  |   . core.tasks.sample_task

Nous pouvons voir que Celery a récupéré notre exemple de tâche, core.tasks.sample_task.

Chaque minute, vous devriez voir une ligne dans le journal qui se termine par "L'exemple de tâche vient d'être exécuté." :

celery_1  | [2021-07-01 03:06:00,003: INFO/MainProcess]
              Task core.tasks.sample_task[b8041b6c-bf9b-47ce-ab00-c37c1e837bc7] received
celery_1  | [2021-07-01 03:06:00,004: INFO/ForkPoolWorker-8]
              core.tasks.sample_task[b8041b6c-bf9b-47ce-ab00-c37c1e837bc7]:
              The sample task just ran.

Commande d'administration Django personnalisée

Django fournit un certain nombre de django-admincommandes intégrées, telles que :

  • migrate
  • startproject
  • startapp
  • dumpdata
  • makemigrations

En plus des commandes intégrées, Django nous donne également la possibilité de créer nos propres commandes personnalisées :

Les commandes de gestion personnalisées sont particulièrement utiles pour l'exécution de scripts autonomes ou pour les scripts exécutés périodiquement à partir de la crontab UNIX ou du panneau de configuration des tâches planifiées de Windows.

Donc, nous allons d'abord configurer une nouvelle commande, puis utiliser Celery Beat pour l'exécuter automatiquement.

Commencez par créer un nouveau fichier appelé orders/management/commands/my_custom_command.py . Ensuite, ajoutez le code minimal requis pour qu'il s'exécute :

from django.core.management.base import BaseCommand, CommandError


class Command(BaseCommand):
    help = "A description of the command"

    def handle(self, *args, **options):
        pass

Le BaseCommanda quelques méthodes qui peuvent être remplacées, mais la seule méthode requise est handle. handleest le point d'entrée des commandes personnalisées. En d'autres termes, lorsque nous exécutons la commande, cette méthode est appelée.

Pour tester, nous ajoutons normalement une déclaration d'impression rapide. Cependant, il est recommandé d'utiliser à stdout.writela place selon la documentation de Django :

Lorsque vous utilisez des commandes de gestion et souhaitez fournir une sortie de console, vous devez écrire dans self.stdout et self.stderr, au lieu d'imprimer directement dans stdout et stderr. En utilisant ces proxys, il devient beaucoup plus facile de tester votre commande personnalisée. Notez également que vous n'avez pas besoin de terminer les messages avec un caractère de nouvelle ligne, il sera ajouté automatiquement, sauf si vous spécifiez le paramètre de fin.

Alors, ajoutez une self.stdout.writecommande :

from django.core.management.base import BaseCommand, CommandError


class Command(BaseCommand):
    help = "A description of the command"

    def handle(self, *args, **options):
        self.stdout.write("My sample command just ran.") # NEW

Pour tester, depuis la ligne de commande, exécutez :

$ docker-compose exec web python manage.py my_custom_command

Tu devrais voir:

My sample command just ran.

Sur ce, lions le tout ensemble !

Planifier une commande personnalisée avec Celery Beat

Maintenant que nous avons lancé les conteneurs, testé que nous pouvions programmer une tâche à exécuter périodiquement et écrit un exemple de commande personnalisée Django Admin, il est temps de configurer Celery Beat pour exécuter la commande personnalisée périodiquement.

Installer

Dans le projet, nous avons une application très basique appelée commandes. Il contient deux modèles, Productet Order. Créons une commande personnalisée qui envoie un rapport par e-mail des commandes confirmées de la journée.

Pour commencer, nous allons ajouter quelques produits et commandes à la base de données via le luminaire inclus dans ce projet :

$ docker-compose exec web python manage.py loaddata products.json

Ensuite, ajoutez quelques exemples de commandes via l'interface Django Admin. Pour ce faire, créez d'abord un superutilisateur :

$ docker-compose exec web python manage.py createsuperuser

Remplissez le nom d'utilisateur, l'e-mail et le mot de passe lorsque vous y êtes invité. Accédez ensuite à http://127.0.0.1:1337/admin dans votre navigateur Web. Connectez-vous avec le superutilisateur que vous venez de créer et créez quelques commandes. Assurez-vous qu'au moins un a un confirmed_dated'aujourd'hui.

Créons une nouvelle commande personnalisée pour notre rapport par e-mail.

Créez un fichier nommé orders/management/commands/email_report.py :

from datetime import timedelta, time, datetime

from django.core.mail import mail_admins
from django.core.management import BaseCommand
from django.utils import timezone
from django.utils.timezone import make_aware

from orders.models import Order

today = timezone.now()
tomorrow = today + timedelta(1)
today_start = make_aware(datetime.combine(today, time()))
today_end = make_aware(datetime.combine(tomorrow, time()))


class Command(BaseCommand):
    help = "Send Today's Orders Report to Admins"

    def handle(self, *args, **options):
        orders = Order.objects.filter(confirmed_date__range=(today_start, today_end))

        if orders:
            message = ""

            for order in orders:
                message += f"{order} \n"

            subject = (
                f"Order Report for {today_start.strftime('%Y-%m-%d')} "
                f"to {today_end.strftime('%Y-%m-%d')}"
            )

            mail_admins(subject=subject, message=message, html_message=None)

            self.stdout.write("E-mail Report was sent.")
        else:
            self.stdout.write("No orders confirmed today.")

Dans le code, nous avons interrogé la base de données pour les commandes avec un confirmed_dated'aujourd'hui, combiné les commandes en un seul message pour le corps de l'e-mail et utilisé la mail_adminscommande intégrée de Django pour envoyer les e-mails aux administrateurs.

Ajoutez un e-mail d'administrateur factice et configurez le EMAIL_BACKENDpour qu'il utilise le backend de la console afin que l'e-mail soit envoyé à stdout, dans le fichier de paramètres :

EMAIL_BACKEND = "django.core.mail.backends.console.EmailBackend"
DEFAULT_FROM_EMAIL = "noreply@email.com"
ADMINS = [("testuser", "test.user@email.com"), ]

Il devrait maintenant être possible d'exécuter notre nouvelle commande depuis le terminal.

$ docker-compose exec web python manage.py email_report

Et la sortie devrait ressembler à ceci :

Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Subject: [Django] Order Report for 2021-07-01 to 2021-07-02
From: root@localhost
To: test.user@email.com
Date: Thu, 01 Jul 2021 12:15:50 -0000
Message-ID: <162514175053.40.5705892371538583115@5140844ebb15>

Order: 3947963f-1860-44d1-9b9a-4648fed04581 - product: Coffee
Order: ff449e6e-3dfd-48a8-9d5c-79a145d08253 - product: Rice

-------------------------------------------------------------------------------
E-mail Report was sent.

Battement de céleri

Nous devons maintenant créer une tâche périodique pour exécuter cette commande quotidiennement.

Ajoutez une nouvelle tâche à core/tasks.py :

from celery import shared_task
from celery.utils.log import get_task_logger
from django.core.management import call_command # NEW


logger = get_task_logger(__name__)


@shared_task
def sample_task():
    logger.info("The sample task just ran.")


# NEW
@shared_task
def send_email_report():
    call_command("email_report", )

Donc, nous avons d'abord ajouté une call_commandimportation, qui est utilisée pour appeler par programme les commandes django-admin. Dans la nouvelle tâche, nous avons ensuite utilisé le call_commandavec le nom de notre commande personnalisée comme argument.

Pour planifier cette tâche, ouvrez le fichier core/settings.py et mettez à jour le CELERY_BEAT_SCHEDULEparamètre pour inclure la nouvelle tâche :

CELERY_BEAT_SCHEDULE = {
    "sample_task": {
        "task": "core.tasks.sample_task",
        "schedule": crontab(minute="*/1"),
    },
    "send_email_report": {
        "task": "core.tasks.send_email_report",
        "schedule": crontab(hour="*/1"),
    },
}

Ici, nous avons ajouté une nouvelle entrée au fichier CELERY_BEAT_SCHEDULEappelé send_email_report. Comme nous l'avons fait pour notre tâche précédente, nous avons déclaré la tâche à exécuter -- par exemple, core.tasks.send_email_report-- et utilisé un modèle crontab pour définir la récurrence.

Redémarrez les conteneurs pour vous assurer que les nouveaux paramètres deviennent actifs :

$ docker-compose up -d --build

Ouvrez les journaux associés au celeryservice :

$ docker-compose logs -f 'celery'

Vous devriez voir la send_email_reportliste :

celery_1  |  -------------- [queues]
celery_1  |                 .> celery           exchange=celery(direct) key=celery
celery_1  |
celery_1  |
celery_1  | [tasks]
celery_1  |   . core.tasks.sample_task
celery_1  |   . core.tasks.send_email_report

Environ une minute plus tard, vous devriez voir que le rapport par e-mail est envoyé :

celery_1  | [2021-07-01 12:22:00,071: WARNING/ForkPoolWorker-8] Content-Type: text/plain; charset="utf-8"
celery_1  | MIME-Version: 1.0
celery_1  | Content-Transfer-Encoding: 7bit
celery_1  | Subject: [Django] Order Report for 2021-07-01 to 2021-07-02
celery_1  | From: root@localhost
celery_1  | To: test.user@email.com
celery_1  | Date: Thu, 01 Jul 2021 12:22:00 -0000
celery_1  | Message-ID: <162514212006.17.6080459299558356876@55b9883c5414>
celery_1  |
celery_1  | Order: 3947963f-1860-44d1-9b9a-4648fed04581 - product: Coffee
celery_1  | Order: ff449e6e-3dfd-48a8-9d5c-79a145d08253 - product: Rice
celery_1  |
celery_1  |
celery_1  | [2021-07-01 12:22:00,071: WARNING/ForkPoolWorker-8] -------------------------------------------------------------------------------
celery_1  | [2021-07-01 12:22:00,071: WARNING/ForkPoolWorker-8]
celery_1  |
celery_1  | [2021-07-01 12:22:00,071: WARNING/ForkPoolWorker-8] E-mail Report was sent.

Conclusion

Dans cet article, nous vous avons guidé dans la configuration des conteneurs Docker pour Celery, Celery Beat et Redis. Nous avons ensuite montré comment créer une commande Django Admin personnalisée et une tâche périodique avec Celery Beat pour exécuter cette commande automatiquement.

Vous en voulez plus ?

  1. Configurer Flower pour surveiller et administrer les tâches et les travailleurs de Celery
  2. Tester une tâche Celery avec des tests unitaires et d'intégration

Récupérez le code du repo .

Source :  https://testdrive.io

#django #celery #docker #redis 

Comment Configurer Django, Celery et Redis avec Docker
Wayne  Richards

Wayne Richards

1660927680

Cómo Configurar Django, Celery Y Redis Con Docker

A medida que crea y escala una aplicación de Django, inevitablemente necesitará ejecutar ciertas tareas de forma periódica y automática en segundo plano.

Algunos ejemplos:

  • Generación de informes periódicos
  • Borrando caché
  • Envío de notificaciones por correo electrónico por lotes
  • Ejecución de trabajos de mantenimiento nocturno

Esta es una de las pocas funciones necesarias para crear y escalar una aplicación web que no forma parte del núcleo de Django. Afortunadamente, Celery proporciona una solución poderosa, que es bastante fácil de implementar, llamada Celery Beat.

En el siguiente artículo, le mostraremos cómo configurar Django, Celery y Redis con Docker para ejecutar periódicamente un comando personalizado de administración de Django con Celery Beat.

Objetivos

Al final de este tutorial, debería ser capaz de:

  1. Contenga Django, Celery y Redis con Docker
  2. Integre Celery en una aplicación de Django y cree tareas
  3. Escribir un comando de administración de Django personalizado
  4. Programe un comando de administración de Django personalizado para que se ejecute periódicamente a través de Celery Beat

Configuración del proyecto

Clone el proyecto base del repositorio django-celery-beat y luego revise la rama base :

$ git clone https://github.com/testdrivenio/django-celery-beat --branch base --single-branch
$ cd django-celery-beat

Dado que necesitaremos administrar cuatro procesos en total (Django, Redis, Worker y Scheduler), usaremos Docker para simplificar nuestro flujo de trabajo conectándolos para que todos puedan ejecutarse desde una ventana de terminal con un solo comando. .

Desde la raíz del proyecto, cree las imágenes y active los contenedores de Docker:

$ docker-compose up -d --build

A continuación, aplique las migraciones:

$ docker-compose exec web python manage.py migrate

Una vez que se complete la compilación, vaya a http://localhost:1337 para asegurarse de que la aplicación funcione como se espera. Deberías ver el siguiente texto:

Orders

No orders found!

Eche un vistazo rápido a la estructura del proyecto antes de continuar:

├── .gitignore
├── docker-compose.yml
└── project
    ├── Dockerfile
    ├── core
    │   ├── __init__.py
    │   ├── asgi.py
    │   ├── settings.py
    │   ├── urls.py
    │   └── wsgi.py
    ├── entrypoint.sh
    ├── manage.py
    ├── orders
    │   ├── __init__.py
    │   ├── admin.py
    │   ├── apps.py
    │   ├── migrations
    │   │   ├── 0001_initial.py
    │   │   └── __init__.py
    │   ├── models.py
    │   ├── tests.py
    │   ├── urls.py
    │   └── views.py
    ├── products.json
    ├── requirements.txt
    └── templates
        └── orders
            └── order_list.html

¿Quieres aprender a construir este proyecto? Consulte la publicación de blog Dockerizing Django con Postgres, Gunicorn y Nginx .

Apio y Redis

Ahora, necesitamos agregar contenedores para Celery, Celery Beat y Redis.

Comenzaremos agregando las dependencias al archivo requirements.txt :

Django==3.2.4
celery==5.1.2
redis==3.5.3

A continuación, agregue lo siguiente al final del archivo docker-compose.yml :

redis:
  image: redis:alpine
celery:
  build: ./project
  command: celery -A core worker -l info
  volumes:
    - ./project/:/usr/src/app/
  environment:
    - DEBUG=1
    - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
    - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
  depends_on:
    - redis
celery-beat:
  build: ./project
  command: celery -A core beat -l info
  volumes:
    - ./project/:/usr/src/app/
  environment:
    - DEBUG=1
    - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
    - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
  depends_on:
    - redis

También necesitamos actualizar la depends_onsección del servicio web:

web:
  build: ./project
  command: python manage.py runserver 0.0.0.0:8000
  volumes:
    - ./project/:/usr/src/app/
  ports:
    - 1337:8000
  environment:
    - DEBUG=1
    - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
    - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
  depends_on:
    - redis # NEW

El archivo completo docker-compose.yml ahora debería verse así:

version: '3.8'

services:
  web:
    build: ./project
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - ./project/:/usr/src/app/
    ports:
      - 1337:8000
    environment:
      - DEBUG=1
      - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
      - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
    depends_on:
      - redis
  redis:
    image: redis:alpine
  celery:
    build: ./project
    command: celery -A core worker -l info
    volumes:
      - ./project/:/usr/src/app/
    environment:
      - DEBUG=1
      - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
      - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
    depends_on:
      - redis
  celery-beat:
    build: ./project
    command: celery -A core beat -l info
    volumes:
      - ./project/:/usr/src/app/
    environment:
      - DEBUG=1
      - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
      - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
    depends_on:
      - redis

Antes de construir los nuevos contenedores, debemos configurar Celery en nuestra aplicación Django.

Configuración de apio

Configuración

En el directorio "núcleo", cree un archivo apio.py y agregue el siguiente código:

import os

from celery import Celery


os.environ.setdefault("DJANGO_SETTINGS_MODULE", "core.settings")

app = Celery("core")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()

¿Que esta pasando aqui?

  1. Primero, establecemos un valor predeterminado para la DJANGO_SETTINGS_MODULEvariable de entorno para que Celery sepa cómo encontrar el proyecto Django.
  2. A continuación, creamos una nueva instancia de Celery, con el nombre corey le asignamos el valor a una variable llamada app.
  3. Luego cargamos los valores de configuración de apio desde el objeto de configuración de django.conf. Solíamos namespace="CELERY"evitar conflictos con otras configuraciones de Django. Todos los ajustes de configuración para Celery deben tener el prefijo CELERY_, en otras palabras.
  4. Finalmente, app.autodiscover_tasks()le dice a Celery que busque las tareas de Celery desde las aplicaciones definidas en settings.INSTALLED_APPS.

Agrega el siguiente código a core/__init__.py :

from .celery import app as celery_app

__all__ = ("celery_app",)

Por último, actualice el archivo core/settings.py con la siguiente configuración de Celery para que pueda conectarse a Redis:

CELERY_BROKER_URL = "redis://redis:6379"
CELERY_RESULT_BACKEND = "redis://redis:6379"

Construya los nuevos contenedores para asegurarse de que todo funcione:

$ docker-compose up -d --build

Eche un vistazo a los registros de cada servicio para ver que están listos, sin errores:

$ docker-compose logs 'web'
$ docker-compose logs 'celery'
$ docker-compose logs 'celery-beat'
$ docker-compose logs 'redis'

Si todo salió bien, ahora tenemos cuatro contenedores, cada uno con diferentes servicios.

Ahora estamos listos para crear una tarea de muestra para ver que funciona como debería.

Crear una tarea

Cree un nuevo archivo llamado core/tasks.py y agregue el siguiente código para una tarea de muestra que solo se registra en la consola:

from celery import shared_task
from celery.utils.log import get_task_logger


logger = get_task_logger(__name__)


@shared_task
def sample_task():
    logger.info("The sample task just ran.")

Programar la tarea

Al final de su archivo settings.py , agregue el siguiente código para programar sample_taskque se ejecute una vez por minuto, usando Celery Beat:

CELERY_BEAT_SCHEDULE = {
    "sample_task": {
        "task": "core.tasks.sample_task",
        "schedule": crontab(minute="*/1"),
    },
}

Aquí, definimos una tarea periódica usando la configuración CELERY_BEAT_SCHEDULE . Le dimos a la tarea un nombre, sample_tasky luego declaramos dos configuraciones:

  1. taskdeclara qué tarea ejecutar.
  2. scheduleestablece el intervalo en el que debe ejecutarse la tarea. Puede ser un número entero, un timedelta o un crontab. Usamos un patrón crontab para nuestra tarea para indicarle que se ejecute una vez por minuto. Puede encontrar más información sobre la programación de Celery aquí .

Asegúrate de agregar las importaciones:

from celery.schedules import crontab

import core.tasks

Reinicie el contenedor para obtener la nueva configuración:

$ docker-compose up -d --build

Una vez hecho esto, eche un vistazo a los troncos de apio en el contenedor:

$ docker-compose logs -f 'celery'

Debería ver algo similar a:

celery_1  |  -------------- [queues]
celery_1  |                 .> celery           exchange=celery(direct) key=celery
celery_1  |
celery_1  |
celery_1  | [tasks]
celery_1  |   . core.tasks.sample_task

Podemos ver que Celery recogió nuestra tarea de muestra, core.tasks.sample_task.

Cada minuto, debería ver una fila en el registro que termina con "La tarea de muestra acaba de ejecutarse":

celery_1  | [2021-07-01 03:06:00,003: INFO/MainProcess]
              Task core.tasks.sample_task[b8041b6c-bf9b-47ce-ab00-c37c1e837bc7] received
celery_1  | [2021-07-01 03:06:00,004: INFO/ForkPoolWorker-8]
              core.tasks.sample_task[b8041b6c-bf9b-47ce-ab00-c37c1e837bc7]:
              The sample task just ran.

Comando de administración de Django personalizado

Django proporciona una serie de django-admincomandos integrados, como:

  • migrate
  • startproject
  • startapp
  • dumpdata
  • makemigrations

Junto con los comandos incorporados, Django también nos da la opción de crear nuestros propios comandos personalizados :

Los comandos de administración personalizados son especialmente útiles para ejecutar secuencias de comandos independientes o secuencias de comandos que se ejecutan periódicamente desde el crontab de UNIX o desde el panel de control de tareas programadas de Windows.

Entonces, primero configuraremos un nuevo comando y luego usaremos Celery Beat para ejecutarlo automáticamente.

Comience creando un nuevo archivo llamado orders/management/commands/my_custom_command.py . Luego, agregue el código mínimo requerido para que se ejecute:

from django.core.management.base import BaseCommand, CommandError


class Command(BaseCommand):
    help = "A description of the command"

    def handle(self, *args, **options):
        pass

BaseCommandtiene algunos métodos que se pueden anular, pero el único método que se requiere es handle. handlees el punto de entrada para los comandos personalizados. En otras palabras, cuando ejecutamos el comando, se llama a este método.

Para probar, normalmente agregaríamos una declaración de impresión rápida. Sin embargo, se recomienda usar en su stdout.writelugar según la documentación de Django:

Cuando utilice comandos de administración y desee proporcionar una salida de consola, debe escribir en self.stdout y self.stderr, en lugar de imprimir directamente en stdout y stderr. Al usar estos servidores proxy, se vuelve mucho más fácil probar su comando personalizado. Tenga en cuenta también que no necesita finalizar los mensajes con un carácter de nueva línea, se agregará automáticamente, a menos que especifique el parámetro de finalización.

Entonces, agregue un self.stdout.writecomando:

from django.core.management.base import BaseCommand, CommandError


class Command(BaseCommand):
    help = "A description of the command"

    def handle(self, *args, **options):
        self.stdout.write("My sample command just ran.") # NEW

Para probar, desde la línea de comando, ejecute:

$ docker-compose exec web python manage.py my_custom_command

Debería ver:

My sample command just ran.

Con eso, ¡unamos todo junto!

Programa un Comando Personalizado con Celery Beat

Ahora que hicimos girar los contenedores, probamos que podemos programar una tarea para que se ejecute periódicamente y escribimos un comando de muestra de Django Admin personalizado, es hora de configurar Celery Beat para ejecutar el comando personalizado periódicamente.

Configuración

En el proyecto tenemos una aplicación muy básica llamada pedidos. Contiene dos modelos, Producty Order. Vamos a crear un comando personalizado que envíe un informe por correo electrónico de los pedidos confirmados del día.

Para empezar, agregaremos algunos productos y pedidos a la base de datos a través del accesorio incluido en este proyecto:

$ docker-compose exec web python manage.py loaddata products.json

A continuación, agregue algunos pedidos de muestra a través de la interfaz de administración de Django. Para hacerlo, primero crea un superusuario:

$ docker-compose exec web python manage.py createsuperuser

Complete el nombre de usuario, el correo electrónico y la contraseña cuando se le solicite. Luego navegue a http://127.0.0.1:1337/admin en su navegador web. Inicie sesión con el superusuario que acaba de crear y cree un par de pedidos. Asegúrese de que al menos uno tenga una confirmed_datede hoy.

Vamos a crear un nuevo comando personalizado para nuestro informe de correo electrónico.

Cree un archivo llamado orders/management/commands/email_report.py :

from datetime import timedelta, time, datetime

from django.core.mail import mail_admins
from django.core.management import BaseCommand
from django.utils import timezone
from django.utils.timezone import make_aware

from orders.models import Order

today = timezone.now()
tomorrow = today + timedelta(1)
today_start = make_aware(datetime.combine(today, time()))
today_end = make_aware(datetime.combine(tomorrow, time()))


class Command(BaseCommand):
    help = "Send Today's Orders Report to Admins"

    def handle(self, *args, **options):
        orders = Order.objects.filter(confirmed_date__range=(today_start, today_end))

        if orders:
            message = ""

            for order in orders:
                message += f"{order} \n"

            subject = (
                f"Order Report for {today_start.strftime('%Y-%m-%d')} "
                f"to {today_end.strftime('%Y-%m-%d')}"
            )

            mail_admins(subject=subject, message=message, html_message=None)

            self.stdout.write("E-mail Report was sent.")
        else:
            self.stdout.write("No orders confirmed today.")

En el código, consultamos la base de datos en busca de pedidos con confirmed_datefecha de hoy, combinamos los pedidos en un solo mensaje para el cuerpo del correo electrónico y usamos el mail_adminscomando integrado de Django para enviar los correos electrónicos a los administradores.

Agregue un correo electrónico de administrador ficticio y configure el EMAIL_BACKENDpara usar el backend de la consola , de modo que el correo electrónico se envíe a stdout, en el archivo de configuración:

EMAIL_BACKEND = "django.core.mail.backends.console.EmailBackend"
DEFAULT_FROM_EMAIL = "noreply@email.com"
ADMINS = [("testuser", "test.user@email.com"), ]

Ahora debería ser posible ejecutar nuestro nuevo comando desde la terminal.

$ docker-compose exec web python manage.py email_report

Y la salida debería ser similar a esto:

Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Subject: [Django] Order Report for 2021-07-01 to 2021-07-02
From: root@localhost
To: test.user@email.com
Date: Thu, 01 Jul 2021 12:15:50 -0000
Message-ID: <162514175053.40.5705892371538583115@5140844ebb15>

Order: 3947963f-1860-44d1-9b9a-4648fed04581 - product: Coffee
Order: ff449e6e-3dfd-48a8-9d5c-79a145d08253 - product: Rice

-------------------------------------------------------------------------------
E-mail Report was sent.

Batido de apio

Ahora necesitamos crear una tarea periódica para ejecutar este comando diariamente.

Agregue una nueva tarea a core/tasks.py :

from celery import shared_task
from celery.utils.log import get_task_logger
from django.core.management import call_command # NEW


logger = get_task_logger(__name__)


@shared_task
def sample_task():
    logger.info("The sample task just ran.")


# NEW
@shared_task
def send_email_report():
    call_command("email_report", )

Entonces, primero agregamos una call_commandimportación, que se usa para llamar mediante programación a los comandos django-admin. En la nueva tarea, usamos el call_commandcon el nombre de nuestro comando personalizado como argumento.

Para programar esta tarea, abra el archivo core/settings.py y actualice la CELERY_BEAT_SCHEDULEconfiguración para incluir la nueva tarea:

CELERY_BEAT_SCHEDULE = {
    "sample_task": {
        "task": "core.tasks.sample_task",
        "schedule": crontab(minute="*/1"),
    },
    "send_email_report": {
        "task": "core.tasks.send_email_report",
        "schedule": crontab(hour="*/1"),
    },
}

Aquí agregamos una nueva entrada al CELERY_BEAT_SCHEDULEllamado send_email_report. Como hicimos con nuestra tarea anterior, declaramos qué tarea debería ejecutar, por ejemplo, core.tasks.send_email_reporty usamos un patrón crontab para establecer la recurrencia.

Reinicie los contenedores para asegurarse de que la nueva configuración se active:

$ docker-compose up -d --build

Abra los registros asociados con el celeryservicio:

$ docker-compose logs -f 'celery'

Debería ver la send_email_reportlista:

celery_1  |  -------------- [queues]
celery_1  |                 .> celery           exchange=celery(direct) key=celery
celery_1  |
celery_1  |
celery_1  | [tasks]
celery_1  |   . core.tasks.sample_task
celery_1  |   . core.tasks.send_email_report

Aproximadamente un minuto después, debería ver que se envía el informe por correo electrónico:

celery_1  | [2021-07-01 12:22:00,071: WARNING/ForkPoolWorker-8] Content-Type: text/plain; charset="utf-8"
celery_1  | MIME-Version: 1.0
celery_1  | Content-Transfer-Encoding: 7bit
celery_1  | Subject: [Django] Order Report for 2021-07-01 to 2021-07-02
celery_1  | From: root@localhost
celery_1  | To: test.user@email.com
celery_1  | Date: Thu, 01 Jul 2021 12:22:00 -0000
celery_1  | Message-ID: <162514212006.17.6080459299558356876@55b9883c5414>
celery_1  |
celery_1  | Order: 3947963f-1860-44d1-9b9a-4648fed04581 - product: Coffee
celery_1  | Order: ff449e6e-3dfd-48a8-9d5c-79a145d08253 - product: Rice
celery_1  |
celery_1  |
celery_1  | [2021-07-01 12:22:00,071: WARNING/ForkPoolWorker-8] -------------------------------------------------------------------------------
celery_1  | [2021-07-01 12:22:00,071: WARNING/ForkPoolWorker-8]
celery_1  |
celery_1  | [2021-07-01 12:22:00,071: WARNING/ForkPoolWorker-8] E-mail Report was sent.

Conclusión

En este artículo lo guiamos a través de la configuración de contenedores Docker para Celery, Celery Beat y Redis. Luego mostramos cómo crear un comando Django Admin personalizado y una tarea periódica con Celery Beat para ejecutar ese comando automáticamente.

¿Buscando por mas?

  1. Configure Flower para monitorear y administrar trabajos y trabajadores de Celery
  2. Probar una tarea de Celery con pruebas unitarias y de integración

Tome el código del repositorio .

Fuente:  https://testdriven.io

#django #celery #docker #redis 

Cómo Configurar Django, Celery Y Redis Con Docker

Como Configurar Django, Celery E Redis Com O Docker

À medida que você cria e dimensiona um aplicativo Django, você inevitavelmente precisará executar determinadas tarefas periodicamente e automaticamente em segundo plano.

Alguns exemplos:

  • Gerando relatórios periódicos
  • Limpando o cache
  • Envio de notificações por e-mail em lote
  • Executando trabalhos de manutenção noturnos

Essa é uma das poucas funcionalidades necessárias para construir e dimensionar um aplicativo web que não faz parte do núcleo do Django. Felizmente, o Celery fornece uma solução poderosa, que é bastante fácil de implementar, chamada Celery Beat.

No artigo a seguir, mostraremos como configurar o Django, o Celery e o Redis com o Docker para executar um comando personalizado do Django Admin periodicamente com o Celery Beat.

Objetivos

Ao final deste tutorial, você deverá ser capaz de:

  1. Conteinerize Django, Aipo e Redis com o Docker
  2. Integre o Celery em um aplicativo Django e crie tarefas
  3. Escreva um comando personalizado do Django Admin
  4. Agende um comando personalizado do Django Admin para ser executado periodicamente via Celery Beat

Configuração do projeto

Clone o projeto base do repositório django-celery-beat e, em seguida, verifique o branch base :

$ git clone https://github.com/testdrivenio/django-celery-beat --branch base --single-branch
$ cd django-celery-beat

Como precisaremos gerenciar quatro processos no total (Django, Redis, worker e scheduler), usaremos o Docker para simplificar nosso fluxo de trabalho, conectando-os para que todos possam ser executados em uma janela de terminal com um único comando .

Na raiz do projeto, crie as imagens e ative os contêineres do Docker:

$ docker-compose up -d --build

Em seguida, aplique as migrações:

$ docker-compose exec web python manage.py migrate

Quando a compilação estiver concluída, navegue até http://localhost:1337 para garantir que o aplicativo funcione conforme o esperado. Você deverá ver o seguinte texto:

Orders

No orders found!

Dê uma olhada rápida na estrutura do projeto antes de prosseguir:

├── .gitignore
├── docker-compose.yml
└── project
    ├── Dockerfile
    ├── core
    │   ├── __init__.py
    │   ├── asgi.py
    │   ├── settings.py
    │   ├── urls.py
    │   └── wsgi.py
    ├── entrypoint.sh
    ├── manage.py
    ├── orders
    │   ├── __init__.py
    │   ├── admin.py
    │   ├── apps.py
    │   ├── migrations
    │   │   ├── 0001_initial.py
    │   │   └── __init__.py
    │   ├── models.py
    │   ├── tests.py
    │   ├── urls.py
    │   └── views.py
    ├── products.json
    ├── requirements.txt
    └── templates
        └── orders
            └── order_list.html

Quer aprender a construir este projeto? Confira a postagem no blog Dockerizing Django com Postgres, Gunicorn e Nginx .

Aipo e Redis

Agora, precisamos adicionar contêineres para Celery, Celery Beat e Redis.

Começaremos adicionando as dependências ao arquivo requirements.txt :

Django==3.2.4
celery==5.1.2
redis==3.5.3

Em seguida, adicione o seguinte ao final do arquivo docker-compose.yml :

redis:
  image: redis:alpine
celery:
  build: ./project
  command: celery -A core worker -l info
  volumes:
    - ./project/:/usr/src/app/
  environment:
    - DEBUG=1
    - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
    - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
  depends_on:
    - redis
celery-beat:
  build: ./project
  command: celery -A core beat -l info
  volumes:
    - ./project/:/usr/src/app/
  environment:
    - DEBUG=1
    - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
    - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
  depends_on:
    - redis

Também precisamos atualizar a depends_onseção do serviço web:

web:
  build: ./project
  command: python manage.py runserver 0.0.0.0:8000
  volumes:
    - ./project/:/usr/src/app/
  ports:
    - 1337:8000
  environment:
    - DEBUG=1
    - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
    - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
  depends_on:
    - redis # NEW

O arquivo docker-compose.yml completo agora deve ficar assim:

version: '3.8'

services:
  web:
    build: ./project
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - ./project/:/usr/src/app/
    ports:
      - 1337:8000
    environment:
      - DEBUG=1
      - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
      - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
    depends_on:
      - redis
  redis:
    image: redis:alpine
  celery:
    build: ./project
    command: celery -A core worker -l info
    volumes:
      - ./project/:/usr/src/app/
    environment:
      - DEBUG=1
      - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
      - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
    depends_on:
      - redis
  celery-beat:
    build: ./project
    command: celery -A core beat -l info
    volumes:
      - ./project/:/usr/src/app/
    environment:
      - DEBUG=1
      - SECRET_KEY=dbaa1_i7%*3r9-=z-+_mz4r-!qeed@(-a_r(g@k8jo8y3r27%m
      - DJANGO_ALLOWED_HOSTS=localhost 127.0.0.1 [::1]
    depends_on:
      - redis

Antes de construir os novos contêineres, precisamos configurar o Celery em nosso aplicativo Django.

Configuração de aipo

Configurar

No diretório "core", crie um arquivo celery.py e adicione o seguinte código:

import os

from celery import Celery


os.environ.setdefault("DJANGO_SETTINGS_MODULE", "core.settings")

app = Celery("core")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()

O que está acontecendo aqui?

  1. Primeiro, definimos um valor padrão para a DJANGO_SETTINGS_MODULEvariável de ambiente para que o Celery saiba como encontrar o projeto Django.
  2. Em seguida, criamos uma nova instância Celery, com o nome core, e atribuímos o valor a uma variável chamada app.
  3. Em seguida, carregamos os valores de configuração de aipo do objeto de configurações de django.conf. Costumávamos namespace="CELERY"evitar conflitos com outras configurações do Django. Todas as configurações do Aipo devem ser prefixadas com CELERY_, em outras palavras.
  4. Por fim, app.autodiscover_tasks()diz ao Celery para procurar tarefas do Celery em aplicativos definidos no settings.INSTALLED_APPS.

Adicione o seguinte código a core/__init__.py :

from .celery import app as celery_app

__all__ = ("celery_app",)

Por fim, atualize o arquivo core/settings.py com as seguintes configurações do Celery para que ele possa se conectar ao Redis:

CELERY_BROKER_URL = "redis://redis:6379"
CELERY_RESULT_BACKEND = "redis://redis:6379"

Construa os novos contêineres para garantir que tudo funcione:

$ docker-compose up -d --build

Dê uma olhada nos logs de cada serviço para ver se estão prontos, sem erros:

$ docker-compose logs 'web'
$ docker-compose logs 'celery'
$ docker-compose logs 'celery-beat'
$ docker-compose logs 'redis'

Se tudo correu bem, agora temos quatro contêineres, cada um com serviços diferentes.

Agora estamos prontos para criar uma tarefa de amostra para ver se ela funciona como deveria.

Criar uma tarefa

Crie um novo arquivo chamado core/tasks.py e adicione o seguinte código para uma tarefa de exemplo que apenas registra no console:

from celery import shared_task
from celery.utils.log import get_task_logger


logger = get_task_logger(__name__)


@shared_task
def sample_task():
    logger.info("The sample task just ran.")

Agende a tarefa

No final do arquivo settings.py , adicione o seguinte código para agendar sample_taska execução uma vez por minuto, usando Celery Beat:

CELERY_BEAT_SCHEDULE = {
    "sample_task": {
        "task": "core.tasks.sample_task",
        "schedule": crontab(minute="*/1"),
    },
}

Aqui, definimos uma tarefa periódica usando a configuração CELERY_BEAT_SCHEDULE . Demos um nome à tarefa sample_task, e declaramos duas configurações:

  1. taskdeclara qual tarefa executar.
  2. scheduledefine o intervalo no qual a tarefa deve ser executada. Pode ser um inteiro, um timedelta ou um crontab. Usamos um padrão crontab para nossa tarefa para dizer a ela para ser executada uma vez a cada minuto. Você pode encontrar mais informações sobre a programação do Celery aqui .

Certifique-se de adicionar as importações:

from celery.schedules import crontab

import core.tasks

Reinicie o contêiner para obter as novas configurações:

$ docker-compose up -d --build

Uma vez feito, dê uma olhada nos registros de aipo no contêiner:

$ docker-compose logs -f 'celery'

Você deve ver algo semelhante a:

celery_1  |  -------------- [queues]
celery_1  |                 .> celery           exchange=celery(direct) key=celery
celery_1  |
celery_1  |
celery_1  | [tasks]
celery_1  |   . core.tasks.sample_task

Podemos ver que Celery pegou nossa tarefa de exemplo, core.tasks.sample_task.

A cada minuto, você deve ver uma linha no log que termina com "A tarefa de amostra acabou de ser executada.":

celery_1  | [2021-07-01 03:06:00,003: INFO/MainProcess]
              Task core.tasks.sample_task[b8041b6c-bf9b-47ce-ab00-c37c1e837bc7] received
celery_1  | [2021-07-01 03:06:00,004: INFO/ForkPoolWorker-8]
              core.tasks.sample_task[b8041b6c-bf9b-47ce-ab00-c37c1e837bc7]:
              The sample task just ran.

Comando de administração do Django personalizado

O Django fornece váriosdjango-admin comandos embutidos , como:

  • migrate
  • startproject
  • startapp
  • dumpdata
  • makemigrations

Junto com os comandos embutidos, o Django também nos dá a opção de criar nossos próprios comandos personalizados :

Os comandos de gerenciamento personalizados são especialmente úteis para executar scripts autônomos ou para scripts que são executados periodicamente a partir do crontab UNIX ou do painel de controle de tarefas agendadas do Windows.

Então, vamos primeiro configurar um novo comando e então usar o Celery Beat para executá-lo automaticamente.

Comece criando um novo arquivo chamado orders/management/commands/my_custom_command.py . Em seguida, adicione o código mínimo necessário para que ele seja executado:

from django.core.management.base import BaseCommand, CommandError


class Command(BaseCommand):
    help = "A description of the command"

    def handle(self, *args, **options):
        pass

O BaseCommandtem alguns métodos que podem ser substituídos, mas o único método necessário é handle. handleé o ponto de entrada para comandos personalizados. Em outras palavras, quando executamos o comando, esse método é chamado.

Para testar, normalmente adicionamos uma instrução de impressão rápida. No entanto, é recomendado usar de stdout.writeacordo com a documentação do Django:

Quando estiver usando comandos de gerenciamento e desejar fornecer saída do console, você deve escrever em self.stdout e self.stderr, em vez de imprimir diretamente em stdout e stderr. Ao usar esses proxies, fica muito mais fácil testar seu comando personalizado. Observe também que você não precisa encerrar as mensagens com um caractere de nova linha, ele será adicionado automaticamente, a menos que você especifique o parâmetro de encerramento.

Então, adicione um self.stdout.writecomando:

from django.core.management.base import BaseCommand, CommandError


class Command(BaseCommand):
    help = "A description of the command"

    def handle(self, *args, **options):
        self.stdout.write("My sample command just ran.") # NEW

Para testar, na linha de comando, execute:

$ docker-compose exec web python manage.py my_custom_command

Você deveria ver:

My sample command just ran.

Com isso, vamos amarrar tudo junto!

Agende um comando personalizado com o Celery Beat

Agora que criamos os contêineres, testamos que podemos agendar uma tarefa para ser executada periodicamente e escrevemos um comando de exemplo personalizado do Django Admin, é hora de configurar o Celery Beat para executar o comando personalizado periodicamente.

Configurar

No projeto temos um aplicativo bem básico chamado pedidos. Ele contém dois modelos, Producte Order. Vamos criar um comando personalizado que envie um relatório por e-mail dos pedidos confirmados do dia.

Para começar, adicionaremos alguns produtos e pedidos ao banco de dados por meio do acessório incluído neste projeto:

$ docker-compose exec web python manage.py loaddata products.json

Em seguida, adicione alguns pedidos de amostra por meio da interface do Django Admin. Para fazer isso, primeiro crie um superusuário:

$ docker-compose exec web python manage.py createsuperuser

Preencha o nome de usuário, e-mail e senha quando solicitado. Em seguida, navegue até http://127.0.0.1:1337/admin em seu navegador da web. Faça login com o superusuário que você acabou de criar e crie alguns pedidos. Certifique-se de que pelo menos um tenha um confirmed_datede hoje.

Vamos criar um novo comando personalizado para nosso relatório de e-mail.

Crie um arquivo chamado orders/management/commands/email_report.py :

from datetime import timedelta, time, datetime

from django.core.mail import mail_admins
from django.core.management import BaseCommand
from django.utils import timezone
from django.utils.timezone import make_aware

from orders.models import Order

today = timezone.now()
tomorrow = today + timedelta(1)
today_start = make_aware(datetime.combine(today, time()))
today_end = make_aware(datetime.combine(tomorrow, time()))


class Command(BaseCommand):
    help = "Send Today's Orders Report to Admins"

    def handle(self, *args, **options):
        orders = Order.objects.filter(confirmed_date__range=(today_start, today_end))

        if orders:
            message = ""

            for order in orders:
                message += f"{order} \n"

            subject = (
                f"Order Report for {today_start.strftime('%Y-%m-%d')} "
                f"to {today_end.strftime('%Y-%m-%d')}"
            )

            mail_admins(subject=subject, message=message, html_message=None)

            self.stdout.write("E-mail Report was sent.")
        else:
            self.stdout.write("No orders confirmed today.")

No código, consultamos o banco de dados para pedidos com um confirmed_datede hoje, combinamos os pedidos em uma única mensagem para o corpo do e-mail e usamos o mail_adminscomando interno do Django para enviar os e-mails para os administradores.

Adicione um e-mail de administrador fictício e defina o EMAIL_BACKENDpara usar o back- end do console , para que o e-mail seja enviado para stdout, no arquivo de configurações:

EMAIL_BACKEND = "django.core.mail.backends.console.EmailBackend"
DEFAULT_FROM_EMAIL = "noreply@email.com"
ADMINS = [("testuser", "test.user@email.com"), ]

Agora deve ser possível executar nosso novo comando a partir do terminal.

$ docker-compose exec web python manage.py email_report

E a saída deve ser semelhante a esta:

Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Subject: [Django] Order Report for 2021-07-01 to 2021-07-02
From: root@localhost
To: test.user@email.com
Date: Thu, 01 Jul 2021 12:15:50 -0000
Message-ID: <162514175053.40.5705892371538583115@5140844ebb15>

Order: 3947963f-1860-44d1-9b9a-4648fed04581 - product: Coffee
Order: ff449e6e-3dfd-48a8-9d5c-79a145d08253 - product: Rice

-------------------------------------------------------------------------------
E-mail Report was sent.

Batida de aipo

Agora precisamos criar uma tarefa periódica para executar esse comando diariamente.

Adicione uma nova tarefa a core/tasks.py :

from celery import shared_task
from celery.utils.log import get_task_logger
from django.core.management import call_command # NEW


logger = get_task_logger(__name__)


@shared_task
def sample_task():
    logger.info("The sample task just ran.")


# NEW
@shared_task
def send_email_report():
    call_command("email_report", )

Então, primeiro adicionamos uma call_commandimportação, que é usada para chamar comandos django-admin programaticamente. Na nova tarefa, usamos o call_commandcom o nome do nosso comando personalizado como argumento.

Para agendar esta tarefa, abra o arquivo core/settings.py e atualize a CELERY_BEAT_SCHEDULEconfiguração para incluir a nova tarefa:

CELERY_BEAT_SCHEDULE = {
    "sample_task": {
        "task": "core.tasks.sample_task",
        "schedule": crontab(minute="*/1"),
    },
    "send_email_report": {
        "task": "core.tasks.send_email_report",
        "schedule": crontab(hour="*/1"),
    },
}

Aqui nós adicionamos uma nova entrada ao CELERY_BEAT_SCHEDULEchamado send_email_report. Como fizemos para nossa tarefa anterior, declaramos qual tarefa ela deveria executar -- por exemplo, core.tasks.send_email_report-- e usamos um padrão crontab para definir a recorrência.

Reinicie os contêineres para garantir que as novas configurações fiquem ativas:

$ docker-compose up -d --build

Abra os logs associados ao celeryserviço:

$ docker-compose logs -f 'celery'

Você deve ver o send_email_reportlistado:

celery_1  |  -------------- [queues]
celery_1  |                 .> celery           exchange=celery(direct) key=celery
celery_1  |
celery_1  |
celery_1  | [tasks]
celery_1  |   . core.tasks.sample_task
celery_1  |   . core.tasks.send_email_report

Cerca de um minuto depois, você verá que o relatório por e-mail foi enviado:

celery_1  | [2021-07-01 12:22:00,071: WARNING/ForkPoolWorker-8] Content-Type: text/plain; charset="utf-8"
celery_1  | MIME-Version: 1.0
celery_1  | Content-Transfer-Encoding: 7bit
celery_1  | Subject: [Django] Order Report for 2021-07-01 to 2021-07-02
celery_1  | From: root@localhost
celery_1  | To: test.user@email.com
celery_1  | Date: Thu, 01 Jul 2021 12:22:00 -0000
celery_1  | Message-ID: <162514212006.17.6080459299558356876@55b9883c5414>
celery_1  |
celery_1  | Order: 3947963f-1860-44d1-9b9a-4648fed04581 - product: Coffee
celery_1  | Order: ff449e6e-3dfd-48a8-9d5c-79a145d08253 - product: Rice
celery_1  |
celery_1  |
celery_1  | [2021-07-01 12:22:00,071: WARNING/ForkPoolWorker-8] -------------------------------------------------------------------------------
celery_1  | [2021-07-01 12:22:00,071: WARNING/ForkPoolWorker-8]
celery_1  |
celery_1  | [2021-07-01 12:22:00,071: WARNING/ForkPoolWorker-8] E-mail Report was sent.

Conclusão

Neste artigo, orientamos você na configuração de contêineres do Docker para Celery, Celery Beat e Redis. Em seguida, mostramos como criar um comando Django Admin personalizado e uma tarefa periódica com Celery Beat para executar esse comando automaticamente.

Procurando mais?

  1. Configure o Flower para monitorar e administrar trabalhos e trabalhadores de aipo
  2. Teste uma tarefa de aipo com testes de unidade e de integração

Pegue o código do repositório .

Fonte:  https://testdrive.io

#django #celery #docker 

Como Configurar Django, Celery E Redis Com O Docker
Emmy  Monahan

Emmy Monahan

1660308480

Guide for Celery and Django Database Transactions

In this article, we'll look at how to prevent a Celery task dependent on a Django database transaction from executing before the database commits the transaction. This is a fairly common issue.

Source: https://testdriven.io

#django #celery 

Guide for Celery and Django Database Transactions
Hoang Tran

Hoang Tran

1660301040

Hướng Dẫn Giao Dịch Cơ Sở Dữ Liệu Celery Và Django

Trong bài viết này, chúng ta sẽ xem xét cách ngăn một tác vụ Celery phụ thuộc vào giao dịch cơ sở dữ liệu Django thực thi trước khi cơ sở dữ liệu thực hiện giao dịch. Đây là một vấn đề khá phổ biến.

Mục tiêu

Sau khi đọc, bạn sẽ có thể:

  1. Mô tả giao dịch cơ sở dữ liệu là gì và cách sử dụng nó trong Django
  2. Giải thích tại sao bạn có thể gặp DoesNotExistlỗi trong Celery worker và cách giải quyết nó
  3. Ngăn một tác vụ thực thi trước khi cơ sở dữ liệu thực hiện một giao dịch

Giao dịch cơ sở dữ liệu là gì?

Một giao dịch cơ sở dữ liệu là một đơn vị công việc được cam kết (áp dụng cho cơ sở dữ liệu) hoặc được hoàn tác (hoàn tác khỏi cơ sở dữ liệu) như một đơn vị.

Hầu hết các cơ sở dữ liệu sử dụng mẫu sau:

  1. Bắt đầu giao dịch.
  2. Thực hiện một tập hợp các thao tác và / hoặc truy vấn dữ liệu.
  3. Nếu không có lỗi xảy ra, sau đó thực hiện giao dịch.
  4. Nếu xảy ra lỗi, hãy khôi phục giao dịch.

Như bạn có thể thấy, giao dịch là một cách rất hữu ích để giữ cho dữ liệu của bạn tránh xa sự hỗn loạn.

Cách sử dụng các giao dịch cơ sở dữ liệu trong Django

Bạn có thể tìm thấy mã nguồn của bài viết này trên Github . Nếu bạn quyết định sử dụng mã này khi làm việc trong bài viết này, hãy nhớ rằng tên người dùng phải là duy nhất. Bạn có thể sử dụng trình tạo tên người dùng ngẫu nhiên cho mục đích thử nghiệm như Faker .

Đầu tiên chúng ta hãy xem chế độ xem Django này:

def test_view(request):
    user = User.objects.create_user('john', 'lennon@thebeatles.com', 'johnpassword')
    logger.info(f'create user {user.pk}')
    raise Exception('test')

Điều gì xảy ra khi bạn truy cập chế độ xem này?

Hành vi mặc định

Hành vi mặc định của Django là tự động gửi : Mỗi truy vấn được cam kết trực tiếp với cơ sở dữ liệu trừ khi một giao dịch đang hoạt động. Nói cách khác, với autocommit, mỗi truy vấn bắt đầu một giao dịch và cam kết hoặc quay trở lại giao dịch đó. Nếu bạn có một chế độ xem với ba truy vấn, thì từng truy vấn sẽ chạy từng truy vấn một. Nếu một trong hai thất bại, hai người còn lại sẽ được cam kết.

Vì vậy, theo quan điểm trên, ngoại lệ được nêu ra sau khi giao dịch được cam kết, tạo ra người dùng john.

Kiểm soát rõ ràng

Nếu bạn muốn có nhiều quyền kiểm soát hơn đối với các giao dịch cơ sở dữ liệu, bạn có thể ghi đè hành vi mặc định với transaction.atomic . Trong chế độ này, trước khi gọi một hàm xem, Django bắt đầu một giao dịch. Nếu phản hồi được tạo ra mà không có vấn đề, Django cam kết giao dịch. Mặt khác, nếu chế độ xem tạo ra một ngoại lệ, Django sẽ khôi phục giao dịch. Nếu bạn có ba truy vấn và một truy vấn không thành công, thì sẽ không có truy vấn nào được cam kết.

Vì vậy, hãy viết lại khung nhìn bằng cách sử dụng transaction.atomic:

def transaction_test(request):
    with transaction.atomic():
        user = User.objects.create_user('john1', 'lennon@thebeatles.com', 'johnpassword')
        logger.info(f'create user {user.pk}')
        raise Exception('force transaction to rollback')

Bây giờ user createhoạt động sẽ quay trở lại khi ngoại lệ được nâng lên, vì vậy cuối cùng người dùng sẽ không được tạo.

transaction.atomiclà một công cụ rất hữu ích có thể giữ cho dữ liệu của bạn có tổ chức, đặc biệt là khi bạn cần thao tác dữ liệu trong các mô hình.

Nó cũng có thể được sử dụng như một đồ trang trí như vậy:

@transaction.atomic
def transaction_test2(request):
    user = User.objects.create_user('john1', 'lennon@thebeatles.com', 'johnpassword')
    logger.info(f'create user {user.pk}')
    raise Exception('force transaction to rollback')

Vì vậy, nếu một số lỗi xuất hiện trong chế độ xem và chúng tôi không tìm thấy lỗi đó, thì giao dịch sẽ quay trở lại.

Nếu bạn muốn sử dụng transaction.atomiccho tất cả các chức năng xem, bạn có thể đặt ATOMIC_REQUESTSthành Truetrong tệp cài đặt Django của mình:

ATOMIC_REQUESTS=True

# or

DATABASES["default"]["ATOMIC_REQUESTS"] = True

Sau đó, bạn có thể ghi đè hành vi để chế độ xem chạy ở chế độ tự động gửi:

@transaction.non_atomic_requests

Ngoại lệ DoesNotExist

Nếu bạn không có hiểu biết vững chắc về cách Django quản lý các giao dịch cơ sở dữ liệu, bạn có thể khá bối rối khi gặp các lỗi ngẫu nhiên liên quan đến cơ sở dữ liệu trong Celery worker.

Hãy xem một ví dụ:

@transaction.atomic
def transaction_celery(request):
    username = random_username()
    user = User.objects.create_user(username, 'lennon@thebeatles.com', 'johnpassword')
    logger.info(f'create user {user.pk}')
    task_send_welcome_email.delay(user.pk)

    time.sleep(1)
    return HttpResponse('test')

Mã nhiệm vụ có dạng như sau:

@shared_task()
def task_send_welcome_email(user_pk):
    user = User.objects.get(pk=user_pk)
    logger.info(f'send email to {user.email} {user.pk}')
  1. Vì chế độ xem sử dụng trình transaction.atomictrang trí, tất cả các hoạt động cơ sở dữ liệu chỉ được thực hiện nếu lỗi không xuất hiện trong chế độ xem, bao gồm cả tác vụ Cần tây.
  2. Nhiệm vụ khá đơn giản: Chúng tôi tạo một người dùng và sau đó chuyển khóa chính cho nhiệm vụ để gửi một email chào mừng.
  3. time.sleep(1)được sử dụng để giới thiệu một điều kiện chủng tộc.

Khi chạy, bạn sẽ thấy lỗi sau:

django.contrib.auth.models.User.DoesNotExist: User matching query does not exist.

Tại sao?

  1. Chúng tôi tạm dừng trong 1 giây sau khi xếp hàng đợi nhiệm vụ.
  2. Vì nhiệm vụ thực thi ngay lập tức, user = User.objects.get(pk=user_pk)không thành công khi người dùng không có trong cơ sở dữ liệu vì giao dịch trong Django chưa được cam kết.

Dung dịch

Có ba cách để giải quyết vấn đề này:

Vô hiệu hóa giao dịch cơ sở dữ liệu, vì vậy Django sẽ sử dụng autocommittính năng này. Để làm như vậy, bạn chỉ cần loại bỏ trình transaction.atomictrang trí. Tuy nhiên, điều này không được khuyến khích vì giao dịch cơ sở dữ liệu nguyên tử là một công cụ mạnh mẽ.

Buộc chạy tác vụ Cần tây sau một khoảng thời gian.

Ví dụ: để tạm dừng trong 10 giây:

task_send_welcome_email.apply_async (args = [user.pk], countdown = 10)

Django có một hàm gọi lại được gọi là transaction.on_committhực thi sau khi giao dịch được cam kết thành công. Để sử dụng cái này, hãy cập nhật dạng xem như sau:

@ transaction.atomic def transaction_celery2 (request): username = random_username () user = User.objects.create_user (username, 'lennon@thebeatles.com', 'johnpassword') logger.info (f'create user {user.pk} ')     # tác vụ không được gọi cho đến khi giao dịch được cam kết    transaction.on_commit ( lambda : task_send_welcome_email.delay (user.pk)) time.sleep (1)     return HttpResponse (' test ')

Bây giờ, tác vụ không được gọi cho đến sau khi cam kết giao dịch cơ sở dữ liệu. Vì vậy, khi Celery worker tìm thấy người dùng, nó có thể được tìm thấy vì mã trong worker luôn chạy sau khi giao dịch cơ sở dữ liệu Django cam kết thành công.

Đây là giải pháp được khuyến nghị .

Cần lưu ý rằng bạn có thể không muốn giao dịch của mình được cam kết ngay lập tức, đặc biệt nếu bạn đang chạy trong môi trường quy mô lớn. Nếu cơ sở dữ liệu hoặc phiên bản đang ở mức sử dụng cao, việc ép buộc một cam kết sẽ chỉ thêm vào mức sử dụng hiện có. Trong trường hợp này, bạn có thể muốn sử dụng giải pháp thứ hai và đợi một khoảng thời gian đủ (có thể là 20 giây) để đảm bảo rằng các thay đổi được thực hiện đối với cơ sở dữ liệu trước khi tác vụ thực thi.

Thử nghiệm

Django TestCasekết thúc mỗi thử nghiệm trong một giao dịch, sau đó sẽ được quay lại sau mỗi lần thử nghiệm. Vì không có giao dịch nào được cam kết nên cũng on_commit()không bao giờ chạy. Vì vậy, nếu bạn cần kiểm tra mã được kích hoạt trong một lệnh on_commitgọi lại, bạn có thể sử dụng TransactionTestCase hoặc TestCase.captureOnCommitCallbacks () trong mã kiểm tra của mình.

Giao dịch cơ sở dữ liệu trong tác vụ Cần tây

Nếu tác vụ Cần tây của bạn cần cập nhật bản ghi cơ sở dữ liệu, bạn nên sử dụng giao dịch cơ sở dữ liệu trong tác vụ Cần tây.

Một cách đơn giản là with transaction.atomic():

@shared_task()
def task_transaction_test():
    with transaction.atomic():
        from .views import random_username
        username = random_username()
        user = User.objects.create_user(username, 'lennon@thebeatles.com', 'johnpassword')
        user.save()
        logger.info(f'send email to {user.pk}')
        raise Exception('test')

Một cách tiếp cận tốt hơn là viết một tùy chỉnh decoratortransactionhỗ trợ:

class custom_celery_task:
    """
    This is a decorator we can use to add custom logic to our Celery task
    such as retry or database transaction
    """
    def __init__(self, *args, **kwargs):
        self.task_args = args
        self.task_kwargs = kwargs

    def __call__(self, func):
        @functools.wraps(func)
        def wrapper_func(*args, **kwargs):
            try:
                with transaction.atomic():
                    return func(*args, **kwargs)
            except Exception as e:
                # task_func.request.retries
                raise task_func.retry(exc=e, countdown=5)

        task_func = shared_task(*self.task_args, **self.task_kwargs)(wrapper_func)
        return task_func

...

@custom_celery_task(max_retries=5)
def task_transaction_test():
    # do something

Sự kết luận

Bài viết này xem xét cách làm cho Celery hoạt động tốt với các giao dịch cơ sở dữ liệu Django.

Mã nguồn của bài viết này có thể được tìm thấy trên GitHub .

Nguồn:  https://testdriven.io

#django #celery 

Hướng Dẫn Giao Dịch Cơ Sở Dữ Liệu Celery Và Django

How to Automatically Retry Failed Celery Tasks

In this article, we'll look at how to automatically retry failed Celery tasks.

Objectives

After reading, you should be able to:

  1. Retry a failed Celery task with both the retry method and a decorator argument
  2. Use exponential backoff when retrying a failed task
  3. Use a class-based task to reuse retry arguments

Source: https://testdriven.io

#celery #python 

How to Automatically Retry Failed Celery Tasks