How to Configure Redis Queue (RQ) To Handle Running Tasks in A Flask

Learn How to Configure Redis Queue (RQ) To Handle Running Tasks in A Flask

By the end of this post you should be able to:

Integrate Redis Queue into a Flask app and create tasks.
Containerize Flask and Redis with Docker.
Run long-running tasks in the background with a separate worker process.
Set up RQ Dashboard to monitor queues, jobs, and workers.
Scale the worker count with Docker.

Redis Transactions & Long-Running Lua Scripts

Redis offers two mechanisms for handling transactions – MULTI/EXEC based transactions and Lua scripts evaluation. Redis Lua scripting is the recommended approach and is fairly popular in usage.

Our Redis™ customers who have Lua scripts deployed often report this error – “BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE”. In this post, we will explain the Redis transactional property of scripts, what this error is about, and why we must be extra careful about it on Sentinel-managed systems that can failover.

Transactional Nature of Redis Lua Scripts

Redis “transactions” aren’t really transactions as understood conventionally – in case of errors, there is no rollback of writes made by the script.

“Atomicity” of Redis scripts is guaranteed in the following manner:

Once a script begins executing, all other commands/scripts are blocked until the script completes. So, other clients either see the changes made by the script or they don’t. This is because they can only execute either before the script or after the script.
However, Redis doesn’t do rollbacks, so on an error within a script, any changes already made by the script will be retained and future commands/scripts will see those partial changes.
Since all other clients are blocked while the script executes, it is critical that the script is well-behaved and finishes in time.

The ‘lua-time-limit’ Value

It is highly recommended that the script complete within a time limit. Redis enforces this in a weak manner with the ‘lua-time-limit’ value. This is the maximum allowed time (in ms) that the script is allowed to run. The default value is 5 seconds. This is a really long time for CPU-bound activity (scripts have limited access and can’t run commands that access the disk).

However, the script is not killed when it executes beyond this time. Redis starts accepting client commands again, but responds to them with a BUSY error.

If you must kill the script at this point, there are two options available:

SCRIPT KILL command can be used to stop a script that hasn’t yet done any writes.
If the script has already performed writes to the server and must still be killed, use the SHUTDOWN NOSAVE to shutdown the server completely.

It is usually better to just wait for the script to complete its operation. The complete information on methods to kill the script execution and related behavior are available in the documentation.

#cloud #database #developer #high availability #howto #redis #scalegrid #lua-time-limit #redis diagram #redis master #redis scripts #redis sentinel #redis servers #redis transactions #sentinel-managed #server failures

Flask Framework

1631888830

How to Configure Redis Queue (RQ) To Handle Running Tasks in A Flask

Learn How to Configure Redis Queue (RQ) To Handle Running Tasks in A Flask

By the end of this post you should be able to:

Integrate Redis Queue into a Flask app and create tasks.
Containerize Flask and Redis with Docker.
Run long-running tasks in the background with a separate worker process.
Set up RQ Dashboard to monitor queues, jobs, and workers.
Scale the worker count with Docker.

#flask #redis

React Native Dev

1652543820

Background Fetch for React Native Apps

react-native-background-fetch

Background Fetch is a very simple plugin which attempts to awaken an app in the background about every 15 minutes, providing a short period of background running-time. This plugin will execute your provided callbackFn whenever a background-fetch event occurs.

There is no way to increase the rate which a fetch-event occurs and this plugin sets the rate to the most frequent possible — you will never receive an event faster than 15 minutes. The operating-system will automatically throttle the rate the background-fetch events occur based upon usage patterns. Eg: if user hasn't turned on their phone for a long period of time, fetch events will occur less frequently or if an iOS user disables background refresh they may not happen at all.

:new: Background Fetch now provides a scheduleTask method for scheduling arbitrary "one-shot" or periodic tasks.

iOS

There is no way to increase the rate which a fetch-event occurs and this plugin sets the rate to the most frequent possible — you will never receive an event faster than 15 minutes. The operating-system will automatically throttle the rate the background-fetch events occur based upon usage patterns. Eg: if user hasn't turned on their phone for a long period of time, fetch events will occur less frequently.
scheduleTask seems only to fire when the device is plugged into power.
⚠️ When your app is terminated, iOS no longer fires events — There is no such thing as stopOnTerminate: false for iOS.
iOS can take days before Apple's machine-learning algorithm settles in and begins regularly firing events. Do not sit staring at your logs waiting for an event to fire. If your simulated events work, that's all you need to know that everything is correctly configured.
If the user doesn't open your iOS app for long periods of time, iOS will stop firing events.

Android

The Android plugin provides a HeadlessJS implementation allowing you to continue handling events even after app-termination (see @config enableHeadless)

Installing the plugin

⚠️ If you have a previous version of react-native-background-fetch < 2.7.0 installed into react-native >= 0.60, you should first unlink your previous version as react-native link is no longer required.

$ react-native unlink react-native-background-fetch

With `yarn`

$ yarn add react-native-background-fetch

With `npm`

$ npm install --save react-native-background-fetch

Setup Guides

iOS Setup

`react-native >= 0.60`

Auto-linking Setup

Android Setup

`react-native >= 0.60`

Auto-linking Setup

Example

ℹ️ This repo contains its own Example App. See /example

import React from 'react';
import {
  SafeAreaView,
  StyleSheet,
  ScrollView,
  View,
  Text,
  FlatList,
  StatusBar,
} from 'react-native';

import {
  Header,
  Colors
} from 'react-native/Libraries/NewAppScreen';

import BackgroundFetch from "react-native-background-fetch";

class App extends React.Component {
  constructor(props) {
    super(props);
    this.state = {
      events: []
    };
  }

  componentDidMount() {
    // Initialize BackgroundFetch ONLY ONCE when component mounts.
    this.initBackgroundFetch();
  }

  async initBackgroundFetch() {
    // BackgroundFetch event handler.
    const onEvent = async (taskId) => {
      console.log('[BackgroundFetch] task: ', taskId);
      // Do your background work...
      await this.addEvent(taskId);
      // IMPORTANT:  You must signal to the OS that your task is complete.
      BackgroundFetch.finish(taskId);
    }

    // Timeout callback is executed when your Task has exceeded its allowed running-time.
    // You must stop what you're doing immediately BackgroundFetch.finish(taskId)
    const onTimeout = async (taskId) => {
      console.warn('[BackgroundFetch] TIMEOUT task: ', taskId);
      BackgroundFetch.finish(taskId);
    }

    // Initialize BackgroundFetch only once when component mounts.
    let status = await BackgroundFetch.configure({minimumFetchInterval: 15}, onEvent, onTimeout);

    console.log('[BackgroundFetch] configure status: ', status);
  }

  // Add a BackgroundFetch event to <FlatList>
  addEvent(taskId) {
    // Simulate a possibly long-running asynchronous task with a Promise.
    return new Promise((resolve, reject) => {
      this.setState(state => ({
        events: [...state.events, {
          taskId: taskId,
          timestamp: (new Date()).toString()
        }]
      }));
      resolve();
    });
  }

  render() {
    return (
      <>
        <StatusBar barStyle="dark-content" />
        <SafeAreaView>
          <ScrollView
            contentInsetAdjustmentBehavior="automatic"
            style={styles.scrollView}>
            <Header />

            <View style={styles.body}>
              <View style={styles.sectionContainer}>
                <Text style={styles.sectionTitle}>BackgroundFetch Demo</Text>
              </View>
            </View>
          </ScrollView>
          <View style={styles.sectionContainer}>
            <FlatList
              data={this.state.events}
              renderItem={({item}) => (<Text>[{item.taskId}]: {item.timestamp}</Text>)}
              keyExtractor={item => item.timestamp}
            />
          </View>
        </SafeAreaView>
      </>
    );
  }
}

const styles = StyleSheet.create({
  scrollView: {
    backgroundColor: Colors.lighter,
  },
  body: {
    backgroundColor: Colors.white,
  },
  sectionContainer: {
    marginTop: 32,
    paddingHorizontal: 24,
  },
  sectionTitle: {
    fontSize: 24,
    fontWeight: '600',
    color: Colors.black,
  },
  sectionDescription: {
    marginTop: 8,
    fontSize: 18,
    fontWeight: '400',
    color: Colors.dark,
  },
});

export default App;

Executing Custom Tasks

In addition to the default background-fetch task defined by BackgroundFetch.configure, you may also execute your own arbitrary "oneshot" or periodic tasks (iOS requires additional Setup Instructions). However, all events will be fired into the Callback provided to BackgroundFetch#configure:

⚠️ iOS:

scheduleTask on iOS seems only to run when the device is plugged into power.
scheduleTask on iOS are designed for low-priority tasks, such as purging cache files — they tend to be unreliable for mission-critical tasks. scheduleTask will never run as frequently as you want.
The default fetch event is much more reliable and fires far more often.
scheduleTask on iOS stop when the user terminates the app. There is no such thing as stopOnTerminate: false for iOS.

// Step 1:  Configure BackgroundFetch as usual.
let status = await BackgroundFetch.configure({
  minimumFetchInterval: 15
}, async (taskId) => {  // <-- Event callback
  // This is the fetch-event callback.
  console.log("[BackgroundFetch] taskId: ", taskId);

  // Use a switch statement to route task-handling.
  switch (taskId) {
    case 'com.foo.customtask':
      print("Received custom task");
      break;
    default:
      print("Default fetch task");
  }
  // Finish, providing received taskId.
  BackgroundFetch.finish(taskId);
}, async (taskId) => {  // <-- Task timeout callback
  // This task has exceeded its allowed running-time.
  // You must stop what you're doing and immediately .finish(taskId)
  BackgroundFetch.finish(taskId);
});

// Step 2:  Schedule a custom "oneshot" task "com.foo.customtask" to execute 5000ms from now.
BackgroundFetch.scheduleTask({
  taskId: "com.foo.customtask",
  forceAlarmManager: true,
  delay: 5000  // <-- milliseconds
});

API Documentation

Config

Common Options

`@param {Integer} minimumFetchInterval [15]`

The minimum interval in minutes to execute background fetch events. Defaults to 15 minutes. Note: Background-fetch events will never occur at a frequency higher than every 15 minutes. Apple uses a secret algorithm to adjust the frequency of fetch events, presumably based upon usage patterns of the app. Fetch events can occur less often than your configured minimumFetchInterval.

`@param {Integer} delay (milliseconds)`

ℹ️ Valid only for BackgroundFetch.scheduleTask. The minimum number of milliseconds in future that task should execute.

`@param {Boolean} periodic [false]`

ℹ️ Valid only for BackgroundFetch.scheduleTask. Defaults to false. Set true to execute the task repeatedly. When false, the task will execute just once.

Android Options

`@config {Boolean} stopOnTerminate [true]`

Set false to continue background-fetch events after user terminates the app. Default to true.

`@config {Boolean} startOnBoot [false]`

Set true to initiate background-fetch events when the device is rebooted. Defaults to false.

❗ NOTE: startOnBoot requires stopOnTerminate: false.

`@config {Boolean} forceAlarmManager [false]`

By default, the plugin will use Android's JobScheduler when possible. The JobScheduler API prioritizes for battery-life, throttling task-execution based upon device usage and battery level.

Configuring forceAlarmManager: true will bypass JobScheduler to use Android's older AlarmManager API, resulting in more accurate task-execution at the cost of higher battery usage.

let status = await BackgroundFetch.configure({
  minimumFetchInterval: 15,
  forceAlarmManager: true
}, async (taskId) => {  // <-- Event callback
  console.log("[BackgroundFetch] taskId: ", taskId);
  BackgroundFetch.finish(taskId);
}, async (taskId) => {  // <-- Task timeout callback
  // This task has exceeded its allowed running-time.
  // You must stop what you're doing and immediately .finish(taskId)
  BackgroundFetch.finish(taskId);
});
.
.
.
// And with with #scheduleTask
BackgroundFetch.scheduleTask({
  taskId: 'com.foo.customtask',
  delay: 5000,       // milliseconds
  forceAlarmManager: true,
  periodic: false
});

`@config {Boolean} enableHeadless [false]`

Set true to enable React Native's Headless JS mechanism, for handling fetch events after app termination.

📂 index.js (MUST BE IN index.js):

import BackgroundFetch from "react-native-background-fetch";

let MyHeadlessTask = async (event) => {
  // Get task id from event {}:
  let taskId = event.taskId;
  let isTimeout = event.timeout;  // <-- true when your background-time has expired.
  if (isTimeout) {
    // This task has exceeded its allowed running-time.
    // You must stop what you're doing immediately finish(taskId)
    console.log('[BackgroundFetch] Headless TIMEOUT:', taskId);
    BackgroundFetch.finish(taskId);
    return;
  }
  console.log('[BackgroundFetch HeadlessTask] start: ', taskId);

  // Perform an example HTTP request.
  // Important:  await asychronous tasks when using HeadlessJS.
  let response = await fetch('https://reactnative.dev/movies.json');
  let responseJson = await response.json();
  console.log('[BackgroundFetch HeadlessTask] response: ', responseJson);

  // Required:  Signal to native code that your task is complete.
  // If you don't do this, your app could be terminated and/or assigned
  // battery-blame for consuming too much time in background.
  BackgroundFetch.finish(taskId);
}

// Register your BackgroundFetch HeadlessTask
BackgroundFetch.registerHeadlessTask(MyHeadlessTask);

`@config {integer} requiredNetworkType [BackgroundFetch.NETWORK_TYPE_NONE]`

Set basic description of the kind of network your job requires.

If your job doesn't need a network connection, you don't need to use this option as the default value is BackgroundFetch.NETWORK_TYPE_NONE.

NetworkType	Description
`BackgroundFetch.NETWORK_TYPE_NONE`	This job doesn't care about network constraints, either any or none.
`BackgroundFetch.NETWORK_TYPE_ANY`	This job requires network connectivity.
`BackgroundFetch.NETWORK_TYPE_CELLULAR`	This job requires network connectivity that is a cellular network.
`BackgroundFetch.NETWORK_TYPE_UNMETERED`	This job requires network connectivity that is unmetered. Most WiFi networks are unmetered, as in "you can upload as much as you like".
`BackgroundFetch.NETWORK_TYPE_NOT_ROAMING`	This job requires network connectivity that is not roaming (being outside the country of origin)

`@config {Boolean} requiresBatteryNotLow [false]`

Specify that to run this job, the device's battery level must not be low.

This defaults to false. If true, the job will only run when the battery level is not low, which is generally the point where the user is given a "low battery" warning.

`@config {Boolean} requiresStorageNotLow [false]`

Specify that to run this job, the device's available storage must not be low.

This defaults to false. If true, the job will only run when the device is not in a low storage state, which is generally the point where the user is given a "low storage" warning.

`@config {Boolean} requiresCharging [false]`

Specify that to run this job, the device must be charging (or be a non-battery-powered device connected to permanent power, such as Android TV devices). This defaults to false.

`@config {Boolean} requiresDeviceIdle [false]`

When set true, ensure that this job will not run if the device is in active use.

The default state is false: that is, the for the job to be runnable even when someone is interacting with the device.

This state is a loose definition provided by the system. In general, it means that the device is not currently being used interactively, and has not been in use for some time. As such, it is a good time to perform resource heavy jobs. Bear in mind that battery usage will still be attributed to your application, and shown to the user in battery stats.

Methods

Method Name	Arguments	Returns	Notes
`configure`	`{FetchConfig}`, `callbackFn`, `timeoutFn`	`Promise<BackgroundFetchStatus>`	Configures the plugin's `callbackFn` and `timeoutFn`. This callback will fire each time a background-fetch event occurs in addition to events from `#scheduleTask`. The `timeoutFn` will be called when the OS reports your task is nearing the end of its allowed background-time.
`scheduleTask`	`{TaskConfig}`	`Promise<boolean>`	Executes a custom task. The task will be executed in the same `Callback` function provided to `#configure`.
`status`	`callbackFn`	`Promise<BackgroundFetchStatus>`	Your callback will be executed with the current `status (Integer)` `0: Restricted`, `1: Denied`, `2: Available`. These constants are defined as `BackgroundFetch.STATUS_RESTRICTED`, `BackgroundFetch.STATUS_DENIED`, `BackgroundFetch.STATUS_AVAILABLE` (NOTE: Android will always return `STATUS_AVAILABLE`)
`finish`	`String taskId`	`Void`	You MUST call this method in your `callbackFn` provided to `#configure` in order to signal to the OS that your task is complete. iOS provides only 30s of background-time for a fetch-event -- if you exceed this 30s, iOS will kill your app.
`start`	`none`	`Promise<BackgroundFetchStatus>`	Start the background-fetch API. Your `callbackFn` provided to `#configure` will be executed each time a background-fetch event occurs. NOTE the `#configure` method automatically calls `#start`. You do not have to call this method after you `#configure` the plugin
`stop`	`[taskId:String]`	`Promise<boolean>`	Stop the background-fetch API and all `#scheduleTask` from firing events. Your `callbackFn` provided to `#configure` will no longer be executed. If you provide an optional `taskId`, only that `#scheduleTask` will be stopped.

Debugging

iOS

🆕 `BGTaskScheduler` API for iOS 13+

⚠️ At the time of writing, the new task simulator does not yet work in Simulator; Only real devices.
See Apple docs Starting and Terminating Tasks During Development
After running your app in XCode, Click the [||] button to initiate a Breakpoint.
In the console (lldb), paste the following command (Note: use cursor up/down keys to cycle through previously run commands):

e -l objc -- (void)[[BGTaskScheduler sharedScheduler] _simulateLaunchForTaskWithIdentifier:@"com.transistorsoft.fetch"]

Click the [ > ] button to continue. The task will execute and the Callback function provided to BackgroundFetch.configure will receive the event.

Simulating task-timeout events

Only the new BGTaskScheduler api supports simulated task-timeout events. To simulate a task-timeout, your fetchCallback must not call BackgroundFetch.finish(taskId):

let status = await BackgroundFetch.configure({
  minimumFetchInterval: 15
}, async (taskId) => {  // <-- Event callback.
  // This is the task callback.
  console.log("[BackgroundFetch] taskId", taskId);
  //BackgroundFetch.finish(taskId); // <-- Disable .finish(taskId) when simulating an iOS task timeout
}, async (taskId) => {  // <-- Event timeout callback
  // This task has exceeded its allowed running-time.
  // You must stop what you're doing and immediately .finish(taskId)
  print("[BackgroundFetch] TIMEOUT taskId:", taskId);
  BackgroundFetch.finish(taskId);
});

Now simulate an iOS task timeout as follows, in the same manner as simulating an event above:

e -l objc -- (void)[[BGTaskScheduler sharedScheduler] _simulateExpirationForTaskWithIdentifier:@"com.transistorsoft.fetch"]

Old `BackgroundFetch` API

Simulate background fetch events in XCode using Debug->Simulate Background Fetch
iOS can take some hours or even days to start a consistently scheduling background-fetch events since iOS schedules fetch events based upon the user's patterns of activity. If Simulate Background Fetch works, your can be sure that everything is working fine. You just need to wait.

Android

Observe plugin logs in $ adb logcat:

$ adb logcat *:S ReactNative:V ReactNativeJS:V TSBackgroundFetch:V

Simulate a background-fetch event on a device (insert <your.application.id>) (only works for sdk 21+:

$ adb shell cmd jobscheduler run -f <your.application.id> 999

For devices with sdk <21, simulate a "Headless JS" event with (insert <your.application.id>)

$ adb shell am broadcast -a <your.application.id>.event.BACKGROUND_FETCH

Download Details:
Author: transistorsoft
Source Code: https://github.com/transistorsoft/react-native-background-fetch
License: MIT license

#react #reactnative #mobileapp #javascript

Vicenta Hauck

1660215360

Configure Redis Queue to Handle Long-Running Tasks in A Flask App

This tutorial looks at how to configure Redis Queue (RQ) to handle long-running tasks in a Flask app.

Source: https://testdriven.io

#flask #redis

Jamar Hodkiewicz

1668680340

How to Asynchronous Tasks with Flask and Redis Queue

If a long-running task is part of your application's workflow you should handle it in the background, outside the normal flow.

Perhaps your web application requires users to submit a thumbnail (which will probably need to be re-sized) and confirm their email when they register. If your application processed the image and sent a confirmation email directly in the request handler, then the end user would have to wait for them both to finish. Instead, you'll want to pass these tasks off to a task queue and let a separate worker process deal with it, so you can immediately send a response back to the client. The end user can do other things on the client-side and your application is free to respond to requests from other users.

This tutorial looks at how to configure Redis Queue (RQ) to handle long-running tasks in a Flask app.

Celery is a viable solution as well. Check out Asynchronous Tasks with Flask and Celery for more.

Objectives

By the end of this tutorial, you will be able to:

Integrate Redis Queue into a Flask app and create tasks.
Containerize Flask and Redis with Docker.
Run long-running tasks in the background with a separate worker process.
Set up RQ Dashboard to monitor queues, jobs, and workers.
Scale the worker count with Docker.

Workflow

Our goal is to develop a Flask application that works in conjunction with Redis Queue to handle long-running processes outside the normal request/response cycle.

The end user kicks off a new task via a POST request to the server-side
Within the view, a task is added to the queue and the task id is sent back to the client-side
Using AJAX, the client continues to poll the server to check the status of the task while the task itself is running in the background

flask and redis queue user flow

In the end, the app will look like this:

final app

Project Setup

Want to follow along? Clone down the base project, and then review the code and project structure:

$ git clone https://github.com/mjhea0/flask-redis-queue --branch base --single-branch
$ cd flask-redis-queue

Since we'll need to manage three processes in total (Flask, Redis, worker), we'll use Docker to simplify our workflow so they can be managed from a single terminal window.

To test, run:

$ docker-compose up -d --build

Open your browser to http://localhost:5004. You should see:

flask, redis queue, docker

Trigger a Task

An event handler in project/client/static/main.js is set up that listens for a button click and sends an AJAX POST request to the server with the appropriate task type: 1, 2, or 3.

$('.btn').on('click', function() {
  $.ajax({
    url: '/tasks',
    data: { type: $(this).data('type') },
    method: 'POST'
  })
  .done((res) => {
    getStatus(res.data.task_id);
  })
  .fail((err) => {
    console.log(err);
  });
});

On the server-side, a view is already configured to handle the request in project/server/main/views.py:

@main_blueprint.route("/tasks", methods=["POST"])
def run_task():
    task_type = request.form["type"]
    return jsonify(task_type), 202

We just need to wire up Redis Queue.

Redis Queue

So, we need to spin up two new processes: Redis and a worker. Add them to the docker-compose.yml file:

version: '3.8'

services:

  web:
    build: .
    image: web
    container_name: web
    ports:
      - 5004:5000
    command: python manage.py run -h 0.0.0.0
    volumes:
      - .:/usr/src/app
    environment:
      - FLASK_DEBUG=1
      - APP_SETTINGS=project.server.config.DevelopmentConfig
    depends_on:
      - redis

  worker:
    image: web
    command: python manage.py run_worker
    volumes:
      - .:/usr/src/app
    environment:
      - APP_SETTINGS=project.server.config.DevelopmentConfig
    depends_on:
      - redis

  redis:
    image: redis:6.2-alpine

Add the task to a new file called tasks.py in "project/server/main":

# project/server/main/tasks.py


import time


def create_task(task_type):
    time.sleep(int(task_type) * 10)
    return True

Update the view to connect to Redis, enqueue the task, and respond with the id:

@main_blueprint.route("/tasks", methods=["POST"])
def run_task():
    task_type = request.form["type"]
    with Connection(redis.from_url(current_app.config["REDIS_URL"])):
        q = Queue()
        task = q.enqueue(create_task, task_type)
    response_object = {
        "status": "success",
        "data": {
            "task_id": task.get_id()
        }
    }
    return jsonify(response_object), 202

Don't forget the imports:

import redis
from rq import Queue, Connection
from flask import render_template, Blueprint, jsonify, request, current_app

from project.server.main.tasks import create_task

Update BaseConfig:

class BaseConfig(object):
    """Base configuration."""

    WTF_CSRF_ENABLED = True
    REDIS_URL = "redis://redis:6379/0"
    QUEUES = ["default"]

Did you notice that we referenced the redis service (from docker-compose.yml) in the REDIS_URL rather than localhost or some other IP? Review the Docker Compose docs for more info on connecting to other services via the hostname.

Finally, we can use a Redis Queue worker, to process tasks at the top of the queue.

manage.py:

@cli.command("run_worker")
def run_worker():
    redis_url = app.config["REDIS_URL"]
    redis_connection = redis.from_url(redis_url)
    with Connection(redis_connection):
        worker = Worker(app.config["QUEUES"])
        worker.work()

Here, we set up a custom CLI command to fire the worker.

It's important to note that the @cli.command() decorator will provide access to the application context along with the associated config variables from project/server/config.py when the command is executed.

Add the imports as well:

import redis
from rq import Connection, Worker

Add the dependencies to the requirements file:

redis==4.1.1
rq==1.10.1

Build and spin up the new containers:

$ docker-compose up -d --build

To trigger a new task, run:

$ curl -F type=0 http://localhost:5004/tasks

You should see something like:

{
  "data": {
    "task_id": "bdad64d0-3865-430e-9cc3-ec1410ddb0fd"
  },
  "status": "success"
}

Task Status

Turn back to the event handler on the client-side:

$('.btn').on('click', function() {
  $.ajax({
    url: '/tasks',
    data: { type: $(this).data('type') },
    method: 'POST'
  })
  .done((res) => {
    getStatus(res.data.task_id);
  })
  .fail((err) => {
    console.log(err);
  });
});

Once the response comes back from the original AJAX request, we then continue to call getStatus() with the task id every second. If the response is successful, a new row is added to the table on the DOM.

function getStatus(taskID) {
  $.ajax({
    url: `/tasks/${taskID}`,
    method: 'GET',
  })
  .done((res) => {
    const html = `
    <tr>
      <td>${res.data.task_id}</td>
      <td>${res.data.task_status}</td>
      <td>${res.data.task_result}</td>
    </tr>`;
    $('#tasks').prepend(html);
    const taskStatus = res.data.task_status;
    if (taskStatus === 'finished' || taskStatus === 'failed') return false;
    setTimeout(function () {
      getStatus(res.data.task_id);
    }, 1000);
  })
  .fail((err) => {
    console.log(err);
  });
}

Update the view:

@main_blueprint.route("/tasks/<task_id>", methods=["GET"])
def get_status(task_id):
    with Connection(redis.from_url(current_app.config["REDIS_URL"])):
        q = Queue()
        task = q.fetch_job(task_id)
    if task:
        response_object = {
            "status": "success",
            "data": {
                "task_id": task.get_id(),
                "task_status": task.get_status(),
                "task_result": task.result,
            },
        }
    else:
        response_object = {"status": "error"}
    return jsonify(response_object)

Add a new task to the queue:

$ curl -F type=1 http://localhost:5004/tasks

Then, grab the task_id from the response and call the updated endpoint to view the status:

$ curl http://localhost:5004/tasks/5819789f-ebd7-4e67-afc3-5621c28acf02

{
  "data": {
    "task_id": "5819789f-ebd7-4e67-afc3-5621c28acf02",
    "task_result": true,
    "task_status": "finished"
  },
  "status": "success"
}

Test it out in the browser as well:

flask, redis queue, docker

Dashboard

RQ Dashboard is a lightweight, web-based monitoring system for Redis Queue.

To set up, first add a new directory to the "project" directory called "dashboard". Then, add a new Dockerfile to that newly created directory:

FROM python:3.10-alpine

RUN pip install rq-dashboard

# https://github.com/rq/rq/issues/1469
RUN pip uninstall -y click
RUN pip install click==7.1.2

EXPOSE 9181

CMD ["rq-dashboard"]

Simply add the service to the docker-compose.yml file like so:

version: '3.8'

services:

  web:
    build: .
    image: web
    container_name: web
    ports:
      - 5004:5000
    command: python manage.py run -h 0.0.0.0
    volumes:
      - .:/usr/src/app
    environment:
      - FLASK_DEBUG=1
      - APP_SETTINGS=project.server.config.DevelopmentConfig
    depends_on:
      - redis

  worker:
    image: web
    command: python manage.py run_worker
    volumes:
      - .:/usr/src/app
    environment:
      - APP_SETTINGS=project.server.config.DevelopmentConfig
    depends_on:
      - redis

  redis:
    image: redis:6.2-alpine

  dashboard:
    build: ./project/dashboard
    image: dashboard
    container_name: dashboard
    ports:
      - 9181:9181
    command: rq-dashboard -H redis
    depends_on:
      - redis

Build the image and spin up the container:

$ docker-compose up -d --build

Navigate to http://localhost:9181 to view the dashboard:

rq dashboard

Kick off a few jobs to fully test the dashboard:

rq dashboard

Try adding a few more workers to see how that affects things:

$ docker-compose up -d --build --scale worker=3

Conclusion

This has been a basic guide on how to configure Redis Queue to run long-running tasks in a Flask app. You should let the queue handle any processes that could block or slow down the user-facing code.

Looking for some challenges?

Deploy this application across a number of DigitalOcean droplets using Kubernetes or Docker Swarm.
Write unit tests for the new endpoints. (Mock out the Redis instance with fakeredis)
Instead of polling the server, try using Flask-SocketIO to open up a websocket connection.

Grab the code from the repo.

Original article source at: https://testdriven.io/

#flask #redis #queue

How to Configure Redis Queue (RQ) To Handle Running Tasks in A Flask

What is GEEK

Buddha Community

Redis Transactions & Long-Running Lua Scripts

Transactional Nature of Redis Lua Scripts

The ‘lua-time-limit’ Value

How to Configure Redis Queue (RQ) To Handle Running Tasks in A Flask

Background Fetch for React Native Apps

react-native-background-fetch

iOS

Android

Installing the plugin

With yarn

With npm

Setup Guides

iOS Setup

react-native >= 0.60

Android Setup

react-native >= 0.60

Example

Executing Custom Tasks

⚠️ iOS:

Config

Common Options

@param {Integer} minimumFetchInterval [15]

@param {Integer} delay (milliseconds)

@param {Boolean} periodic [false]

Android Options

@config {Boolean} stopOnTerminate [true]

@config {Boolean} startOnBoot [false]

@config {Boolean} forceAlarmManager [false]

@config {Boolean} enableHeadless [false]

@config {integer} requiredNetworkType [BackgroundFetch.NETWORK_TYPE_NONE]

@config {Boolean} requiresBatteryNotLow [false]

@config {Boolean} requiresStorageNotLow [false]

@config {Boolean} requiresCharging [false]

@config {Boolean} requiresDeviceIdle [false]

Methods

Debugging

iOS

🆕 BGTaskScheduler API for iOS 13+

Simulating task-timeout events

Old BackgroundFetch API

Android

Configure Redis Queue to Handle Long-Running Tasks in A Flask App

How to Asynchronous Tasks with Flask and Redis Queue

Objectives

Workflow

Project Setup

Trigger a Task

Redis Queue

Task Status

Dashboard

Conclusion

With `yarn`

With `npm`

`react-native >= 0.60`

`react-native >= 0.60`

`@param {Integer} minimumFetchInterval [15]`

`@param {Integer} delay (milliseconds)`

`@param {Boolean} periodic [false]`

`@config {Boolean} stopOnTerminate [true]`

`@config {Boolean} startOnBoot [false]`

`@config {Boolean} forceAlarmManager [false]`

`@config {Boolean} enableHeadless [false]`

`@config {integer} requiredNetworkType [BackgroundFetch.NETWORK_TYPE_NONE]`

`@config {Boolean} requiresBatteryNotLow [false]`

`@config {Boolean} requiresStorageNotLow [false]`

`@config {Boolean} requiresCharging [false]`

`@config {Boolean} requiresDeviceIdle [false]`

🆕 `BGTaskScheduler` API for iOS 13+

Old `BackgroundFetch` API