Distributed Task Queues With Django, RabbitMQ, and Celery

Distributed Task Queues With Django, RabbitMQ, and Celery

Learn about distributed task queues for making asynchronous API requests

What happens when a user sends a request, but processing that request takes longer than the HTTP request-response cycle? What if you’re accessing multiple databases or want to return a document too large to process within the time window? What if you want to access an API, but the number of requests is throttled to a maximum of n requests per t time window?

These are part of the questions that were raised during the data collection process for my master’s thesis. For my research, microposts from Twitter were scraped via the Twitter API. Two main issues arose that are resolved by distributed task queues:

  1. The Twitter API limits requests to a maximum of 900 GET statuses/lookups per request window of 15 minutes. Data collection consisted of well over 100k requests, or 30+ hours. Mitigating this process to a server proved indispensable in the planning.
  2. Database operations, in particular the creation of instances for annotators in our server-hosted annotation tool, exceeded the request/response time window. To be able to create these instances, I needed to use a distributed task queue.

programming django software-engineering python data-science

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

Data Science With Python Training | Python Data Science Course | Intellipaat

🔵 Intellipaat Data Science with Python course: https://intellipaat.com/python-for-data-science-training/In this Data Science With Python Training video, you...

How To Build A Data Science Career In 2021

In Conversation With Dr Suman Sanyal, NIIT University,he shares his insights on how universities can contribute to this highly promising sector and what aspirants can do to build a successful data science career.

Data Science with Python Certification Training in Chennai

Enroll in our Data Science with Python training in Chennai. Best Data Science with Python Training courses in Chennai for 100% Job Placements Support.

Python for Data Science | Data Science With Python | Python Data Science Tutorial

🔥Intellipaat Python for Data Science Course: https://intellipaat.com/python-for-data-science-training/In this python for data science video you will learn e...

Applied Data Science with Python Certification Training Course -IgmGuru

Master Applied Data Science with Python and get noticed by the top Hiring Companies with IgmGuru's Data Science with Python Certification Program. Enroll Now