The Background

In the few months ago, i have working on my task to create an export and import data feature. I was creating the feature, testing, and running smoothly. Until a few days of enjoying the feature, the enemy is coming to broke my comfort. The enemy is Thread Blocking. Yep, the export and import feature running on main thread and blocking another processes that running on it. I have googling, stack overflowing, and diving in the internet to find the solution. A bit keyword but valuable coming from my engineering lead. Use Celery to run the processes in another thread. Aha, i feel found the good solution and my brain try to assume and imagine how powerful celery is. I try to find the sample how to integrate my application with celery, try to googling and stack overflowing again. Until this article published, i haven’t found the article with complete solution.

According to my experience, i try to make some complete article to help another people who want to build the similar feature.

Prerequisites

My company build the application using Django Web Framework as our main framework to build our web application and RESTful API.

The complete prerequisites is listing below:

Django

Django is one of the best framework i ever seen to build an application faster. This framework based on Python and provide rich feature and module that help us to building, testing, securing, and deploying our application. You can found the detail description about Django in the official Django site.

Celery

Celery is one of the powerful python library to help us improve the performance of our application. According to the official celery documentation site, celery is . . .

Celery is a simple, flexible, and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system.

Pandas

Pandas is one of the most python library that used by Data Scientist to processing the large dataset. Pandas has rich feature that allow us to load data from several document format such us Microsoft Excel 2007, Microsoft Excel 2010, CSV, Plain Text, and another unlisted format. Find the complete description of pandas in the official documentation.

Openpyxl

Openpyxl is one of python library that allow us to modify Excel file. The main reason why i choose openpyxl is this library able to modify the existing Excel document to inserting row by row, inserting into specific row and column, and easy to styling the cell. I love to use this library in my application.

celery-progress

Drop in, dependency-free progress bars for your Django/Celery applications. Super simple setup. Lots of customization available.

The complete application’s requirements could found here.

Talk is cheap, show me the code ~ Linus Torvald

The Code

To starting the code, i assume you have a basic knowledge about Django and understand how to configure Django project.

Take the coffee and stay reading . . .

1. Creating project and install the requirements

Create Django project and place the file as your favorite directory structuring. If your project already created, install the listed requirements above or found the complete requirements here.

Run the following command

pip install requirements.txt

2. Configure application settings

After all of the requirements installed, configure your application settings by register the app_name of the library into the settings.py in specific location at INSTALLED_APPS section.

....

INSTALLED_APPS = [
     "...."
     "celery_progress",
     "django_extensions",
     "corsheaders",
     "core",
]

#pandas #web-development #django #python3 #python