Python Django with Docker and Gitlab CI

Python Django with Docker and Gitlab CI

Python Django with Docker and Gitlab CI - In this article i will describe how we set up Gitlab CI to run tests for Django project. But first couple of words about what tools we were using...✈️✈️✈️✈️✈️

Python Django with Docker and Gitlab CI - In this article i will describe how we set up Gitlab CI to run tests for Django project. But first couple of words about what tools we were using..

For a project I was specifically asked to build an API using Python Django. So, my first starting point was to google “django cookiecutter” which immediately brought me to this amazing cookiecutter project. What I am going to demonstrate here is how to quickly setup the project (for the sake of completeness) and use Gitlab Continuous Integration to automatically unit test, run linters, generate documentation, build a container and release it.

Setup the project

We start with initiating the project using the mentioned cookiecutter project,although you can also use another cookiecutter or build on your existing project; you probably need to make some alterations here and there. Here is a small list of prerequisites:

  • you have docker installed locally
  • you have Python installed locally
  • you have a Gitlab account and you can push using ssh keys

Now, install cookiecutter and generate the project:

pip install "cookiecutter>=1.4.0"
cookiecutter https://github.com/pydanny/cookiecutter-django

Provide the options in any way you like, so the Django project will be created. Type y when asked to include Docker (because that is why we are here!!!).

Walk trough the options for the Django Cookiecutter

Enter the project, create a git repo and push it there:

cd my_django_api
git init
git add .
git commit -m "first awesome commit"
git remote add origin [email protected]:jwdobken/my-django-api.git
git push -u origin master

Obviously replace my-django-api with your project name and jwdobken with your own Gitlab account name.

You can read here how to develop running docker locally. It is something I do with all my projects of any type; the dev and production environments are more alike and it has been years since I worked with something like virtual environments and I am not missing it!

Add a test environment

Make a test environment by copying the local environment:

cp local.yml test.yml
cp requirements/local.txt requirements/test.txt
cp -r compose/local compose/test

In compose/test/django/Dockerfile change requirements/local.txt to requirements/test.txt . You can make more alterations to the test environment later.

The Gitlab-CI file

Finally we get to the meat. Here is the .gitlab-ci.yml file:

image: docker:latest
	services:
	  - docker:dind
	

	variables:
	  DOCKER_HOST: tcp://docker:2375
	  DOCKER_DRIVER: overlay2
	  CONTAINER_TEST_IMAGE: $CI_REGISTRY_IMAGE:$CI_BUILD_REF_SLUG
	  CONTAINER_RELEASE_IMAGE: $CI_REGISTRY_IMAGE:latest
	

	stages:
	  - test
	  - build
	  - release
	

	test:
	  stage: test
	  image: tiangolo/docker-with-compose
	  script:
	    - docker-compose -f test.yml build
	    # - docker-compose -f test.yml run --rm django pydocstyle
	    - docker-compose -f test.yml run --rm django flake8
	    - docker-compose -f test.yml run django coverage run -m pytest
	    - docker-compose -f local.yml run --rm django coverage html
	    - docker-compose -f local.yml run --rm django /bin/sh -c "cd docs && apk add make && make html"
	    - docker-compose -f local.yml run django coverage report
	  coverage: "/TOTAL.+ ([0-9]{1,3}%)/"
	  artifacts:
	    paths:
	      - htmlcov
	      - docs/_build
	    expire_in: 5 days
	

	build:
	  stage: build
	  script:
	    - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
	    - docker build -t $CONTAINER_TEST_IMAGE -f compose/production/django/Dockerfile .
	    - docker push $CONTAINER_TEST_IMAGE
	

	release:
	  stage: release
	  script:
	    - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
	    - docker pull $CONTAINER_TEST_IMAGE
	    - docker tag $CONTAINER_TEST_IMAGE $CONTAINER_RELEASE_IMAGE
	    - docker push $CONTAINER_RELEASE_IMAGE
	  only:
	    - master
	

	pages:
	  stage: release
	  script:
	    - mkdir -p public/coverage
	    - mv htmlcov/* public/coverage
	    - mkdir -p public/docs
	    - mv -v docs/_build/html/* public/docs
	  artifacts:
	    paths:
	      - public
	    expire_in: 30 days
	  only:
	    - master

The test stage builds the container stack in the test environment, runs the unit tests with flake8, copies the html coverage report and catches the total coverage. Also, we misuse the test build to generate the sphinxdocumentation for which we need to install Make.

The build stage builds the production container and pushes it to the Gitlab container registry.

The release stage pulls the build container and tags it as the latest release before pushing it to the container registry.

The page part publishes the test and documentation artifacts with Gitlab Pages.

Push your code to Gitlab where you should find a running pipeline.

the pipeline is running

the pipeline has succesfully finished

In the container registry of the project you can find two images: the latest master image and the latest release image. The page itself explains how to pull images from here to anywhere.

Badges

Gitlab enables badges on the repo page to give any specific information. On the Gitlab project page, go to Settings, go to Badges. Here you can add the following badges:

Pipeline status of the master branch:

  • Link: https://gitlab.com/%{project_path}/pipelines

  • Badge image URL: https://gitlab.com/%{project_path}/badges/%{default_branch}/pipeline.svg

Test coverage and report:

  • Link: https://.gitlab.io/my-django-api/coverage/

  • Badge image URL: https://gitlab.com/%{project_path}/badges/%{default_branch}/coverage.svg?job=test

Documentation:

  • Link:https://.gitlab.io/my-django-api/docs/

  • Badge image URL: https://img.shields.io/static/v1.svg?label=sphinx&message=documentation&color=blue

Note that the URL link of Gitlab Pages, for the test coverage report and documentation, is not straightforward. Replace your username with a groupname if you work in a group. In the case of a subgroup, provide the full path. Usually I end up with a bit of trial-and-error; this article explains most of it.

status badges shown on the repo page

Pydocstyle

Finally, I highly recommend to check the existence and quality of your docstrings using pydocstyle. Add the following line to requirements/test.txtand requirements/local.txt in the Code quality section:

pydocstyle==3.0.0  # https://github.com/PyCQA/pydocstyle

Add the following lines to setup.cfg to configure pydocstyle:

[pydocstyle]
match = (?!\d{4}_).*\.py

And finally add the following line to .gitlab-ci.yml in the script section of the test stage (just after the build):

- docker-compose -f test.yml run — rm django pydocstyle

Be warned that the project does not comply with pydocstyle by default, so you will have to complete the code with docstrings to pass the test again.

Finally

We now have a fresh Django project with a neat CI pipeline on Gitlab for automated unit tests, documentation and container image release. You can later include Continuous Deployment to the pipeline; I left it out of the scope, because it depends too much on your production environment. You can read more about Gitlab CI here.

Currently the pipeline is quite slow mainy caused by the build of the images. The running time can be accelerated by caching dependencies.

There is a soft (10GB) size restriction for registry on GitLab.com, as part of the repository size limit. Therefore, when the number of images increases, you probably need to archive old images manually.

===================================================================

Thanks for reading :heart: If you liked this post, share it with all of your programming buddies! Follow me on Facebook | Twitter

Python Django Tutorial | Django Course

Python Django Tutorial | Django Course

🔥Intellipaat Django course: https://intellipaat.com/python-django-training/ 👉This Python Django tutorial will help you learn what is django web development &...

This Python Django tutorial will help you learn what is django web development & application, what is django and introduction to django framework, how to install django and start programming, how to create a django project and how to build django app. There is a short django project as well to master this python django framework.

Why should you watch this Django tutorial?

You can learn Django much faster than any other programming language and this Django tutorial helps you do just that. Our Django tutorial has been created with extensive inputs from the industry so that you can learn Django and apply it for real world scenarios.

Developing Restful APIs with Python, Django and Django Rest Framework

Developing Restful APIs with Python, Django and Django Rest Framework

This article is a definitive guide for starters who want to develop projects with RESTful APIs using Python, Django and Django Rest Framework.

This article is a definitive guide for starters who want to develop projects with RESTful APIs using Python, Django and Django Rest Framework.

Introduction
  • Django is a web framework written in Python
  • Python is an interpreted high-level programming language for general-purpose programming
  • API or Application Programming Interface is a set of rules and mechanisms by which one application or component interacts with the others
  • REST or Representational State Transfer is a software architecture

REST APIs

As described in a dissertion by Roy Fielding,

REST is an "architectural style' that basically exploits the existing technology and protocols of the web.
In simple definition, it is the data representation for a client in the format that is suitable for it.

Hence, RESTful + API is a commonly used terminology for the implementation of such architecture and constraints (eg. in web services).

Here is an example GET request from GitHub's API

$ curl https://api.github.com/users/joshuadeguzman

You will see an output similar to this

{
  "login": "joshuadeguzman",
  "id": 20706361,
  "node_id": "MDQ6VXNlcjIwNzA2MzYx",
  "avatar_url": "https://avatars1.githubusercontent.com/u/20706361?v=4",
  "gravatar_id": "",
  "url": "https://api.github.com/users/joshuadeguzman",
  "html_url": "https://github.com/joshuadeguzman",
  "followers_url": "https://api.github.com/users/joshuadeguzman/followers",
  "following_url": "https://api.github.com/users/joshuadeguzman/following{/other_user}",
  "gists_url": "https://api.github.com/users/joshuadeguzman/gists{/gist_id}",
  "starred_url": "https://api.github.com/users/joshuadeguzman/starred{/owner}{/repo}",
  "subscriptions_url": "https://api.github.com/users/joshuadeguzman/subscriptions",
  "organizations_url": "https://api.github.com/users/joshuadeguzman/orgs",
  "repos_url": "https://api.github.com/users/joshuadeguzman/repos",
  "events_url": "https://api.github.com/users/joshuadeguzman/events{/privacy}",
  "received_events_url": "https://api.github.com/users/joshuadeguzman/received_events",
  "type": "User",
  "site_admin": false,
  "name": "Joshua de Guzman",
  "company": "@freelancer",
  "blog": "https://joshuadeguzman.me",
  "location": "Manila, PH",
  "email": null,
  "hireable": true,
  "bio": "Android Engineer at @freelancer. Building tools for humans.",
  "public_repos": 75,
  "public_gists": 2,
  "followers": 38,
  "following": 10,
  "created_at": "2016-07-28T15:19:54Z",
  "updated_at": "2019-06-16T10:26:39Z"
}

Shown above is a data set in JSON format.

JSON or JavaScript Object Notation is an open-standard file format that uses human-readable text to transmit data objects consisting of attribute–value pairs and array data types.
Other formats include XML, INI, CSV, etc. But today, JSON is widely use for its structure is intuitive, making it comfortable to read and map domain objects no matter what programming language is being used.

Python and Django

Python, according to its creator, Guido van Rossum, is a

high-level programming language, and its core design philosophy is all about code readability and a syntax which allows programmers to express concepts in a few lines of code.
Python uses english like words representation (eg. for methods, reserve keywords and control flow) that makes it easier for any beginner to jump right into it. It also features dynamic type system meaning it verifies the type safety of program at runtime. It also does automatic memory management.

print(5 + 5) # This will result to 10

Django is a high-level Python Web Framework that enables developers to deliver projects on time with clean and pragmatic design.

Its flagship features include a design for fast development, a secure and scalable product.

Quick Django Overview

Django's way of propagating changes to your database schema is by means of its migration modules.

Sample User model

from django.db import models

class User(models.Model):
    first_name = models.CharField(max_length=50)
    middle_name = models.CharField(max_length=50)
    last_name = models.CharField(max_length=50)

    def __str__(self):
        return self.name

If any changes are made on your models, run makemigrations

$ python manage.py makemigrations

Finally, you can synchronize the database with the set of models and migrations

$ python manage.py migrate

REST APIs with Django Rest Framework

DRF or Django REST Framework is a powerful and flexible toolkit for building Web APIs. It helps the developers to not reinvent the wheel by rolling out complex and solid REST API from scratch by themselves. Because when your projects become more and more complex, you will soon realise the need of using DRF or other helpful rest framework.

1. Installation & Project Setup

Create project directory

$ mkdir djangoapi

Install virtualenv via pip

A virtual environment enables a project to have additional libraries or changes in packages within its environment without disturbing global or libraries of other environments.

pip is a package management system used to install and manage software packages written in Python.

$ pip install virtualenv

To create an environment folder in your project's directory

$ cd djangoapi
$ virtualenv venv

To activate the environment

$ source venv/bin/activate

To undo these changes to your path, simply run deactivate. More on virtualenv.

Install django, djangorestframework

$ pip install django
$ pip install djangorestframework

Creating a django project

$ django-admin startproject blog

Running your project

$ python manage.py runserver

System check identified no issues (0 silenced).

You have 15 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.

August 16, 2018 - 09:58:36
Django version 2.1, using settings 'blog.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

The unapplied migrations refer to the default migration files included when you start a django project.

To synchronize these migration files, simply run migrate

$ python manage.py migrate

Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying sessions.0001_initial... OK

The default database in our project is currently set to SQLite named db.sqlite3.

Creating a django project's app

$ cd blog
$ python manage.py startapp posts

The project structure should look like

$ find .
./posts
./posts/migrations
./posts/migrations/__init__.py
./posts/models.py
./posts/__init__.py
./posts/apps.py
./posts/admin.py
./posts/tests.py
./posts/views.py
./db.sqlite3
./blog
./blog/__init__.py
./blog/__pycache__
./blog/__pycache__/settings.cpython-36.pyc
./blog/__pycache__/wsgi.cpython-36.pyc
./blog/__pycache__/__init__.cpython-36.pyc
./blog/__pycache__/urls.cpython-36.pyc
./blog/settings.py
./blog/urls.py
./blog/wsgi.py
./manage.py

2. Model

Each model instance is a definitive source of the information about your data. In general, each model pertains to a single table in your database.

# djangoapi/blog/posts/models.py
from django.db import models

# Create your models here.

class Post(models.Model):
    title = models.CharField(max_length=255)
    content = models.TextField()
    is_featured = models.BooleanField(default=False)

    def __str__(self):
        return self.name

__str__ is called by the str() built-in function and by the print statement to compute the "informal" string representation of an object.
If you try running makemigrations, django won't see those changes yet.

$ No changes detected

To solve this, add your posts app to your project's installed apps.

# djangoapi/blog/blog/settings.py

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'posts' # Add it here
]

To continue with the migration of models

$ python manage.py makemigrations

Migrations for 'posts':
  posts/migrations/0001_initial.py
    - Create model Post

$ python manage.py migrate

Operations to perform:
  Apply all migrations: admin, auth, contenttypes, posts, sessions
Running migrations:
  Applying posts.0001_initial... OK


3. Serialization

Serializers allow data structure or object state to be translated into a format that can be stored or transmitted and be reconstructed later on.

Create API's serializers.py and views.py files and isolate them like this

# posts/api
posts/api/serializers.py
posts/api/views.py

# posts/migrations
posts/migrations/

# posts
posts/admin.py
posts/apps.py
posts/models.py
posts/tests.py
posts/views.py
# posts/api/serializers.py

from ..models import Post
from rest_framework import serializers

class PostSerializer(serializers.ModelSerializer):
    class Meta:
        model = Post
        fields = ('title', 'content', 'is_featured') # if not declared, all fields of the model will be shown

In this tutorial we have used ModelSerializer, more on this.

4. Views

A view function, or view for short, is a Python function that takes a Web request and returns a Web response.

# posts/api/views.py

from ..models import Post
from . import serializers
from rest_framework import generics, status
from rest_framework.response import Response

class PostListView(generics.ListAPIView):
    queryset = Post.objects.all()
    serializer_class = serializers.PostSerializer

As seen above, ListAPIView is used for read-only endpoints to represent a collection of model instances.

In this code snippet, we use generics view methods from the rest_framework, more on this.

5. URLs

This is where we setup our routes or URL paths to our designated views in which we expect specific responses for each.

# posts/urls.py

from django.urls import path
from . import views
from .api import views

urlpatterns = [
    path('', views.PostListView.as_view(), name=None)
]

6. Finalizing Setup

Ensure that the rest_framework is added to our project's apps.

# djangoapi/blog/blog/settings.py

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'rest_framework', # Add it here
    'posts'
]

7. Django Admin

Since we haven't setup our POST requests yet, we will be populating the database through django's admin panel.

To do that, create a superuser account admin with password 1234password.

$ python manage.py createsuperuser --email [email protected] --username admin

Password:
Password (again):
This password is too common.
Bypass password validation and create user anyway? [y/N]: y
Superuser created successfully.

Register the model in the admin panel.

# posts/admin.py

from django.contrib import admin
from .models import Post

# Register your models here.
admin.site.register(Post)

That's it. Visit the admin panel and update posts model's records. More on this.

8. Testing our API

$ python manage.py runserver
GET /api/v1/posts/
HTTP 200 OK
Allow: GET, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

[
    {
        "title": "Example Post #1",
        "content": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.",
        "is_featured": false
    },
    {
        "title": "Example Post #2",
        "content": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.",
        "is_featured": true
    }
]

Great. Now it's time for us to update our views and finish the standard CRUD operations.

9. Adding more views

POST is a method used for creating (sometimes updating) a resource in the database.

# posts/api/views.py

from ..models import Post
from . import serializers
from rest_framework import generics, status
from rest_framework.response import Response

class PostCreateView(generics.CreateAPIView):
    queryset = Post.objects.all()
    serializer_class = serializers.PostSerializer

    def create(self, request, *args, **kwargs):
        super(PostCreateView, self).create(request, args, kwargs)
        response = {"status_code": status.HTTP_200_OK,
                    "message": "Successfully created",
                    "result": request.data}
        return Response(response)

Most often, we separate List and Create view classes when we want to expose a list of data set while easily preventing a certain request to POST or create a resource in the database for that specific List view.

Usecase always varies for apps, you are opt to use ListCreateAPIView or even ViewSets for combining the logic for a set of related views.

Optional: Since we want to display the data in a more systematic way, we override create method and map our inline custom response handler.

Adding more views with methods GET, PATCH, DELETE to handle a specific blog post detail.

class PostDetailView(generics.RetrieveUpdateDestroyAPIView):
    queryset = Post.objects.all()
    serializer_class = serializers.PostSerializer

    def retrieve(self, request, *args, **kwargs):
        super(PostDetailView, self).retrieve(request, args, kwargs)
        instance = self.get_object()
        serializer = self.get_serializer(instance)
        data = serializer.data
        response = {"status_code": status.HTTP_200_OK,
                    "message": "Successfully retrieved",
                    "result": data}
        return Response(response)

    def patch(self, request, *args, **kwargs):
        super(PostDetailView, self).patch(request, args, kwargs)
        instance = self.get_object()
        serializer = self.get_serializer(instance)
        data = serializer.data
        response = {"status_code": status.HTTP_200_OK,
                    "message": "Successfully updated",
                    "result": data}
        return Response(response)

    def delete(self, request, *args, **kwargs):
        super(PostDetailView, self).delete(request, args, kwargs)
        response = {"status_code": status.HTTP_200_OK,
                    "message": "Successfully deleted"}
        return Response(response)

10. Updating URLs

# posts/urls.py

from django.urls import path
from . import views
from .api import views

urlpatterns = [
    path('', views.PostListView.as_view(), name=None),
    path('create/', views.PostCreateView.as_view(), name=None),
    path('<int:pk>/', views.PostDetailView.as_view(), name=None)
]

Now you can send requests to your API via Postman, your app or do a GETrequests from your browser, examples:

POST /api/v1/posts/create/
HTTP 200 OK
Allow: POST, OPTIONS
Content-Type: application/json
Vary: Accept

{
    "status_code": 200,
    "message": "Successfully created",
    "result": {
        "csrfmiddlewaretoken": "rnSUN3XOIghnXA0yKghnQgxg0do39xhorYene5ALw3gWGThK5MjG6YjL8VUb7v2h",
        "title": "Creating a resource",
        "content": "Howdy mate!"
    }
}
GET /api/v1/posts/1/
HTTP 200 OK
Allow: GET, PUT, PATCH, DELETE, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

{
    "status_code": 200,
    "message": "Successfully retrieved",
    "result": {
        "title": "Sample Post",
        "content": "Sample Post Content",
        "is_featured": false
    }
}

That's it. You have successfully managed to develop RESTful APIs with DRF! Cheers!

Source code

Available on GitHub.