Morioh
Login
Dylan North

Dylan North

5 years ago

Python Django with Docker and Gitlab CI

Python Django with Docker and Gitlab CI - In this article i will describe how we set up Gitlab CI to run tests for Django project. But first couple of words about what tools we were using…

For a project I was specifically asked to build an API using Python Django. So, my first starting point was to google “django cookiecutter” which immediately brought me to this amazing cookiecutter project. What I am going to demonstrate here is how to quickly setup the project (for the sake of completeness) and use Gitlab Continuous Integration to automatically unit test, run linters, generate documentation, build a container and release it.

Setup the project

We start with initiating the project using the mentioned cookiecutter project,although you can also use another cookiecutter or build on your existing project; you probably need to make some alterations here and there. Here is a small list of prerequisites:

  • you have docker installed locally
  • you have Python installed locally
  • you have a Gitlab account and you can push using ssh keys

Now, install cookiecutter and generate the project:

pip install "cookiecutter>=1.4.0"
cookiecutter https://github.com/pydanny/cookiecutter-django

Provide the options in any way you like, so the Django project will be created. Type y when asked to include Docker (because that is why we are here!!!).

Walk trough the options for the Django Cookiecutter

Enter the project, create a git repo and push it there:

cd my_django_api
git init
git add .
git commit -m "first awesome commit"
git remote add origin git@gitlab.com:jwdobken/my-django-api.git
git push -u origin master

Obviously replace my-django-api with your project name and jwdobken with your own Gitlab account name.

You can read here how to develop running docker locally. It is something I do with all my projects of any type; the dev and production environments are more alike and it has been years since I worked with something like virtual environments and I am not missing it!

Add a test environment

Make a test environment by copying the local environment:

cp local.yml test.yml
cp requirements/local.txt requirements/test.txt
cp -r compose/local compose/test

In compose/test/django/Dockerfile change requirements/local.txt to requirements/test.txt . You can make more alterations to the test environment later.

The Gitlab-CI file

Finally we get to the meat. Here is the .gitlab-ci.yml file:

image: docker:latest
	services:
	  - docker:dind
	

	variables:
	  DOCKER_HOST: tcp://docker:2375
	  DOCKER_DRIVER: overlay2
	  CONTAINER_TEST_IMAGE: $CI_REGISTRY_IMAGE:$CI_BUILD_REF_SLUG
	  CONTAINER_RELEASE_IMAGE: $CI_REGISTRY_IMAGE:latest
	

	stages:
	  - test
	  - build
	  - release
	

	test:
	  stage: test
	  image: tiangolo/docker-with-compose
	  script:
	    - docker-compose -f test.yml build
	    # - docker-compose -f test.yml run --rm django pydocstyle
	    - docker-compose -f test.yml run --rm django flake8
	    - docker-compose -f test.yml run django coverage run -m pytest
	    - docker-compose -f local.yml run --rm django coverage html
	    - docker-compose -f local.yml run --rm django /bin/sh -c "cd docs && apk add make && make html"
	    - docker-compose -f local.yml run django coverage report
	  coverage: "/TOTAL.+ ([0-9]{1,3}%)/"
	  artifacts:
	    paths:
	      - htmlcov
	      - docs/_build
	    expire_in: 5 days
	

	build:
	  stage: build
	  script:
	    - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
	    - docker build -t $CONTAINER_TEST_IMAGE -f compose/production/django/Dockerfile .
	    - docker push $CONTAINER_TEST_IMAGE
	

	release:
	  stage: release
	  script:
	    - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
	    - docker pull $CONTAINER_TEST_IMAGE
	    - docker tag $CONTAINER_TEST_IMAGE $CONTAINER_RELEASE_IMAGE
	    - docker push $CONTAINER_RELEASE_IMAGE
	  only:
	    - master
	

	pages:
	  stage: release
	  script:
	    - mkdir -p public/coverage
	    - mv htmlcov/* public/coverage
	    - mkdir -p public/docs
	    - mv -v docs/_build/html/* public/docs
	  artifacts:
	    paths:
	      - public
	    expire_in: 30 days
	  only:
	    - master

The test stage builds the container stack in the test environment, runs the unit tests with flake8, copies the html coverage report and catches the total coverage. Also, we misuse the test build to generate the sphinxdocumentation for which we need to install Make.

The build stage builds the production container and pushes it to the Gitlab container registry.

The release stage pulls the build container and tags it as the latest release before pushing it to the container registry.

The page part publishes the test and documentation artifacts with Gitlab Pages.

Push your code to Gitlab where you should find a running pipeline.

the pipeline is running

the pipeline has succesfully finished

In the container registry of the project you can find two images: the latest master image and the latest release image. The page itself explains how to pull images from here to anywhere.

Badges

Gitlab enables badges on the repo page to give any specific information. On the Gitlab project page, go to Settings, go to Badges. Here you can add the following badges:

Pipeline status of the master branch:

Test coverage and report:

Documentation:

Note that the URL link of Gitlab Pages, for the test coverage report and documentation, is not straightforward. Replace your username with a groupname if you work in a group. In the case of a subgroup, provide the full path. Usually I end up with a bit of trial-and-error; this article explains most of it.

status badges shown on the repo page

Pydocstyle

Finally, I highly recommend to check the existence and quality of your docstrings using pydocstyle. Add the following line to requirements/test.txtand requirements/local.txt in the Code quality section:

pydocstyle==3.0.0  # https://github.com/PyCQA/pydocstyle

Add the following lines to setup.cfg to configure pydocstyle:

[pydocstyle]
match = (?!\d{4}_).*\.py

And finally add the following line to .gitlab-ci.yml in the script section of the test stage (just after the build):

- docker-compose -f test.yml run — rm django pydocstyle

Be warned that the project does not comply with pydocstyle by default, so you will have to complete the code with docstrings to pass the test again.

Finally

We now have a fresh Django project with a neat CI pipeline on Gitlab for automated unit tests, documentation and container image release. You can later include Continuous Deployment to the pipeline; I left it out of the scope, because it depends too much on your production environment. You can read more about Gitlab CI here.

Currently the pipeline is quite slow mainy caused by the build of the images. The running time can be accelerated by caching dependencies.

There is a soft (10GB) size restriction for registry on GitLab.com, as part of the repository size limit. Therefore, when the number of images increases, you probably need to archive old images manually.

===================================================================

Thanks for reading :heart: If you liked this post, share it with all of your programming buddies! Follow me on Facebook | Twitter

#python #django

Python Django with Docker and Gitlab CI
8 Likes155.45 GEEK
2.00 Comments