How to set up a clean Python package structure

How to set up a clean Python package structure

The goal of this article is to describe a clean package structure, making it easier for developers to test, build, and publish it, writing as few configurations as possible, while taking advantage of conventions.

According to developers, Python is among the top five programming languages in 2019. Based on the strength of its open-source community and high adoption levels in emerging fields such as big data, analytics, and machine learning, no one should be surprised when noticing its popularity growing in the coming years. The number of packages available for Python developers shall keep growing, too. And maybe you’ll be responsible for some of them.

When it happens, keep in mind Python is very flexible in terms of package setup … there are lots of docs and blog posts on this subject, by the way. But sometimes we may get confused among so many options, mainly when getting started to package development and distribution.

A Clean Package Structure

The proposed folders and files, including testing stuff, are presented below:

project-root
├──src/
   └──package_name
├──tests/
   └──package_name
├──.coveragerc
├──.gitignore
├──LICENSE
├──pytest.ini
├──README.md
├──setup.cfg
└──setup.py

I’ll explore all of them but .gitignore, LICENSE, and README.md since these are widely known files. Please ask Google in case you have questions about their contents.

Let me start from setup.py, which is the package’s descriptor file. It consists of a Python script where multiple properties can be set declaratively, as shown below. The properties declared in this file are recognized by package managers such as pip and IDEs such as PyCharm, which means this is a must-have for any package.

from setuptools import find_namespace_packages, setup

packages = [package for package in find_namespace_packages(where='./src', include='package_cheat_sheet.*')]

setup(
    name='python-package-cheat-sheet',
    version='1.0.0',
    author='Your Name',
    author_email='[email protected]',
    description='Python package developer\'s cheat sheet',
    platforms='Posix; MacOS X; Windows',
    packages=packages,
    package_dir={
        '': 'src'
    },
    include_package_data=True,
    install_requires=(
        'stringcase',
    ),
    classifiers=[
        'Development Status :: 1 - Alpha',
        'Natural Language :: English',
        'Programming Language :: Python',
        'Programming Language :: Python :: 3.6',
        'Programming Language :: Python :: 3.7',
    ],
)

Python package cheat sheet — initial setup file Some properties’ meanings are pretty straightforward: name, version, author … but others require a bit of explanation:

  • packages, package_dir: used to set where your package source files are located and the namespaces they declare. In the above example, src is configured as the sources rootfolder (in case it has subfolders, they’re included in the package by default), and such files declare namespaces starting by package_cheat_sheet
  • install_requires: a tuple with all dependencies your package needs to work — warning: add only operational dependencies; nothing related to testing or building should be put here (test dependencies are covered in the next section). You can also think of this as a partial replacement for requirements.txt in case you’re familiar with it.

These properties allow us to bundle only the files users need when working with the packages we provide them, which results in smaller distribution files. Also, they will download only the dependencies each package needs to run, avoiding unnecessary network and storage usage.

There’s a practical exercise to see it in action. Sample code for this article is available on GitHub (https://github.com/ricardolsmendes/python-package-cheat-sheet), and pip allows us to install packages hosted there. Please install Python 3.6+ and activate a virtualenv. Then:

pip install git+https://github.com/ricardolsmendes/python-package-cheat-sheet
pip freeze

You'll notice two packages were installed, as follows:

python-package-cheat-sheet==1.0.0
stringcase==1.2.0

The first one is declared in setup.py, available on the GitHub repo. The second is a required (operational) dependency for that package.

Now, let’s call the package_cheat_sheet.StringFormatter.format_to_snakecase method using the Python Interactive Shell:

python
>>> from package_cheat_sheet import StringFormatter
>>> print(StringFormatter.format_to_snakecase('FooBar'))
foo_bar
>>> exit()

As you can see, foo_bar is the output for StringFormatter.format_to_snakecase('FooBar'), which means the package installation works as expected. This a quick demonstration of how you can set up a Python package and make it available for users with a few lines of code.

Packages Also Need Automated Tests

Modern software relies on automated tests, and we can’t even think about starting the development of a Python package without them. Pytest is the most used library for this purpose, so let’s see how to integrate it into the package setup.

In the first exercise we wore a user’s hat — now it’s time to wear a developer’s one.

First of all, please uninstall the package gathered from GitHub, clone the full sample code, and reinstall from the local sources:

pip uninstall python-package-cheat-sheet
git clone https://github.com/ricardolsmendes/python-package-cheat-sheet.git
cd python-package-cheat-sheet
pip install --editable .

The command to trigger a test suite based on the setup file is python setup.py test. It does not use pytest by default, but there’s a way to replace the default testing tool: Create a setup.cfg file in the package’s root folder, setting an alias for the test command.

[aliases]
test=pytest
  • pytest dependencies are required from now on; otherwise, the command will fail after the alias is created. The dependencies will be added to setup.py, using distinct properties:
setup_requires=(
    'pytest-runner',
),
tests_require=(
    'pytest-cov',
)
  • pytest-runner is responsible for adding pytest support for setup tasks
  • pytest-cov will help us to generate coverage statistics for our code, as we’ll see next

Two more config files must be included in the package’s root folder: pytest.ini and .coveragerc:

  • pytest.ini contains additional parameters for pytest execution. For example, presenting coverage results both in the console and HTML files:
[pytest]
addopts=--cov --cov-report html --cov-report term-missing
  • .coveragerc controls the coverage script scope. This is pretty useful when you have folders in your project that don’t need to be monitored by the tool. In the proposed clean structure, only the src folder must be covered:
[run]
source =
    src

And we’re ready to run python setup.py test, now powered by pytest. By default, pytest looks for test files inside the tests folder. For the GitHub repo you just cloned, the expected output is:

plugins: cov-2.7.1
collected 10 items
tests\package_cheat_sheet\string_formatter_test.py ..........                                                                                                                                                                            [100%]
----------- coverage: platform win32, python 3.7.2-final-0 ---------
Name                                                 Stmts   Miss  Cover   Missing
--------------------------------------------------------------------
src\package_cheat_sheet\__init__.py                      1      0   100%
src\package_cheat_sheet\string_formatter.py             17      0   100%
tests\package_cheat_sheet\string_formatter_test.py      31      0   100%
--------------------------------------------------------------------
TOTAL                                                   49      0   100%
Coverage HTML written to dir htmlcov

Please also check htmlcov/index.html after running pytest, since HTML output helps a lot when you need a deeper understanding of the coverage reports.

python

Summary

This article presents a clean Python package structure, covering both general setup and testing instrumentation. It proposes an explicit separation of sources and tests files, using Python standards, convention over configuration, and common tools to get the job done writing as little as possible.

And that’s it!

Originally published by Ricardo Mendes at medium.com

python

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

Basic Data Types in Python | Python Web Development For Beginners

In the programming world, Data types play an important role. Each Variable is stored in different data types and responsible for various functions. Python had two different objects, and They are mutable and immutable objects.

How To Compare Tesla and Ford Company By Using Magic Methods in Python

Magic Methods are the special methods which gives us the ability to access built in syntactical features such as ‘<’, ‘>’, ‘==’, ‘+’ etc.. You must have worked with such methods without knowing them to be as magic methods. Magic methods can be identified with their names which start with __ and ends with __ like __init__, __call__, __str__ etc. These methods are also called Dunder Methods, because of their name starting and ending with Double Underscore (Dunder).

Python Programming: A Beginner’s Guide

Python is an interpreted, high-level, powerful general-purpose programming language. You may ask, Python’s a snake right? and Why is this programming language named after it?

Hire Python Developers

Are you looking for experienced, reliable, and qualified Python developers? If yes, you have reached the right place. At **[HourlyDeveloper.io](https://hourlydeveloper.io/ "HourlyDeveloper.io")**, our full-stack Python development services...

Python any: How to Check If Element is Iterable or Not

Python any() function returns True if any element of an iterable is True otherwise any() function returns False. The syntax is any().