Open sourcing apiron: A Python package for declarative RESTful API interaction

This post originally appeared&nbsp;<a href="https://medium.com/build-smarter/open-sourcing-apiron-1f2010393675" target="_blank">on Build Smarter</a>.

This post originally appeared on Build Smarter.


At ITHAKA our web teams write applications that each interact with a large handful of services—sometimes as many as ten. Each of those services provide multiple endpoints, each with their own set of path variables and query parameters.

Gathering data from multiple services has become a ubiquitous task for web application developers. The complexity can grow quickly: calling an API endpoint with multiple parameter sets, calling multiple API endpoints, calling multiple endpoints in multiple APIs. While the business logic can get hairy, the code to interact with those APIs doesn’t have to.

We created a module some time ago for low-level HTTP interactions, and use it throughout our code base. For a good while, though, the actual details of each service call—the service name, endpoint path, query parameters—were scattered throughout the code. This inevitably led to duplication as well as a bug or two when we made an update in one place and forgot about the other.

To reduce the pains from this, we eventually took stock of these scattered configurations and centralized them in one registry module. This module essentially contains a giant dictionary of all the services we interact with:

service_endpoints = {
    'CONTENT_SERVICE': {
        'SERVICE': 'content-service',
        'METADATA_ENDPOINT': '/content/{id}',
        'CITATION_ENDPOINT': '/citation/{citation_type}/{id}',
    },
    'SEARCH_SERVICE': {
        'SERVICE': 'search',
        'SEARCH_ENDPOINT': '/search',
        'EXCERPTS_ENDPOINT': '/excerpt?contentId={content_id}',
    },
    ...
}

Each service has a 'SERVICE' key containing the name of the service used to discover hosts, and some number of '*_ENDPOINT' keys that describe an endpoint and its parameters. Calling these services looks like this:

from http import make_get_request_with_timeout
from services.registry import service_endpoints

CONTENT_SERVICE = service_endpoints.get('CONTENT_SERVICE', {})
METADATA_ENDPOINT = CONTENT_SERVICE.get('METADATA_ENDPOINT', '')

determine content_id...

metadata = make_get_request_with_timeout(
service_name=CONTENT_SERVICE.get('SERVICE'),
endpoint=METADATA_ENDPOINT.format(id=content_id),
headers={'Accept': 'application/json'},
request_timeout=5,
)

As you can see, there are a variety of shapes to these endpoints. This solved the issue of duplication across the codebase, but we still faced a couple of problems with this approach:

  1. Strings as endpoint descriptors don’t result in structured data. This is pretty difficult to introspect or validate.
  2. Even with fully-formattable strings, sometimes a call needed to exclude a parameter all together, or add a new one. This had to be done ad-hoc after the fact.
  3. Our HTTP module still had a laundry list of methods, each with slightly different behavior and unclear names like make_get_request_fast (how fast?). Many of these methods called the same underlying methods with different default parameters, and the stack got pretty deep sometimes. Choosing the right method for a call was hard.

In order to address the high variability of behaviors and lack of structured data of this problem, we built a new paradigm for HTTP interactions that provided a declarative interface for configuring services. We wanted a few things out of it:

  1. Code describes how a service interaction looks, not the details of how to make the underlying HTTP call happen.
  2. The endpoint descriptors are structured and support introspection.
  3. Default behaviors can be declared in the service configuration, but can also be easily overridden dynamically at call time.

With these desires in mind, we came up with apiron. With apiron the same definition from above looks more like this:

from services import IthakaDiscoverableService
from apiron.endpoint import Endpoint

class ContentService(IthakaDiscoverableService):
service_name = 'content-service'

metadata = Endpoint(path='/content/{id}')
citation = Endpoint(path='/citation/{citation_type}/{id}')

And the code to call the service looks more like this:

from apiron.client import ServiceCaller, Timeout
from services import ContentService

CONTENT_SERVICE = ContentService()

determine content_id...

metadata = ServiceCaller.call(
service=CONTENT_SERVICE,
endpoint=CONTENT_SERVICE.metadata,
path_kwargs={'content_id': content_id},
headers={'Accept': 'application/json'},
timeout_spec=Timeout(read_timeout=5),
)

We can now define what ContentService looks like and easily refer back to that class whenever we need to understand its shape. Service discovery is now a plugin system. Endpoints can be introspected and have their parameters validated and enforced.

With apiron we’ve been able to replace many of our existing service calls quickly and with little pain. The code has become clearer and with the cognitive load out of the way we can begin focusing on other gains like streaming responses and data compression. It’s been nice for us, and we’d like to make it nice for you too.

You can install apiron from PyPI with pip (or your favorite package manager):

$ pip install apiron


By : Dane Hillard


How to Write Python C Extension Modules using the Python API

How to Write Python C Extension Modules using the Python API

There are several ways in which you can extend the functionality of Python. One of these is to write your Python module in C or C++. In this tutorial, you’ll discover how to use the Python API to write Python C extension modules.

You’ll learn how to:

  • Invoke C functions from within Python
  • Pass arguments from Python to C and parse them accordingly
  • Raise exceptions from C code and create custom Python exceptions in C
  • Define global constants in C and make them accessible in Python
  • Test, package, and distribute your Python C extension module

Table of Contents

  • Extending Your Python Program
  • Writing a Python Interface in C
    • Understanding fputs()
    • Writing the C Function for fputs()
    • Wrapping fputs()
    • Writing the Init Function
    • Putting It All Together
  • Packaging Your Python C Extension Module
    • Building Your Module
    • Running Your Module
  • Raising Exceptions
    • Raising Exceptions From C Code
    • Raising Custom Exceptions
  • Defining Constants
  • Testing Your Module
  • Considering Alternatives
  • Conclusion
Extending Your Python Program

One of the lesser-known yet incredibly powerful features of Python is its ability to call functions and libraries defined in compiled languages such as C or C++. This allows you to extend the capabilities of your program beyond what Python’s built-in features have to offer.

There are many languages you could choose from to extend the functionality of Python. So, why should you use C? Here are a few reasons why you might decide to build a Python C extension module:

  1. To implement new built-in object types: It’s possible to write a Python class in C, and then instantiate and extend that class from Python itself. There can be many reasons for doing this, but more often than not, performance is primarily what drives developers to turn to C. Such a situation is rare, but it’s good to know the extent to which Python can be extended.

  2. To call C library functions and system calls: Many programming languages provide interfaces to the most commonly used system calls. Still, there may be other lesser-used system calls that are only accessible through C. The os module in Python is one example.

This is not an exhaustive list, but it gives you the gist of what can be done when extending Python using C or any other language.

To write Python modules in C, you’ll need to use the Python API, which defines the various functions, macros, and variables that allow the Python interpreter to call your C code. All of these tools and more are collectively bundled in the Python.h header file.

Writing a Python Interface in C

In this tutorial, you’ll write a small wrapper for a C library function, which you’ll then invoke from within Python. Implementing a wrapper yourself will give you a better idea about when and how to use C to extend your Python module.

Understanding fputs()

fputs() is the C library function that you’ll be wrapping:

int fputs(const char *, FILE *)

This function takes two arguments:

  1. const char * is an array of characters.
  2. FILE * is a file stream pointer.

fputs() writes the character array to the file specified by the file stream and returns a non-negative value. If the operation is successful, then this value will denote the number of bytes written to the file. If there’s an error, then it returns EOF. You can read more about this C library function and its other variants in the manual page entry.

Writing the C Function for fputs()

This is a basic C program that uses fputs() to write a string to a file stream:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int main() {
    FILE *fp = fopen("write.txt", "w");
    fputs("Real Python!", fp);
    fclose(fp);
    return 1;
}

This snippet of code can be summarized as follows:

  1. Open the file write.txt.
  2. Write the string "Real Python!" to the file.

Note: The C code in this article should build on most systems. It has been tested on GCC without using any special flags.

In the following section, you’ll write a wrapper for this C function.

Wrapping fputs()

It might seem a little weird to see the full code before an explanation of how it works. However, taking a moment to inspect the final product will supplement your understanding in the following sections. The code block below shows the final wrapped version of your C code:

static PyObject *method_fputs(PyObject *self, PyObject *args) {

    char *str, *filename = NULL;

    int bytes_copied = -1;


    /* Parse arguments */

    if(!PyArg_ParseTuple(args, "ss", &str, &filename)) {

        return NULL;

    }


    FILE *fp = fopen(filename, "w");

    bytes_copied = fputs(str, fp);

    fclose(fp);


    return PyLong_FromLong(bytes_copied);

}

This code snippet references three object structures:

  1. PyObject
  2. PyArg_ParseTuple()
  3. PyLong_FromLong()

These are used for data type definition for the Python language. You’ll go through each of them now.

PyObject

PyObject is an object structure that you use to define object types for Python. All Python objects share a small number of fields that are defined using the PyObject structure. All other object types are extensions of this type.

PyObject tells the Python interpreter to treat a pointer to an object as an object. For instance, setting the return type of the above function as PyObject defines the common fields that are required by the Python interpreter in order to recognize this as a valid Python type.

Take another look at the first few lines of your C code:

static PyObject *method_fputs(PyObject *self, PyObject *args) {

    char *str, *filename = NULL;

    int bytes_copied = -1;


    /* Snip */

In line 2, you declare the argument types you wish to receive from your Python code:

  1. char *str is the string you want to write to the file stream.
  2. char *filename is the name of the file to write to.

PyArg_ParseTuple()

PyArg_ParseTuple() parses the arguments you’ll receive from your Python program into local variables:

static PyObject *method_fputs(PyObject *self, PyObject *args) {

    char *str, *filename = NULL;

    int bytes_copied = -1;


    /* Parse arguments */

    if(!PyArg_ParseTuple(args, "ss", &str, &filename)) {

        return NULL;

    }


    /* Snip */

If you look at line 6, then you’ll see that PyArg_ParseTuple() takes the following arguments:

  • args are of type PyObject.

  • "ss" is the format specifier that specifies the data type of the arguments to parse. (You can check out the official documentation for a complete reference.)

  • &str and &filename are pointers to local variables to which the parsed values will be assigned.

PyArg_ParseTuple() evaluates to false on failure. If it fails, then the function will return NULL and not proceed any further.

fputs()

As you’ve seen before, fputs() takes two arguments, one of which is the FILE * object. Since you can’t parse a Python textIOwrapper object using the Python API in C, you’ll have to use a workaround:

static PyObject *method_fputs(PyObject *self, PyObject *args) {

    char *str, *filename = NULL;

    int bytes_copied = -1;


    /* Parse arguments */

    if(!PyArg_ParseTuple(args, "ss", &str, &filename)) {

        return NULL;

    }


    FILE *fp = fopen(filename, "w");

    bytes_copied = fputs(str, fp);

    fclose(fp);


    return PyLong_FromLong(bytes_copied);

}

Here’s a breakdown of what this code does:

  • In line 10, you’re passing the name of the file that you’ll use to create a FILE * object and pass it on to the function.
  • In line 11, you call fputs() with the following arguments:
    • str is the string you want to write to the file.
    • fp is the FILE * object you defined in line 10.

You then store the return value of fputs() in bytes_copied. This integer variable will be returned to the fputs() invocation within the Python interpreter.

PyLong_FromLong(bytes_copied)

PyLong_FromLong() returns a PyLongObject, which represents an integer object in Python. You can find it at the very end of your C code:

static PyObject *method_fputs(PyObject *self, PyObject *args) {

    char *str, *filename = NULL;

    int bytes_copied = -1;


    /* Parse arguments */

    if(!PyArg_ParseTuple(args, "ss", &str, &filename)) {

        return NULL;

    }


    FILE *fp = fopen(filename, "w");

    bytes_copied = fputs(str, fp);

    fclose(fp);


    return PyLong_FromLong(bytes_copied);

}

Line 14 generates a PyLongObject for bytes_copied, the variable to be returned when the function is invoked in Python. You must return a PyObject* from your Python C extension module back to the Python interpreter.

Writing the Init Function

You’ve written the code that makes up the core functionality of your Python C extension module. However, there are still a few extra functions that are necessary to get your module up and running. You’ll need to write definitions of your module and the methods it contains, like so:

static PyMethodDef FputsMethods[] = {
    {"fputs", method_fputs, METH_VARARGS, "Python interface for fputs C library function"},
    {NULL, NULL, 0, NULL}
};


static struct PyModuleDef fputsmodule = {
    PyModuleDef_HEAD_INIT,
    "fputs",
    "Python interface for the fputs C library function",
    -1,
    FputsMethods
};

These functions include meta information about your module that will be used by the Python interpreter. Let’s go through each of the structs above to see how they work.

PyMethodDef

In order to call the methods defined in your module, you’ll need to tell the Python interpreter about them first. To do this, you can use PyMethodDef. This is a structure with 4 members representing a single method in your module.

Ideally, there will be more than one method in your Python C extension module that you want to be callable from the Python interpreter. This is why you need to define an array of PyMethodDef structs:

static PyMethodDef FputsMethods[] = {
    {"fputs", method_fputs, METH_VARARGS, "Python interface for fputs C library function"},
    {NULL, NULL, 0, NULL}
};

Each individual member of the struct holds the following info:

  • "fputs" is the name the user would write to invoke this particular function.

  • method_fputs is the name of the C function to invoke.

  • METH_VARARGS is a flag that tells the interpreter that the function will accept two arguments of type PyObject*:

    1. self is the module object.
    2. args is a tuple containing the actual arguments to your function. As explained previously, these arguments are unpacked using PyArg_ParseTuple().
  • The final string is a value to represent the method docstring.

PyModuleDef

Just as PyMethodDef holds information about the methods in your Python C extension module, the PyModuleDef struct holds information about your module itself. It is not an array of structures, but rather a single structure that’s used for module definition:

static struct PyModuleDef fputsmodule = {
    PyModuleDef_HEAD_INIT,
    "fputs",
    "Python interface for the fputs C library function",
    -1,
    FputsMethods
};

There are a total of 9 members in this struct, but not all of them are required. In the code block above, you initialize the following five:

  1. PyModuleDef_HEAD_INIT is a member of type PyModuleDef_Base, which is advised to have just this one value.

  2. "fputs" is the name of your Python C extension module.

  3. The string is the value that represents your module docstring. You can use NULL to have no docstring, or you can specify a docstring by passing a const char * as shown in the snippet above. It is of type Py_ssize_t. You can also use PyDoc_STRVAR() to define a docstring for your module.

  4. -1 is the amount of memory needed to store your program state. It’s helpful when your module is used in multiple sub-interpreters, and it can have the following values:

    • A negative value indicates that this module doesn’t have support for sub-interpreters.
    • A non-negative value enables the re-initialization of your module. It also specifies the memory requirement of your module to be allocated on each sub-interpreter session.
  5. FputsMethods is the reference to your method table. This is the array of PyMethodDef structs you defined earlier.

For more information, check out the official Python documentation on PyModuleDef.

PyMODINIT_FUNC

Now that you’ve defined your Python C extension module and method structures, it’s time to put them to use. When a Python program imports your module for the first time, it will call PyInit_fputs():

PyMODINIT_FUNC PyInit_fputs(void) {
    return PyModule_Create(&fputsmodule);
}

PyMODINIT_FUNC does 3 things implicitly when stated as the function return type:

  1. It implicitly sets the return type of the function as PyObject*.
  2. It declares any special linkages.
  3. It declares the function as extern “C.” In case you’re using C++, it tells the C++ compiler not to do name-mangling on the symbols.

PyModule_Create() will return a new module object of type PyObject *. For the argument, you’ll pass the address of the method structure that you’ve already defined previously, fputsmodule.

Note: In Python 3, your init function must return a PyObject * type. However, if you’re using Python 2, then PyMODINIT_FUNC declares the function return type as void.

Putting It All Together

Now that you’ve written the necessary parts of your Python C extension module, let’s take a step back to see how it all fits together. The following diagram shows the components of your module and how they interact with the Python interpreter:

When you import your Python C extension module, PyInit_fputs() is the first method to be invoked. However, before a reference is returned to the Python interpreter, the function makes a subsequent call to PyModule_Create(). This will initialize the structures PyModuleDef and PyMethodDef, which hold meta information about your module. It makes sense to have them ready since you’ll make use of them in your init function.

Once this is complete, a reference to the module object is finally returned to the Python interpreter. The following diagram shows the internal flow of your module:

The module object returned by PyModule_Create() has a reference to the module structure PyModuleDef, which in turn has a reference to the method table PyMethodDef. When you call a method defined in your Python C extension module, the Python interpreter uses the module object and all of the references it carries to execute the specific method. (While this isn’t exactly how the Python interpreter handles things under the hood, it’ll give you an idea of how it works.)

Similarly, you can access various other methods and properties of your module, such as the module docstring or the method docstring. These are defined inside their respective structures.

Now you have an idea of what happens when you call fputs() from the Python interpreter. The interpreter uses your module object as well as the module and method references to invoke the method. Finally, let’s take a look at how the interpreter handles the actual execution of your Python C extension module:

Once method_fputs() is invoked, the program executes the following steps:

  1. Parse the arguments you passed from the Python interpreter with PyArg_ParseTuple()
  2. Pass these arguments to fputs(), the C library function that forms the crux of your module
  3. Use PyLong_FromLong to return the value from fputs()

To see these same steps in code, take a look at method_fputs() again:

static PyObject *method_fputs(PyObject *self, PyObject *args) {

    char *str, *filename = NULL;

    int bytes_copied = -1;


    /* Parse arguments */

    if(!PyArg_ParseTuple(args, "ss", &str, &filename)) {

        return NULL;

    }


    FILE *fp = fopen(filename, "w");

    bytes_copied = fputs(str, fp);

    fclose(fp);


    return PyLong_FromLong(bytes_copied);

}

To recap, your method will parse the arguments passed to your module, send them on to fputs(), and return the results.

Packaging Your Python C Extension Module

Before you can import your new module, you first need to build it. You can do this by using the Python package distutils.

You’ll need a file called setup.py to install your application. For this tutorial, you’ll be focusing on the part specific to the Python C extension module.

A minimal setup.py file for your module should look like this:

from distutils.core import setup, Extension

def main():
    setup(name="fputs",
          version="1.0.0",
          description="Python interface for the fputs C library function",
          author="<your name>",
          author_email="[email protected]",
          ext_modules=[Extension("fputs", ["fputsmodule.c"])])

if __name__ == "__main__":
    main()

The code block above shows the standard arguments that are passed to setup(). Take a closer look at the last positional argument, ext_modules. This takes a list of objects of the Extensions class. An object of the Extensions class describes a single C or C++ extension module in a setup script. Here, you pass two keyword arguments to its constructor, namely:

  • name is the name of the module.
  • [filename] is a list of paths to files with the source code, relative to the setup script.
[Remove ads](/account/join/)

Building Your Module

Now that you have your setup.py file, you can use it to build your Python C extension module. It’s strongly advised that you use a virtual environment to avoid conflicts with your Python environment.

Navigate to the directory containing setup.py and run the following command:

$ python3 setup.py install

This command will compile and install your Python C extension module in the current directory. If there are any errors or warnings, then your program will throw them now. Make sure you fix these before you try to import your module.

By default, the Python interpreter uses clang for compiling the C code. If you want to use gcc or any other C compiler for the job, then you need to set the CC environment variable accordingly, either inside the setup script or directly on the command line. For instance, you can tell the Python interpreter to use gcc to compile and build your module this way:

$ CC=gcc python3 setup.py install

However, the Python interpreter will automatically fall back to gcc if clang is not available.

Running Your Module

Now that everything is in place, it’s time to see your module in action! Once it’s successfully built, fire up the interpreter to test run your Python C extension module:

>>> import fputs
>>> fputs.__doc__
'Python interface for the fputs C library function'
>>> fputs.__name__
'fputs'
>>> # Write to an empty file named `write.txt`
>>> fputs.fputs("Real Python!", "write.txt")
13
>>> with open("write.txt", "r") as f:
>>>     print(f.read())
'Real Python!'

Your function performs as expected! You pass a string "Real Python!" and a file to write this string to, write.txt. The call to fputs() returns the number of bytes written to the file. You can verify this by printing the contents of the file.

Also recall how you passed certain arguments to the PyModuleDef and PyMethodDef structures. You can see from this output that Python has used these structures to assign things like the function name and docstring.

With that, you have a basic version of your module ready, but there’s a lot more that you can do! You can improve your module by adding things like custom exceptions and constants.

Raising Exceptions

Python exceptions are very different from C++ exceptions. If you want to raise Python exceptions from your C extension module, then you can use the Python API to do so. Some of the functions provided by the Python API for exception raising are as follows:

You can use any of these to raise an exception. However, which to use and when depends entirely on your requirements. The Python API has all the standard exceptions pre-defined as PyObject types.

Raising Exceptions From C Code

While you can’t raise exceptions in C, the Python API will allow you to raise exceptions from your Python C extension module. Let’s test this functionality by adding PyErr_SetString() to your code. This will raise an exception whenever the length of the string to be written is less than 10 characters:

static PyObject *method_fputs(PyObject *self, PyObject *args) {

    char *str, *filename = NULL;

    int bytes_copied = -1;


    /* Parse arguments */

    if(!PyArg_ParseTuple(args, "ss", &str, &fd)) {

        return NULL;

    }


    if (strlen(str) < 10) {

        PyErr_SetString(PyExc_ValueError, "String length must be greater than 10");

        return NULL;

    }


    fp = fopen(filename, "w");

    bytes_copied = fputs(str, fp);

    fclose(fp);


    return PyLong_FromLong(bytes_copied);

}

Here, you check the length of the input string immediately after you parse the arguments and before you call fputs(). If the string passed by the user is shorter than 10 characters, then your program will raise a ValueError with a custom message. The program execution stops as soon as the exception occurs.

Note how method_fputs() returns NULL after raising the exception. This is because whenever you raise an exception using PyErr_*(), it automatically sets an internal entry in the exception table and returns it. The calling function is not required to subsequently set the entry again. For this reason, the calling function returns a value that indicates failure, usually NULL or -1. (This should also explain why there was a need to return NULL when you parse arguments in method_fputs() using PyArg_ParseTuple().)

Raising Custom Exceptions

You can also raise custom exceptions in your Python C extension module. However, things are a bit different. Previously, in PyMODINIT_FUNC, you were simply returning the instance returned by PyModule_Create and calling it a day. But for your custom exception to be accessible by the user of your module, you need to add your custom exception to your module instance before you return it:

static PyObject *StringTooShortError = NULL;

PyMODINIT_FUNC PyInit_fputs(void) {
    /* Assign module value */
    PyObject *module = PyModule_Create(&fputsmodule);

    /* Initialize new exception object */
    StringTooShortError = PyErr_NewException("fputs.StringTooShortError", NULL, NULL);

    /* Add exception object to your module */
    PyModule_AddObject(module, "StringTooShortError", StringTooShortError);

    return module;
}

As before, you start off by creating a module object. Then you create a new exception object using PyErr_NewException. This takes a string of the form module.classname as the name of the exception class that you wish to create. Choose something descriptive to make it easier for the user to interpret what has actually gone wrong.

Next, you add this to your module object using PyModule_AddObject. This takes your module object, the name of the new object being added, and the custom exception object itself as arguments. Finally, you return your module object.

Now that you’ve defined a custom exception for your module to raise, you need to update method_fputs() so that it raises the appropriate exception:

static PyObject *method_fputs(PyObject *self, PyObject *args) {

    char *str, *filename = NULL;

    int bytes_copied = -1;


    /* Parse arguments */

    if(!PyArg_ParseTuple(args, "ss", &str, &fd)) {

        return NULL;

    }


    if (strlen(str) < 10) {

        /* Passing custom exception */

        PyErr_SetString(StringTooShortError, "String length must be greater than 10");

        return NULL;

    }


    fp = fopen(filename, "w");

    bytes_copied = fputs(str, fp);

    fclose(fp);


    return PyLong_FromLong(bytes_copied);

}

After building the module with the new changes, you can test that your custom exception is working as expected by trying to write a string that is less than 10 characters in length:

>>> import fputs
>>> # Custom exception
>>> fputs.fputs("RP!", fp.fileno())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
fputs.StringTooShortError: String length must be greater than 10

When you try to write a string with fewer than 10 characters, your custom exception is raised with a message explaining what went wrong.

Defining Constants

There are cases where you’ll want to use or define constants in your Python C extension module. This is quite similar to how you defined custom exceptions in the previous section. You can define a new constant and add it to your module instance using PyModule_AddIntConstant():

PyMODINIT_FUNC PyInit_fputs(void) {
    /* Assign module value */
    PyObject *module = PyModule_Create(&fputsmodule);

    /* Add int constant by name */
    PyModule_AddIntConstant(module, "FPUTS_FLAG", 64);

    /* Define int macro */
    #define FPUTS_MACRO 256

    /* Add macro to module */
    PyModule_AddIntMacro(module, FPUTS_MACRO);

    return module;
}

This Python API function takes the following arguments:

  • The instance of your module
  • The name of the constant
  • The value of the constant

You can do the same for macros using PyModule_AddIntMacro():

PyMODINIT_FUNC PyInit_fputs(void) {
    /* Assign module value */
    PyObject *module = PyModule_Create(&fputsmodule);

    /* Add int constant by name */
    PyModule_AddIntConstant(module, "FPUTS_FLAG", 64);

    /* Define int macro */
    #define FPUTS_MACRO 256

    /* Add macro to module */
    PyModule_AddIntMacro(module, FPUTS_MACRO);

    return module;
}

This function takes the following arguments:

  • The instance of your module
  • The name of the macro that has already been defined

Note: If you want to add string constants or macros to your module, then you can use PyModule_AddStringConstant() and PyModule_AddStringMacro(), respectively.

Open up the Python interpreter to see if your constants and macros are working as expected:

>>> import fputs
>>> # Constants
>>> fputs.FPUTS_FLAG
64
>>> fputs.FPUTS_MACRO
256

Here, you can see that the constants are accessible from within the Python interpreter.

[Remove ads](/account/join/)
Testing Your Module

You can test your Python C extension module just as you would any other Python module. This can be demonstrated by writing a small test function for pytest:

import fputs

def test_copy_data():
    content_to_copy = "Real Python!"
    bytes_copied = fputs.fputs(content_to_copy, 'test_write.txt')

    with open('test_write.txt', 'r') as f:
        content_copied = f.read()

    assert content_copied == content_to_copy

In the test script above, you use fputs.fputs() to write the string "Real Python!" to an empty file named test_write.txt. Then, you read in the contents of this file and use an assert statement to compare it to what you had originally written.

You can run this test suite to make sure your module is working as expected:

$ pytest -q
test_fputs.py                                                 [100%]
1 passed in 0.03 seconds
Considering Alternatives

In this tutorial, you’ve built an interface for a C library function to understand how to write Python C extension modules. However, there are times when all you need to do is invoke some system calls or a few C library functions, and you want to avoid the overhead of writing two different languages. In these cases, you can use Python libraries such as ctypes or cffi.

These are Foreign Function libraries for Python that provide access to C library functions and data types. Though the community itself is divided as to which library is best, both have their benefits and drawbacks. In other words, either would make a good choice for any given project, but there are a few things to keep in mind when you need to decide between the two:

  • The ctypes library comes included in the Python standard library. This is very important if you want to avoid external dependencies. It allows you to write wrappers for other languages in Python.

  • The cffi library is not yet included in the standard library. This might be a dealbreaker for your particular project. In general, it’s more Pythonic in nature, but it doesn’t handle preprocessing for you.

For more information on these libraries, check out Extending Python With C Libraries and the “ctypes” Module and Interfacing Python and C: The CFFI Module.

Note: Apart from ctypes and cffi, there are various other tools available. For instance, you can also use swig and boost::Py.

Conclusion

In this tutorial, you’ve learned how to write a Python interface in the C programming language using the Python API. You wrote a Python wrapper for the fputs() C library function. You also added custom exceptions and constants to your module before building and testing it.

The Python API provides a host of features for writing complex Python interfaces in the C programming language. At the same time, libraries such as cffi or ctypes can lower the amount of overhead involved in writing Python C extension modules. Make sure you weigh all the factors before making a decision!

How to build a JSON API with Python

How to build a JSON API with Python

The JSON API specification is a powerful way for enabling communication between client and server. It specifies the structure of the requests and responses sent between the two, using the JSON format. The [JSON API...

The JSON API specification is a powerful way for enabling communication between client and server. It specifies the structure of the requests and responses sent between the two, using the JSON format.

The JSON API specification is a powerful way for enabling communication between client and server. It specifies the structure of the requests and responses sent between the two, using the JSON format.

As a data format, JSON has the advantages of being lightweight and readable. This makes it very easy to work with quickly and productively. The specification is designed to minimise the number of requests and the amount of data that needs sending between client and server.

Here, you can learn how to create a basic JSON API using Python and Flask. Then, the rest of the article will show you how to try out some of the features the JSON API specification has to offer.

Flask is a Python library that provides a 'micro-framework' for web development. It is great for rapid development as it comes with a simple-yet-extensible core functionality.

A really basic example of how to send a JSON-like response using Flask is shown below:

from flask import Flask

app = Flask(__name__)

@app.route('/')
def example():
   return '{"name":"Bob"}'

if __name__ == '__main__':
    app.run()

This article will use two add-ons for Flask:

The big picture

The end goal is to create an API that allows client-side interaction with an underlying database. There will be a couple of layers between the database and the client - a data abstraction layer and a resource manager layer.

Here's an overview of the steps involved:

  1. Define a database using Flask-SQLAlchemy
  2. Create a data abstraction with Marshmallow-JSONAPI
  3. Create resource managers with Flask-REST-JSONAPI
  4. Create URL endpoints and start the server with Flask

This example will use a simple schema describing modern artists and their relationships to different artworks.

Install everything

Before getting started, you'll need to set up the project. This involves creating a workspace and virtual environment, installing the modules required, and creating the main Python and database files for the project.

From the command line create a new directory and navigate inside.

$ mkdir flask-jsonapi-demo
$ cd flask-jsonapi-demo/

It is good practice to create virtual environments for each of your Python projects. You can skip this step, but it is strongly recommended.

$ python -m venv .venv
$ source .venv/bin/activate

Once your virtual environment has been created and activated, you can install the modules needed for this project.

$ pip install flask-rest-jsonapi flask-sqlalchemy

Everything you'll need will be installed as the requirements for these two extensions. This includes Flask itself, and SQLAlchemy.

The next step is to create a Python file and database for the project.

$ touch application.py artists.db

Create the database schema

Here, you will start modifying application.py to define and create the database schema for the project.

Open application.py in your preferred text editor. Begin by importing some modules. For clarity, modules will be imported as you go.

Next, create an object called app as an instance of the Flask class.

After that, use SQLAlchemy to connect to the database file you created. The final step is to define and create a table called artists.

from flask import Flask
from flask_sqlalchemy import SQLAlchemy

# Create a new Flask application
app = Flask(__name__)

# Set up SQLAlchemy
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:////artists.db'
db = SQLAlchemy(app)

# Define a class for the Artist table
class Artist(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    name = db.Column(db.String)
    birth_year = db.Column(db.Integer)
    genre = db.Column(db.String)

# Create the table
db.create_all()

Creating an abstraction layer

The next step uses the Marshmallow-JSONAPI module to create a logical data abstraction over the tables just defined.

The reason to create this abstraction layer is simple. It gives you more control over how your underlying data is exposed via the API. Think of this layer as a lens through which the API client can view the underlying data clearly, and only the bits they need to see.

In the code below, the data abstraction layer is defined as a class which inherits from Marshmallow-JSONAPI's Schema class. It will provide access via the API to both single records and multiple records from the artists table.

Inside this block, the Meta class defines some metadata. Specifically, the name of the URL endpoint for interacting with single records will be artist_one, where each artist will be identified by a URL parameter <id>. The name of the endpoint for interacting with many records will be artist_many.

The remaining attributes defined relate to the columns in the artists table. Here, you can control further how each is exposed via the API.

For example, when making POST requests to add new artists to the database, you can make sure the name field is mandatory by setting required=True.

And if for any reason you didn't want the birth_year field to be returned when making GET requests, you can specify so by setting load_only=True.

from marshmallow_jsonapi.flask import Schema
from marshmallow_jsonapi import fields

# Create data abstraction layer
class ArtistSchema(Schema):
    class Meta:
        type_ = 'artist'
        self_view = 'artist_one'
        self_view_kwargs = {'id': '<id>'}
        self_view_many = 'artist_many'

    id = fields.Integer()
    name = fields.Str(required=True)
    birth_year = fields.Integer(load_only=True)
    genre = fields.Str()

Create resource managers and URL endpoints

The final piece of the puzzle is to create a resource manager and corresponding endpoint for each of the routes /artists and /artists/id.

Each resource manager is defined as a class that inherits from the Flask-REST-JSONAPI classes ResourceList and ResourceDetail.

Here they take two attributes. schema is used to indicate the data abstraction layer the resource manager uses, and data_layer indicates the session and data model that will be used for the data layer.

Next, define api as an instance of Flask-REST-JSONAPI's Api class, and create the routes for the API with api.route(). This method takes three arguments - the data abstraction layer class, the endpoint name, and the URL path.

The last step is to write a main loop to launch the app in debug mode when the script is run directly. Debug mode is great for development, but it is not suitable for running in production.

# Create resource managers and endpoints

from flask_rest_jsonapi import Api, ResourceDetail, ResourceList

class ArtistMany(ResourceList):
    schema = ArtistSchema
    data_layer = {'session': db.session,
                  'model': Artist}

class ArtistOne(ResourceDetail):
    schema = ArtistSchema
    data_layer = {'session': db.session,
                  'model': Artist}

api = Api(app)
api.route(ArtistMany, 'artist_many', '/artists')
api.route(ArtistOne, 'artist_one', '/artists/<int:id>')

# main loop to run app in debug mode
if __name__ == '__main__':
    app.run(debug=True)

Make GET and POST requests

Now you can start using the API to make HTTP requests. This could be from a web browser, or from a command line tool like curl, or from within another program (e.g., a Python script using the Requests library).

To launch the server, run the application.py script with:

$ python application.py

In your browser, navigate to http://localhost:5000/artists.  You will see a JSON output of all the records in the database so far. Except, there are none.

To start adding records to the database, you can make a POST request. One way of doing this is from the command line using curl. Alternatively, you could use a tool like Insomnia, or perhaps code up a simple HTML user interface that posts data using a form.

With curl, from the command line:

curl -i -X POST -H 'Content-Type: application/json' -d '{"data":{"type":"artist", "attributes":{"name":"Salvador Dali", "birth_year":1904, "genre":"Surrealism"}}}' http://localhost:5000/artists

Now if you navigate to http://localhost:5000/artists, you will see the record you just added. If you were to add more records, they would all show here as well, as this URL path calls the artists_many endpoint.

To view just a single artist by their id number, you can navigate to the relevant URL. For example, to see the first artist, try http://localhost:5000/artists/1.

Filtering and sorting

One of the neat features of the JSON API specification is the ability to return the response in more useful ways by defining some parameters in the URL. For instance, you can sort the results according to a chosen field, or filter based on some criteria.

Flask-REST-JSONAPI comes with this built in.

To sort artists in order of birth year, just navigate to http://localhost:5000/artists?sort=birth_year. In a web application, this would save you from needing to sort results on the client side, which could be costly in terms of performance and therefore impact the user experience.

Filtering is also easy. You append to the URL the criteria you wish to filter on, contained in square brackets. There are three pieces of information to include:

  • "name" - the field you are filtering by (e.g., birth_year)
  • "op" - the filter operation ("equal to", "greater than", "less than" etc.)
  • "val" - the value to filter against (e.g., 1900)

For example, the URL below retrieves artists whose birth year is greater than 1900:

http://localhost:5000/artists?filter=[{"name":"birth_year","op":"gt","val":1900}]

This functionality makes it much easier to retrieve only relevant information when calling the API. This is valuable for improving performance, especially when retrieving potentially large volumes of data over a slow connection.

Pagination

Another feature of the JSON API specification that aids performance is pagination. This is when large responses are sent over several "pages", rather than all in one go. You can control the page size and the number of the page you request in the URL.

So, for example, you could receive 100 results over 10 pages instead of loading all 100 in one go. The first page would contain results 1-10, the second page would contain results 11-20, and so on.

To specify the number of results you want to receive per page, you can add the parameter ?page[size]=X to the URL, where X is the number of results. Flask-REST-JSONAPI uses 30 as the default page size.

To request a given page number, you can add the parameter ?page[number]=X, where is the page number. You can combine both parameters as shown below:

http://localhost:5000/artists?page[size]=2&page[number]=2

This URL sets the page size to two results per page, and asks for the second page of results. This would return the third and fourth results from the overall response.

Relationships

Almost always, data in one table will be related to data stored in another. For instance, if you have a table of artists, chances are you might also want a table of artworks. Each artwork is related to the artist who created it.

The JSON API specification allows you to work with relational data easily, and the Flask-REST-JSONAPI lets you take advantage of this. Here, this will be demonstrated by adding an artworks table to the database, and including relationships between artist and artwork.

To implement the artworks example, it will be necessary to make a few changes to the code in application.py.

First, make a couple of extra imports, then create a new table which relates each artwork to an artist:

from marshmallow_jsonapi.flask import Relationship
from flask_rest_jsonapi import ResourceRelationship

# Define the Artwork table
class Artwork(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    title = db.Column(db.String)
    artist_id = db.Column(db.Integer, 
        db.ForeignKey('artist.id'))
    artist = db.relationship('Artist',
        backref=db.backref('artworks'))

Next, rewrite the abstraction layer:

# Create data abstraction 
class ArtistSchema(Schema):
    class Meta:
        type_ = 'artist'
        self_view = 'artist_one'
        self_view_kwargs = {'id': '<id>'}
        self_view_many = 'artist_many'

    id = fields.Integer()
    name = fields.Str(required=True)
    birth_year = fields.Integer(load_only=True)
    genre = fields.Str()
    artworks = Relationship(self_view = 'artist_artworks',
        self_view_kwargs = {'id': '<id>'},
        related_view = 'artwork_many',
        many = True,
        schema = 'ArtworkSchema',
        type_ = 'artwork')

class ArtworkSchema(Schema):
    class Meta:
        type_ = 'artwork'
        self_view = 'artwork_one'
        self_view_kwargs = {'id': '<id>'}
        self_view_many = 'artwork_many'

    id = fields.Integer()
    title = fields.Str(required=True)
    artist_id = fields.Integer(required=True)

This defines an abstraction layer for the artwork table, and adds a relationship between artist and artwork to the ArtistSchema class.

Next, define new resource managers for accessing artworks many at once and one at a time, and also for accessing the relationships between artist and artwork.

class ArtworkMany(ResourceList):
    schema = ArtworkSchema
    data_layer = {'session': db.session,
                  'model': Artwork}

class ArtworkOne(ResourceDetail):
    schema = ArtworkSchema
    data_layer = {'session': db.session,
                  'model': Artwork}

class ArtistArtwork(ResourceRelationship):
    schema = ArtistSchema
    data_layer = {'session': db.session,
                  'model': Artist}

Finally, add some new endpoints:

api.route(ArtworkOne, 'artwork_one', '/artworks/<int:id>')
api.route(ArtworkMany, 'artwork_many', '/artworks')
api.route(ArtistArtwork, 'artist_artworks',
    '/artists/<int:id>/relationships/artworks')

Run application.py and trying posting some data from the command line via curl:

curl -i -X POST -H 'Content-Type: application/json' -d '{"data":{"type":"artwork", "attributes":{"title":"The Persistance of Memory", "artist_id":1}}}' http://localhost:5000/artworks

This will create an artwork related to the artist with id=1.

In the browser, navigate to http://localhost:5000/artists/1/relationships/artworks. This should show the artworks related to the artist with id=1. This saves you from writing a more complex URL with parameters to filter artworks by their artist_id field. You can quickly list all the relationships between a given artist and their artworks.

Another feature is the ability to include related results in the response to calling the artists_one endpoint:

http://localhost:5000/artists/1?include=artworks

This will return the usual response for the artists endpoint, and also results for each of that artist's artworks.

Sparse Fields

One last feature worth mentioning - sparse fields. When working with large data resources with many complex relationships, the response sizes can blow up real fast. It is helpful to only retrieve the fields you are interested in.

The JSON API specification lets you do this by adding a fields parameter to the URL. For example URL below gets the response for a given artist and their related artworks. However, instead of returning all the fields for the given artwork, it returns only the title.

http://localhost:5000/artists/1?include=artworks&fields[artwork]=title

This is again very helpful for improving performance, especially over slow connections. As a general rule, you should only make requests to and from the server with the minimal amount of data required.

Final remarks

The JSON API specification is a very useful framework for sending data between server and client in a clean, flexible format. This article has provided an overview of what you can do with it, with a worked example in Python using the Flask-REST-JSONAPI library.

So what will you do next? There are many possibilities. The example in this article has been a simple proof-of-concept, with just two tables and a single relationship between them. You can develop an application as sophisticated as you like, and create a powerful API to interact with it using all the tools provided here.

Thanks for reading, and keep coding in Python!

Connecting to a GraphQL API Using Python

Connecting to a GraphQL API Using Python

Connecting to a GraphQL API Using Python. GraphQL in its simplest terms a query language used for the front end. We send a request and retrieve certain data back.

Let’s automate and remove the margin for error, trying to get rid of the front end one step at a time.

What exactly is GraphQL? GraphQL in its simplest terms a query language used for the front end. We send a request and retrieve certain data back. GraphQL is also advanced enough to make changes in the data called mutations. But that requires a whole new article to explain.

So the main reason I wanted to connect to GraphQL directly is because we have a web application where we must manually fill in fields one by one. Not the most effective use of my time if we know it’s repetitive.

The old method I used was thru Selenium, which also caused room for error whenever front end engineers made some changes. So I did some research and decided why not just send the data directly thru GraphQL. I still grab data from SalesForce, I have an article showing you how to do that here. I would then process all that data and send to the GraphQL endpoint.

But I am getting sidetracked, let’s connect to GraphQL using Python and get some data back!

Getting Started
Assuming you already have Python installed the main modules you need are:
1. requests (used to connect to GraphQL)
2. json (used to parse GraphQL data)
3. pandas (used for visibility of our data)

Let’s import these modules into a new Python script.

import requests
import json
import pandas as pd

For tutorial purposes, we will connect to a GraphQL endpoint that does not require authentication. We will connect to a Rick and Morty GraphQL API!

Let’s start with something simple, a dataset of characters will do. The information I want from each of them is the name, status, species, type and gender. We can set this GraphQL query as a string and set it as a variable like this:

query = """query {
    characters {
    results {
      name
      status
      species
      type
      gender
    }
  }
}"""

The people who made this made it really easy for us to connect. We get a limit of 10000 requests per day, so use them sparingly. The next few lines of code we set the URL, send the request to them aka the query. If successful we should get a status_code of 200 and text in the format of a string.

url = 'https://rickandmortyapi.com/graphql/'
r = requests.post(url, json={'query': query})
print(r.status_code)
print(r.text)

The r.text is in the format of a string. This is where the module json comes in. Let’s transform this string to a JSON format so we can move it to a DataFrame.

json_data = json.loads(r.text)

We only want name, status, species, type, and gender. So before we push it to a DataFrame, since it is a nested JSON we want to move further into the nest. Now we can send it to be translated to a DataFrame. We can do that with the following code:

df_data = json_data[‘data’][‘characters’][‘results’]
df = pd.DataFrame(df_data)

Our DataFrame should now look like this.

Now that is has been transferred to a pandas DataFrame the possibilities are endless!

If you enjoy the content, please feel free to support me on Patreon!