This post originally appeared on Build Smarter.
At ITHAKA our web teams write applications that each interact with a large handful of services—sometimes as many as ten. Each of those services provide multiple endpoints, each with their own set of path variables and query parameters.
Gathering data from multiple services has become a ubiquitous task for web application developers. The complexity can grow quickly: calling an API endpoint with multiple parameter sets, calling multiple API endpoints, calling multiple endpoints in multiple APIs. While the business logic can get hairy, the code to interact with those APIs doesn’t have to.
We created a module some time ago for low-level HTTP interactions, and use it throughout our code base. For a good while, though, the actual details of each service call—the service name, endpoint path, query parameters—were scattered throughout the code. This inevitably led to duplication as well as a bug or two when we made an update in one place and forgot about the other.
To reduce the pains from this, we eventually took stock of these scattered configurations and centralized them in one registry module. This module essentially contains a giant dictionary of all the services we interact with:
service_endpoints = { 'CONTENT_SERVICE': { 'SERVICE': 'content-service', 'METADATA_ENDPOINT': '/content/{id}', 'CITATION_ENDPOINT': '/citation/{citation_type}/{id}', }, 'SEARCH_SERVICE': { 'SERVICE': 'search', 'SEARCH_ENDPOINT': '/search', 'EXCERPTS_ENDPOINT': '/excerpt?contentId={content_id}', }, ... }
Each service has a 'SERVICE' key containing the name of the service used to discover hosts, and some number of '*_ENDPOINT' keys that describe an endpoint and its parameters. Calling these services looks like this:
from http import make_get_request_with_timeout from services.registry import service_endpointsCONTENT_SERVICE = service_endpoints.get(‘CONTENT_SERVICE’, {})
METADATA_ENDPOINT = CONTENT_SERVICE.get(‘METADATA_ENDPOINT’, ‘’)determine content_id…
metadata = make_get_request_with_timeout(
service_name=CONTENT_SERVICE.get(‘SERVICE’),
endpoint=METADATA_ENDPOINT.format(id=content_id),
headers={‘Accept’: ‘application/json’},
request_timeout=5,
)
As you can see, there are a variety of shapes to these endpoints. This solved the issue of duplication across the codebase, but we still faced a couple of problems with this approach:
In order to address the high variability of behaviors and lack of structured data of this problem, we built a new paradigm for HTTP interactions that provided a declarative interface for configuring services. We wanted a few things out of it:
With these desires in mind, we came up with apiron. With apiron the same definition from above looks more like this:
from services import IthakaDiscoverableService
from apiron.endpoint import Endpointclass ContentService(IthakaDiscoverableService):
service_name = ‘content-service’metadata = Endpoint(path='/content/{id}') citation = Endpoint(path='/citation/{citation_type}/{id}')
And the code to call the service looks more like this:
from apiron.client import ServiceCaller, Timeout
from services import ContentServiceCONTENT_SERVICE = ContentService()
determine content_id…
metadata = ServiceCaller.call(
service=CONTENT_SERVICE,
endpoint=CONTENT_SERVICE.metadata,
path_kwargs={‘content_id’: content_id},
headers={‘Accept’: ‘application/json’},
timeout_spec=Timeout(read_timeout=5),
)
We can now define what ContentService looks like and easily refer back to that class whenever we need to understand its shape. Service discovery is now a plugin system. Endpoints can be introspected and have their parameters validated and enforced.
With apiron we’ve been able to replace many of our existing service calls quickly and with little pain. The code has become clearer and with the cognitive load out of the way we can begin focusing on other gains like streaming responses and data compression. It’s been nice for us, and we’d like to make it nice for you too.
You can install apiron from PyPI with pip (or your favorite package manager):
$ pip install apiron
#python #api