A deep dive into boto3 and how AWS built it

AWS defines boto3 as a Python Software Development Kit to create, configure, and manage AWS services. In this article, we’ll look at how boto3 works and how it can help us interact with various AWS services.

Table of contents:

·  1. Boto3 under the hood

·  2. Clients vs. Resources

∘  Why resource is often much easier to use than client

∘  Why you will still use clients for most of your work

·  3. Waiters

∘  Wait until a specific S3 object arrived in S3

·  4. Collections

·  5. Sessions: How to pass IAM credentials to your boto3 code?

∘  How to change a default boto3 session?

·  Conclusion

1. Boto3 under the hood

Both  AWS CLI and boto3 are built on top of  botocore — a low-level Python library that takes care of everything needed to send an API request to AWS and receive a response back. Botocore:

• handles session, credentials, and configuration,
• gives fine-granular access to all operations (ex. ListObjects, DeleteObject) within a specific service (ex. S3),
• takes care of serializing input parameters, signing requests, and deserializing response data into Python dictionaries,
• provides low-level clients and high-level resource abstractions to interact with AWS services from Python.

You can think of botocore as a package that allows us to forget about underlying JSON specifications and use Python (boto3) when interacting with AWS APIs.

2. Clients vs. Resources

In most cases, we should use boto3 rather than botocore. Using boto3, we can choose to either interact with lower-level clients or higher-level object-oriented resource abstractions. The image below shows the relationship between those abstractions.

Level of abstraction in boto3, aws-cli, and botocore based on S3 as an example — image by author

To understand the difference between those components, let’s look at a simple example that will demonstrate the difference between an S3 client and an S3 resource. We want to list all objects from the images directory, i.e., all objects with the prefix images/.

#aws #coffee2021 #python #serverless