How to Install Python Packages for AWS Lambda Layer

How to Install Python Packages for AWS Lambda Layer

A guide on how to build Python Packages for AWS Lambda Layers using Docker containers. When I first heard about AWS Lambda I was quite confused about what it was and tried to use it to train a simple ML model but was hit with a hard 5 minutes execution limit. Fast-forward a few years, I believe Lambda has evolved a lot and so have people’s understanding of event-driven systems and serverless compute. It has become part of many modern applications and data architects.

Lambda and its Layers

When I first heard about AWS Lambda I was quite confused about what it was and tried to use it to train a simple ML model but was hit with a hard 5 minutes execution limit. Fast-forward a few years, I believe Lambda has evolved a lot and so have people’s understanding of event-driven systems and serverless compute. It has become part of many modern applications and data architects.

At re:Invent 2018, Lambda was heavily buffed with custom runtime and an increase in execution runtime limit to 15 minutes. Lambda Layers was also released which allowed you to share common dependencies to ease lambda deployment size and updates. However, AWS still hasn’t addressed the needs of friendly steps to bring in non-native python packages such as Pandas.

The troublesome approaches to bringing in external packages…

Currently, you either have to zip up your Lambda function and Linux compatible dependencies, or upload your dependencies as a Lambda Layers. If you’ve played around with Google Cloud Functions and Azure Function before then you would know that it can be as easy as to writing a wish list in the requirements.txt.

To add extra complexities, some of the Python packages need to compile C or C++ extensions (packages such as Numpy and Pandas). This could be a bit of a hassle if you want to use your macOS or Windows machine to pip install -t . pandasthen zip them up for Lambda Layers which is an Amazon Linux environment.

There are a few ways of bringing in Linux compatible dependencies whether it’s through Serverless or using an EC2 Instance. Now, if you’ve read some of my blogs before, I really enjoy using Lambdas in Hackathons and since time is the essence, I want to show you the easiest and quickest way on how I bring Python dependencies across as a Lambda Layers using Docker.

lambda aws python docker programming

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

Guide to Python Programming Language

Guide to Python Programming Language

Python Programming: A Beginner’s Guide

Python is an interpreted, high-level, powerful general-purpose programming language. You may ask, Python’s a snake right? and Why is this programming language named after it?

Python Hello World Program - Create & Run Your First Python Program in PyCharm

Python Hello World Program - Your first step towards Python world. Learn how to create the Hello World Python program in PyCharm.

Python Programming Tutorials For Beginners

Python Programming Tutorials For Beginners

How to Remove all Duplicate Files on your Drive via Python

Today you're going to learn how to use Python programming in a way that can ultimately save a lot of space on your drive by removing all the duplicates. We gonna use Python OS remove( ) method to remove the duplicates on our drive. Well, that's simple you just call remove ( ) with a parameter of the name of the file you wanna remove done.