Getting Started with AWS Lambda, Python and Flask

Originally published by Ran Ribenzaft at https://epsagon.com

What Is AWS Lambda?

AWS Lambda is a serverless compute service that triggers the execution of code in response to events. The code that executes on AWS Lambda is called a lambda function and allows the user to focus on code without having to worry about the backend, infrastructure, and scalability. It can also extend the capability of various AWS services, meaning, for example, it can work as an API backend serving incoming requests from API Gateways.

What Is Flask?

Flask is the popular microweb framework based on Werkzeug and Jinja 2, particularly written in Python. It is also known as a lightweight WSGI (Web Server Gateway Interface) web framework and is used to deploy a number of applications ranging from simple to complex. It is relatively simple, easy to use, and well documented. Available under a BSD license, it’s also actively developed and maintained. Flask is aimed at building web applications as well as web APIs and is extensible, hence we can plug in any ORM.

Application Overview

We will build a simple application by leveraging the power of AWS cloud services such as Lambda function, Layers, and EC2 instances. The goal of the application is to convert color images (RGB) to grayscale images using a web API and serverless compute. So we will use Flask to create the web API endpoint, and our logic will reside in a lambda function. Finally, we will monitor our API as well as the serverless Lambda function using Epsagon’s insightful dashboard. Below is the architecture diagram of the application that we are going to build with AWS Lambda Flask.

AWS Lambda Flask Architecture

Application Development

While developing this application, you will interact with AWS services such as S3 bucket and AWS Lambda.

S3 Bucket

Keeping the architecture diagram in mind, create an S3 bucket with two directories: colorImage and grayscaleImage. You can then set a trigger on colorImage, and the output will be stored in grayscaleImage. Let’s name the bucket epsagon-image-process.

Deploying an OpenCV Layer

Your first step will be to create layers since we will require OpenCV as an external package for image processing. So, go ahead and create Lambda layer for OpenCV. You use an EC2 instance to create a package for OpenCV and upload it to S3 bucket. For this, you’ll have to create an IAM role. Navigate to IAM management console, and create a new role for EC2 service with AmazonS3FullAccess permissions. Give an appropriate name (e.g. ec2_s3_access). We will use the name “ec2_s3_access” for the purpose of this article.

To create Lambda Layers, you’ll need a package in a zip file, which you will create in the next step. Before moving on to the next step, you can create the S3 bucket or use an existing bucket (e.g. epsagon-opencv-layer) to store the package.

After creating the S3 bucket, navigate to EC2 Management Console and spin up a t2.micro instancewith Ubuntu Server 18.04 LTS operating system. In step three of creating an instance, make sure to assign the IAM role (ec2s3access) that we created in the first step of Deploying OpenCV layer. The rest of the configuration for spinning up an instance remains unchanged.

Deploying an OpenCV Layer

After spinning up the instance, SSH to connect to the instance. You can then execute a series of commands to create a package and upload it to the S3 bucket (epsagon-opencv-layer).

sudo apt-get update
python3 -V # to check the version of Python since we are creating the layers for Python 3.6 runtime
sudo apt-get install python3-pip # installing Python package manager
mkdir -p build/python/lib/python3.6/sites-packages # creating the directory structure
pip3 install opencv-python -t build/python/lib/python3.6/sites-packages/ –system # installing OpenCV package/module in the custom directory, i.e. sites-packages
cd build
zip -r package.zip . # creating package in the form of zip file
aws s3 cp package.zip s3://epsagon-opencv-layer # uploading package to S3 bucket

Once the upload is finished, you will navigate to the Lambda Management Console -> Layers -> Create Layers. The configuration looks like this.

Configuring the AWS Lambda Layer

Click on Create Layer, and now you can terminate the instance.

Setting Up a Lambda Function

A lambda function is where our logic resides. But before creating the lambda function, you have to create an IAM role. So, navigate to the IAM management console, and create a new role for Lambda service with AWSLambdaExecute permissions. Make sure to give an appropriate name (e.g. lambda_role). We will use the name “lambda_role” for the purpose of this article.

To create the function, navigate to Lambda Management console -> Functions -> Create Function. You will select “Author from scratch” and fill in the basic information.

Setting Up an AWS Lambda Function

Note: Make sure to increase the Memory of your lambda function to 1024 MB and the timeout field to five minutes.

For the next step, you’ll need to bind opencv-layer with our lambda function. From the designer pane, click on Layers -> “Add layer to function.” Select the layer and its version and click on Add.

Setting Up the Trigger

Now you’re ready to set up a lambda trigger on the colorImage folder in the bucket (epsagon-image-process) that you created in the very first instance of your application building process.

Setting Up AWS Lambda Trigger

Image Processing Code

The lambda function is responsible for converting a color image into a grayscale image. Therefore, you will write the image processing code in the lambda function on the console itself. Paste the code below into the lambda_function file.

import json
import boto3
import cv2
import numpy as np
def lambda_handler(event, context):

s3 = boto3.client(“s3”)

if event:
extracting event records
file_obj = event[“Records”][0]
extracting bucket name
bucketname = str(file_obj[‘s3’][‘bucket’][‘name’])
extracting filename
filename = str(file_obj[‘s3’][‘object’][‘key’])
retrieving file object
fileObj = s3.get_object(Bucket=bucketname, Key=filename)
reading the file object
file_content = fileObj[“Body”].read()
converting bytes to 1-D array
np_array = np.fromstring(file_content, np.uint8)
reading an image from buffer
image_np = cv2.imdecode(np_array, cv2.IMREAD_COLOR)
converting color image to grayscale
gray = cv2.cvtColor(image_np, cv2.COLOR_BGR2GRAY)
writing image in tmp directory
cv2.imwrite(“/tmp/gray_obj.jpg”, gray)

filename = filename.split(“/”)[1]
uploading output to s3 bucket
s3.put_object(Bucket=bucketname, Key=“grayscaleImage/{}”.format(filename), Body=open(“/tmp/gray_obj.jpg”, “rb”).read())
return {

‘statusCode’: 200,

‘body’: json.dumps(‘Image processed successfully!’)

}

The above snippet will process the image and save the output back to the S3 bucket in the grayscaleImage folder.

Now let’s move on to API Development for the AWS Lambda Flask image processing application.

Monitoring With Epsagon

Epsagon enables developers to closely keep track of logs, traces, and invocations via an insightful dashboard. Here is how you can integrate Epsagon with AWS Lambda function in order to easily monitor your lambda function as well as the flask application. Epsagon brings an agentless monitoring tool to the table and also allows you to set up an alert mechanism for code exceptions, function timeouts, out of memory alerts, function errors, etc. Let’s start with AWS Lambda.

Monitoring AWS Lambda

For distributing tracing, you have to instrument your functions under a single application name. Epsagon monitors all the functions without instrumentation but at an individual level. It also supports auto-instrumentation of the functions for Python and Nodejs. Or you can instrument the functions manually using layers. Below is an example of auto-instrumenting the lambda function with ImgProc as the application name.

Auto-Instrumenting Lambda Functions Using Epsagon

Monitoring a Flask Application

Instrumenting the flask application works manually. Therefore, you will need to add a few lines in our flask application code. But before doing this, you have to grab the token key from here.

Setting the Epsagon Token

Once you have the token, you will go back to your instance, install Epsagon, and navigate to flaskapp.

pip3 install epsagon

It’s time to update app.py file. The final app.py looks like this.

import os

from flask import Flask, render_template, request

from werkzeug import secure_filename

import boto3

s3 = boto3.client(“s3”)

bucketname = “epsagon-image-process”

app = Flask(name)

UPLOAD_DIR = “images”

@app.route(‘/upload’, methods = [‘GET’, ‘POST’])

def upload_file():

if request.method == ‘POST’:

f = request.files[‘file’]

filename = str(os.getpid())+“_”+f.filename

f.save(os.path.join(UPLOAD_DIR, filename))

file_path = os.path.join(UPLOAD_DIR, filename)

s3.put_object(Bucket=bucketname, Key=“colorImage/{}”.format(filename), Body=open(file_path, “rb”).read())

os.remove(file_path)

return ‘file uploaded successfully’

Please make sure to replace your-token-here with the actual token. Now, it’s time to look at some visuals. Let’s start with the architecture that Epsagon has built for AWS Lambda Flask.

Tracing AWS Lambda Flask in Epsagon

You can also look at a few traces and even click on an individual trace to get in-depth information.

Deep-Dive into Traces in Epsagon

Epsagon monitoring is helpful in tracing errors and failures and identifying issues. In this way, it helps you find the root cause of a failure. Without Epsagon, it can be challenging even in the first instance to identify and categorize error-containing logs.

Pros and Cons of AWS Lambda Flask

So, overall what are the advantages and disadvantages of using AWS Lambda Flask.

Pros:

Flask is one of the best frameworks for building an API, just like the one we created here with a couple of lines of code.
The architecture is decoupled. In our case, every component is independently connected.
It is easy to integrate.

Cons:

AWS Lambda Flask is not cost-effective since we have to keep our EC2 instance running, which is billed hourly.
It is hard to debug the cause of failure without a monitoring tool.

What We Learned

With this exercise, you should have been able to grab a high-level of understanding of AWS Lambda Flask and been able to set up the image processing application. Along the way, you learned how to set up a lambda function, layers, and a server for the flask application and how to deploy a flask endpoint as an API with respect to the architecture diagram. Last but not least, you saw how you can monitor your application with Epsagon’s insightful dashboard in order to stay on top of any errors and issues.

Thanks for reading ❤

If you liked this post, share it with all of your programming buddies!

Follow us on Facebook | Twitter

What is GEEK

Buddha Community

Getting Started with AWS Lambda, Python and Flask

Ray Patel

1619510796

Lambda, Map, Filter functions in python

Welcome to my Blog, In this article, we will learn python lambda function, Map function, and filter function.

Lambda function in python: Lambda is a one line anonymous function and lambda takes any number of arguments but can only have one expression and python lambda syntax is

Syntax: x = lambda arguments : expression

Now i will show you some python lambda function examples:

#python #anonymous function python #filter function in python #lambda #lambda python 3 #map python #python filter #python filter lambda #python lambda #python lambda examples #python map

Ray Patel

1619518440

top 30 Python Tips and Tricks for Beginners

Welcome to my Blog , In this article, you are going to learn the top 10 python tips and tricks.

1) swap two numbers.

2) Reversing a string in Python.

3) Create a single string from all the elements in list.

4) Chaining Of Comparison Operators.

5) Print The File Path Of Imported Modules.

6) Return Multiple Values From Functions.

7) Find The Most Frequent Value In A List.

8) Check The Memory Usage Of An Object.

…

#python #python hacks tricks #python learning tips #python programming tricks #python tips #python tips and tricks #python tips and tricks advanced #python tips and tricks for beginners #python tips tricks and techniques #python tutorial #tips and tricks in python #tips to learn python #top 30 python tips and tricks for beginners

Dylan Iqbal

1565930228

Getting Started with AWS Lambda, Python and Flask

Originally published by Ran Ribenzaft at https://epsagon.com

What Is AWS Lambda?

AWS Lambda is a serverless compute service that triggers the execution of code in response to events. The code that executes on AWS Lambda is called a lambda function and allows the user to focus on code without having to worry about the backend, infrastructure, and scalability. It can also extend the capability of various AWS services, meaning, for example, it can work as an API backend serving incoming requests from API Gateways.

What Is Flask?

Flask is the popular microweb framework based on Werkzeug and Jinja 2, particularly written in Python. It is also known as a lightweight WSGI (Web Server Gateway Interface) web framework and is used to deploy a number of applications ranging from simple to complex. It is relatively simple, easy to use, and well documented. Available under a BSD license, it’s also actively developed and maintained. Flask is aimed at building web applications as well as web APIs and is extensible, hence we can plug in any ORM.

Application Overview

We will build a simple application by leveraging the power of AWS cloud services such as Lambda function, Layers, and EC2 instances. The goal of the application is to convert color images (RGB) to grayscale images using a web API and serverless compute. So we will use Flask to create the web API endpoint, and our logic will reside in a lambda function. Finally, we will monitor our API as well as the serverless Lambda function using Epsagon’s insightful dashboard. Below is the architecture diagram of the application that we are going to build with AWS Lambda Flask.

AWS Lambda Flask Architecture

Application Development

While developing this application, you will interact with AWS services such as S3 bucket and AWS Lambda.

S3 Bucket

Keeping the architecture diagram in mind, create an S3 bucket with two directories: colorImage and grayscaleImage. You can then set a trigger on colorImage, and the output will be stored in grayscaleImage. Let’s name the bucket epsagon-image-process.

Deploying an OpenCV Layer

Your first step will be to create layers since we will require OpenCV as an external package for image processing. So, go ahead and create Lambda layer for OpenCV. You use an EC2 instance to create a package for OpenCV and upload it to S3 bucket. For this, you’ll have to create an IAM role. Navigate to IAM management console, and create a new role for EC2 service with AmazonS3FullAccess permissions. Give an appropriate name (e.g. ec2_s3_access). We will use the name “ec2_s3_access” for the purpose of this article.

To create Lambda Layers, you’ll need a package in a zip file, which you will create in the next step. Before moving on to the next step, you can create the S3 bucket or use an existing bucket (e.g. epsagon-opencv-layer) to store the package.

After creating the S3 bucket, navigate to EC2 Management Console and spin up a t2.micro instancewith Ubuntu Server 18.04 LTS operating system. In step three of creating an instance, make sure to assign the IAM role (ec2s3access) that we created in the first step of Deploying OpenCV layer. The rest of the configuration for spinning up an instance remains unchanged.

Deploying an OpenCV Layer

After spinning up the instance, SSH to connect to the instance. You can then execute a series of commands to create a package and upload it to the S3 bucket (epsagon-opencv-layer).

sudo apt-get update
python3 -V # to check the version of Python since we are creating the layers for Python 3.6 runtime
sudo apt-get install python3-pip # installing Python package manager
mkdir -p build/python/lib/python3.6/sites-packages # creating the directory structure
pip3 install opencv-python -t build/python/lib/python3.6/sites-packages/ –system # installing OpenCV package/module in the custom directory, i.e. sites-packages
cd build
zip -r package.zip . # creating package in the form of zip file
aws s3 cp package.zip s3://epsagon-opencv-layer # uploading package to S3 bucket

Once the upload is finished, you will navigate to the Lambda Management Console -> Layers -> Create Layers. The configuration looks like this.

Configuring the AWS Lambda Layer

Click on Create Layer, and now you can terminate the instance.

Setting Up a Lambda Function

A lambda function is where our logic resides. But before creating the lambda function, you have to create an IAM role. So, navigate to the IAM management console, and create a new role for Lambda service with AWSLambdaExecute permissions. Make sure to give an appropriate name (e.g. lambda_role). We will use the name “lambda_role” for the purpose of this article.

To create the function, navigate to Lambda Management console -> Functions -> Create Function. You will select “Author from scratch” and fill in the basic information.

Setting Up an AWS Lambda Function

Note: Make sure to increase the Memory of your lambda function to 1024 MB and the timeout field to five minutes.

For the next step, you’ll need to bind opencv-layer with our lambda function. From the designer pane, click on Layers -> “Add layer to function.” Select the layer and its version and click on Add.

Setting Up the Trigger

Now you’re ready to set up a lambda trigger on the colorImage folder in the bucket (epsagon-image-process) that you created in the very first instance of your application building process.

Setting Up AWS Lambda Trigger

Image Processing Code

The lambda function is responsible for converting a color image into a grayscale image. Therefore, you will write the image processing code in the lambda function on the console itself. Paste the code below into the lambda_function file.

import json
import boto3
import cv2
import numpy as np
def lambda_handler(event, context):

s3 = boto3.client(“s3”)

if event:
extracting event records
file_obj = event[“Records”][0]
extracting bucket name
bucketname = str(file_obj[‘s3’][‘bucket’][‘name’])
extracting filename
filename = str(file_obj[‘s3’][‘object’][‘key’])
retrieving file object
fileObj = s3.get_object(Bucket=bucketname, Key=filename)
reading the file object
file_content = fileObj[“Body”].read()
converting bytes to 1-D array
np_array = np.fromstring(file_content, np.uint8)
reading an image from buffer
image_np = cv2.imdecode(np_array, cv2.IMREAD_COLOR)
converting color image to grayscale
gray = cv2.cvtColor(image_np, cv2.COLOR_BGR2GRAY)
writing image in tmp directory
cv2.imwrite(“/tmp/gray_obj.jpg”, gray)

filename = filename.split(“/”)[1]
uploading output to s3 bucket
s3.put_object(Bucket=bucketname, Key=“grayscaleImage/{}”.format(filename), Body=open(“/tmp/gray_obj.jpg”, “rb”).read())
return {

‘statusCode’: 200,

‘body’: json.dumps(‘Image processed successfully!’)

}

The above snippet will process the image and save the output back to the S3 bucket in the grayscaleImage folder.

Now let’s move on to API Development for the AWS Lambda Flask image processing application.

Monitoring With Epsagon

Epsagon enables developers to closely keep track of logs, traces, and invocations via an insightful dashboard. Here is how you can integrate Epsagon with AWS Lambda function in order to easily monitor your lambda function as well as the flask application. Epsagon brings an agentless monitoring tool to the table and also allows you to set up an alert mechanism for code exceptions, function timeouts, out of memory alerts, function errors, etc. Let’s start with AWS Lambda.

Monitoring AWS Lambda

For distributing tracing, you have to instrument your functions under a single application name. Epsagon monitors all the functions without instrumentation but at an individual level. It also supports auto-instrumentation of the functions for Python and Nodejs. Or you can instrument the functions manually using layers. Below is an example of auto-instrumenting the lambda function with ImgProc as the application name.

Auto-Instrumenting Lambda Functions Using Epsagon

Monitoring a Flask Application

Instrumenting the flask application works manually. Therefore, you will need to add a few lines in our flask application code. But before doing this, you have to grab the token key from here.

Setting the Epsagon Token

Once you have the token, you will go back to your instance, install Epsagon, and navigate to flaskapp.

pip3 install epsagon

It’s time to update app.py file. The final app.py looks like this.

import os

from flask import Flask, render_template, request

from werkzeug import secure_filename

import boto3

s3 = boto3.client(“s3”)

bucketname = “epsagon-image-process”

app = Flask(name)

UPLOAD_DIR = “images”

@app.route(‘/upload’, methods = [‘GET’, ‘POST’])

def upload_file():

if request.method == ‘POST’:

f = request.files[‘file’]

filename = str(os.getpid())+“_”+f.filename

f.save(os.path.join(UPLOAD_DIR, filename))

file_path = os.path.join(UPLOAD_DIR, filename)

s3.put_object(Bucket=bucketname, Key=“colorImage/{}”.format(filename), Body=open(file_path, “rb”).read())

os.remove(file_path)

return ‘file uploaded successfully’

Please make sure to replace your-token-here with the actual token. Now, it’s time to look at some visuals. Let’s start with the architecture that Epsagon has built for AWS Lambda Flask.

Tracing AWS Lambda Flask in Epsagon

You can also look at a few traces and even click on an individual trace to get in-depth information.

Deep-Dive into Traces in Epsagon

Epsagon monitoring is helpful in tracing errors and failures and identifying issues. In this way, it helps you find the root cause of a failure. Without Epsagon, it can be challenging even in the first instance to identify and categorize error-containing logs.

Pros and Cons of AWS Lambda Flask

So, overall what are the advantages and disadvantages of using AWS Lambda Flask.

Pros:

Flask is one of the best frameworks for building an API, just like the one we created here with a couple of lines of code.
The architecture is decoupled. In our case, every component is independently connected.
It is easy to integrate.

Cons:

AWS Lambda Flask is not cost-effective since we have to keep our EC2 instance running, which is billed hourly.
It is hard to debug the cause of failure without a monitoring tool.

What We Learned

With this exercise, you should have been able to grab a high-level of understanding of AWS Lambda Flask and been able to set up the image processing application. Along the way, you learned how to set up a lambda function, layers, and a server for the flask application and how to deploy a flask endpoint as an API with respect to the architecture diagram. Last but not least, you saw how you can monitor your application with Epsagon’s insightful dashboard in order to stay on top of any errors and issues.

Thanks for reading ❤

If you liked this post, share it with all of your programming buddies!

Follow us on Facebook | Twitter

Python Tricks Every Developer Should Know

Python is awesome, it’s one of the easiest languages with simple and intuitive syntax but wait, have you ever thought that there might ways to write your python code simpler?

In this tutorial, you’re going to learn a variety of Python tricks that you can use to write your Python code in a more readable and efficient way like a pro.

Let’s get started

Swapping value in Python

Instead of creating a temporary variable to hold the value of the one while swapping, you can do this instead

>>> FirstName = "kalebu"
>>> LastName = "Jordan"
>>> FirstName, LastName = LastName, FirstName 
>>> print(FirstName, LastName)
('Jordan', 'kalebu')

#python #python-programming #python3 #python-tutorials #learn-python #python-tips #python-skills #python-development

Art Lind

1602666000

How to Remove all Duplicate Files on your Drive via Python

Today you’re going to learn how to use Python programming in a way that can ultimately save a lot of space on your drive by removing all the duplicates.

Intro

In many situations you may find yourself having duplicates files on your disk and but when it comes to tracking and checking them manually it can tedious.

Heres a solution

Instead of tracking throughout your disk to see if there is a duplicate, you can automate the process using coding, by writing a program to recursively track through the disk and remove all the found duplicates and that’s what this article is about.

But How do we do it?

If we were to read the whole file and then compare it to the rest of the files recursively through the given directory it will take a very long time, then how do we do it?

The answer is hashing, with hashing can generate a given string of letters and numbers which act as the identity of a given file and if we find any other file with the same identity we gonna delete it.

There’s a variety of hashing algorithms out there such as

md5
sha1
sha224, sha256, sha384 and sha512

#python-programming #python-tutorials #learn-python #python-project #python3 #python #python-skills #python-tips

Getting Started with AWS Lambda, Python and Flask

What Is AWS Lambda?

What Is Flask?

Application Overview

Application Development

S3 Bucket

Setting Up a Lambda Function

Setting Up the Trigger

Image Processing Code

extracting event records

extracting bucket name

extracting filename

retrieving file object

reading the file object

converting bytes to 1-D array

reading an image from buffer

converting color image to grayscale

writing image in tmp directory

uploading output to s3 bucket

Monitoring With Epsagon

Monitoring AWS Lambda

Monitoring a Flask Application

Pros and Cons of AWS Lambda Flask

What We Learned

Further reading about Python and AWS

What is GEEK

Buddha Community

Lambda, Map, Filter functions in python

top 30 Python Tips and Tricks for Beginners

1) swap two numbers.

2) Reversing a string in Python.

3) Create a single string from all the elements in list.

4) Chaining Of Comparison Operators.

5) Print The File Path Of Imported Modules.

6) Return Multiple Values From Functions.

7) Find The Most Frequent Value In A List.

8) Check The Memory Usage Of An Object.

Getting Started with AWS Lambda, Python and Flask

What Is AWS Lambda?

What Is Flask?

Application Overview

Application Development

S3 Bucket

Setting Up a Lambda Function

Setting Up the Trigger

Image Processing Code

extracting event records

extracting bucket name

extracting filename

retrieving file object

reading the file object

converting bytes to 1-D array

reading an image from buffer

converting color image to grayscale

writing image in tmp directory

uploading output to s3 bucket

Monitoring With Epsagon

Monitoring AWS Lambda

Monitoring a Flask Application

Pros and Cons of AWS Lambda Flask

What We Learned

Further reading about Python and AWS

Python Tricks Every Developer Should Know

Let’s get started

How to Remove all Duplicate Files on your Drive via Python

Intro