A Deep Dive into Serverless Tracing with AWS X Ray & Lambda

Over the past few weeks I’ve been experimenting with building a Serverless API on AWS with the goal of having everything needed to run a production system. One necessary piece was distributed tracing. While I’d seen a bit of what some non-AWS options had to offer, the extra cost of the services themselves, along with actually getting the data to them, was a bit prohibitive for what I imagined should be possible (and cheaper) with only AWS, which brought me to X Ray. So, I wanted to see if I could use AWS X Ray to get everything I wanted out of a tracing solution. First, I’ll set some expectations of what I need in a production tracing solution as a goal to orient the research process.

Requirements

1. Distributed tracing. On a high level, I want to see the path of a request from service to service, along with location of errors and details on latency.
2. Internal tracing, or a “flame graph”. I want to be able to see the breakdown of a request inside of a lambda, along with a stack trace when an error occurs. This will also help me spot any parts of my lambda that are running particularly inefficiently.

Setup

First, a standard Serverless project setup and code walk-through. By the way, all the code I used in this post is available here.

npm i -g serverless
sls create --template aws-nodejs --path lambda-demo-api
cd lambda-demo-api && npm init && npm i --save-dev serverless-offline

The file structure of the lambda we’re going to deploy looks like this:

├── anotherFile.js
├── index.js
├── logger.js
├── node_modules
├── package-lock.json
├── package.json
├── serverless.yml
└── utils.js

I’ll be explaining each of these files later on. For now, let’s check out the initial serverless config.

#nodejs #aws-lambda #aws-x-ray #aws #serverless

What is GEEK

Buddha Community

A Deep Dive into Serverless Tracing with AWS X Ray & Lambda

Over the past few weeks I’ve been experimenting with building a Serverless API on AWS with the goal of having everything needed to run a production system. One necessary piece was distributed tracing. While I’d seen a bit of what some non-AWS options had to offer, the extra cost of the services themselves, along with actually getting the data to them, was a bit prohibitive for what I imagined should be possible (and cheaper) with only AWS, which brought me to X Ray. So, I wanted to see if I could use AWS X Ray to get everything I wanted out of a tracing solution. First, I’ll set some expectations of what I need in a production tracing solution as a goal to orient the research process.

Requirements

1. Distributed tracing. On a high level, I want to see the path of a request from service to service, along with location of errors and details on latency.
2. Internal tracing, or a “flame graph”. I want to be able to see the breakdown of a request inside of a lambda, along with a stack trace when an error occurs. This will also help me spot any parts of my lambda that are running particularly inefficiently.

Setup

First, a standard Serverless project setup and code walk-through. By the way, all the code I used in this post is available here.

npm i -g serverless
sls create --template aws-nodejs --path lambda-demo-api
cd lambda-demo-api && npm init && npm i --save-dev serverless-offline

The file structure of the lambda we’re going to deploy looks like this:

├── anotherFile.js
├── index.js
├── logger.js
├── node_modules
├── package-lock.json
├── package.json
├── serverless.yml
└── utils.js

I’ll be explaining each of these files later on. For now, let’s check out the initial serverless config.

#nodejs #aws-lambda #aws-x-ray #aws #serverless

Serverless Plugin for Microservice Code Management and Deployment

Serverless M

Serverless M (or Serverless Modular) is a plugin for the serverless framework. This plugins helps you in managing multiple serverless projects with a single serverless.yml file. This plugin gives you a super charged CLI options that you can use to create new features, build them in a single file and deploy them all in parallel

Currently this plugin is tested for the below stack only

• AWS
• NodeJS λ
• Rest API (You can use other events as well)

Prerequisites

Make sure you have the serverless CLI installed

# Install serverless globally
$ npm install serverless -g

Getting Started

To start the serverless modular project locally you can either start with es5 or es6 templates or add it as a plugin

ES6 Template install

# Step 1. Download the template
$ sls create --template-url https://github.com/aa2kb/serverless-modular/tree/master/template/modular-es6 --path myModularService

# Step 2. Change directory
$ cd myModularService

# Step 3. Create a package.json file
$ npm init

# Step 3. Install dependencies
$ npm i serverless-modular serverless-webpack webpack --save-dev

ES5 Template install

# Step 1. Download the template
$ sls create --template-url https://github.com/aa2kb/serverless-modular/tree/master/template/modular-es5 --path myModularService

# Step 2. Change directory
$ cd myModularService

# Step 3. Create a package.json file
$ npm init

# Step 3. Install dependencies
$ npm i serverless-modular --save-dev

If you dont want to use the templates above you can just add in your existing project

Adding it as plugin

plugins:
  - serverless-modular

Now you are all done to start building your serverless modular functions

API Reference

The serverless CLI can be accessed by

# Serverless Modular CLI
$ serverless modular

# shorthand
$ sls m

Serverless Modular CLI is based on 4 main commands

• sls m init
• sls m feature
• sls m function
• sls m build
• sls m deploy

init command

sls m init

The serverless init command helps in creating a basic .gitignore that is useful for serverless modular.

The basic .gitignore for serverless modular looks like this

#node_modules
node_modules

#sm main functions
sm.functions.yml

#serverless file generated by build
src/**/serverless.yml

#main serverless directories generated for sls deploy
.serverless

#feature serverless directories generated sls deploy
src/**/.serverless

#serverless logs file generated for main sls deploy
.sm.log

#serverless logs file generated for feature sls deploy
src/**/.sm.log

#Webpack config copied in each feature
src/**/webpack.config.js

---

feature command

The feature command helps in building new features for your project

options (feature Command)

This command comes with three options

--name: Specify the name you want for your feature

--remove: set value to true if you want to remove the feature

--basePath: Specify the basepath you want for your feature, this base path should be unique for all features. helps in running offline with offline plugin and for API Gateway

options	shortcut	required	values	default value
--name	-n	✅	string	N/A
--remove	-r	❎	true, false	false
--basePath	-p	❎	string	same as name

Examples (feature Command)

Creating a basic feature

# Creating a jedi feature
$ sls m feature -n jedi

Creating a feature with different base path

# A feature with different base path
$ sls m feature -n jedi -p tatooine

Deleting a feature

# Anakin is going to delete the jedi feature
$ sls m feature -n jedi -r true

---

function command

The function command helps in adding new function to a feature

options (function Command)

This command comes with four options

--name: Specify the name you want for your function

--feature: Specify the name of the existing feature

--path: Specify the path for HTTP endpoint helps in running offline with offline plugin and for API Gateway

--method: Specify the path for HTTP method helps in running offline with offline plugin and for API Gateway

options	shortcut	required	values	default value
--name	-n	✅	string	N/A
--feature	-f	✅	string	N/A
--path	-p	❎	string	same as name
--method	-m	❎	string	'GET'

Examples (function Command)

Creating a basic function

# Creating a cloak function for jedi feature
$ sls m function -n cloak -f jedi

Creating a basic function with different path and method

# Creating a cloak function for jedi feature with custom path and HTTP method
$ sls m function -n cloak -f jedi -p powers -m POST

---

build command

The build command helps in building the project for local or global scope

options (build Command)

This command comes with four options

--scope: Specify the scope of the build, use this with "--feature" tag

--feature: Specify the name of the existing feature you want to build

options	shortcut	required	values	default value
--scope	-s	❎	string	local
--feature	-f	❎	string	N/A

Saving build Config in serverless.yml

You can also save config in serverless.yml file

custom:
  smConfig:
    build:
      scope: local

Examples (build Command)

all feature build (local scope)

# Building all local features
$ sls m build

Single feature build (local scope)

# Building a single feature
$ sls m build -f jedi -s local

All features build global scope

# Building all features with global scope
$ sls m build -s global

deploy command

The deploy command helps in deploying serverless projects to AWS (it uses sls deploy command)

options (deploy Command)

This command comes with four options

--sm-parallel: Specify if you want to deploy parallel (will only run in parallel when doing multiple deployments)

--sm-scope: Specify if you want to deploy local features or global

--sm-features: Specify the local features you want to deploy (comma separated if multiple)

options	shortcut	required	values	default value
--sm-parallel	❎	❎	true, false	true
--sm-scope	❎	❎	local, global	local
--sm-features	❎	❎	string	N/A
--sm-ignore-build	❎	❎	string	false

Saving deploy Config in serverless.yml

You can also save config in serverless.yml file

custom:
  smConfig:
    deploy:
      scope: local
      parallel: true
      ignoreBuild: true

Examples (deploy Command)

Deploy all features locally

# deploy all local features
$ sls m deploy

Deploy all features globally

# deploy all global features
$ sls m deploy --sm-scope global

Deploy single feature

# deploy all global features
$ sls m deploy --sm-features jedi

Deploy Multiple features

# deploy all global features
$ sls m deploy --sm-features jedi,sith,dark_side

Deploy Multiple features in sequence

# deploy all global features
$ sls m deploy  --sm-features jedi,sith,dark_side --sm-parallel false

---

Author: aa2kb
Source Code: https://github.com/aa2kb/serverless-modular 
License: MIT license

#serverless #aws #node #lambda 

How To Unite AWS KMS with Serverless Application Model (SAM)

The Basics

AWS KMS is a Key Management Service that let you create Cryptographic keys that you can use to encrypt and decrypt data and also other keys. You can read more about it here.

Important points about Keys

Please note that the customer master keys(CMK) generated can only be used to encrypt small amount of data like passwords, RSA key. You can use AWS KMS CMKs to generate, encrypt, and decrypt data keys. However, AWS KMS does not store, manage, or track your data keys, or perform cryptographic operations with data keys.

You must use and manage data keys outside of AWS KMS. KMS API uses AWS KMS CMK in the encryption operations and they cannot accept more than 4 KB (4096 bytes) of data. To encrypt application data, use the server-side encryption features of an AWS service, or a client-side encryption library, such as the AWS Encryption SDK or the Amazon S3 encryption client.

Scenario

We want to create signup and login forms for a website.

Passwords should be encrypted and stored in DynamoDB database.

What do we need?

1. KMS key to encrypt and decrypt data
2. DynamoDB table to store password.
3. Lambda functions & APIs to process Login and Sign up forms.
4. Sign up/ Login forms in HTML.

Lets Implement it as Serverless Application Model (SAM)!

Lets first create the Key that we will use to encrypt and decrypt password.

KmsKey:
    Type: AWS::KMS::Key
    Properties: 
      Description: CMK for encrypting and decrypting
      KeyPolicy:
        Version: '2012-10-17'
        Id: key-default-1
        Statement:
        - Sid: Enable IAM User Permissions
          Effect: Allow
          Principal:
            AWS: !Sub arn:aws:iam::${AWS::AccountId}:root
          Action: kms:*
          Resource: '*'
        - Sid: Allow administration of the key
          Effect: Allow
          Principal:
            AWS: !Sub arn:aws:iam::${AWS::AccountId}:user/${KeyAdmin}
          Action:
          - kms:Create*
          - kms:Describe*
          - kms:Enable*
          - kms:List*
          - kms:Put*
          - kms:Update*
          - kms:Revoke*
          - kms:Disable*
          - kms:Get*
          - kms:Delete*
          - kms:ScheduleKeyDeletion
          - kms:CancelKeyDeletion
          Resource: '*'
        - Sid: Allow use of the key
          Effect: Allow
          Principal:
            AWS: !Sub arn:aws:iam::${AWS::AccountId}:user/${KeyUser}
          Action:
          - kms:DescribeKey
          - kms:Encrypt
          - kms:Decrypt
          - kms:ReEncrypt*
          - kms:GenerateDataKey
          - kms:GenerateDataKeyWithoutPlaintext
          Resource: '*'

The important thing in above snippet is the KeyPolicy. KMS requires a Key Administrator and Key User. As a best practice your Key Administrator and Key User should be 2 separate user in your Organisation. We are allowing all permissions to the root users.

So if your key Administrator leaves the organisation, the root user will be able to delete this key. As you can see **KeyAdmin **can manage the key but not use it and KeyUser can only use the key. ${KeyAdmin} and **${KeyUser} **are parameters in the SAM template.

You would be asked to provide values for these parameters during SAM Deploy.

#aws #serverless #aws-sam #aws-key-management-service #aws-certification #aws-api-gateway #tutorial-for-beginners #aws-blogs

Adding Code to AWS Lambda, Lambda Layers, and Lambda Extensions Using Docker

2020 was a difficult year for all of us, and it was no different for engineering teams. Many software releases were postponed, and the industry slowed its development speed quite a bit.

But at least at AWS, some teams released updates out of the door at the end of the year. AWS Lambda received two significant improvements:

• AWS Lambda Extensions; and
• Support of Docker images for your functions.

With these two new features and Lambda Layers, we now have three ways to add code to Lambda that isn’t directly part of our Lambda function.

The question is now: when should we use what?

In this article, I try to shine some light on the Lambda Layers, Lambda Extensions, and Docker image for Lambda.

First things first. All these Lambda features can be used together. So if you think about where to put your code, at least your decisions aren’t mutually exclusive. You can upload a Docker image and attach a regular Lambda Layer and a Lambda Extension. The same is possible if your Lambda function is based on a ZIP archive.

What does this all mean? Keep reading and find out.

#aws #aws-lambda #serverless #devops #docker #lambda

Creating your first AWS Lambda Function in Node.js | Serverless Saturday

In this Serverless Saturday video, we’ll be going over how to create your first AWS Lambda function!
In the next video, we’ll be covering how to set up CI/CD with your AWS Lambda function so stay tuned and make sure to subscribe!

To get started, log-in to your AWS account here: https://aws.amazon.com/console/

Found this video helpful? Feel free to support this channel here: https://ko-fi.com/jacksonyuan

#node.js #node #lambda #aws #aws lambda #serverless