1656080760
Serverless Plugin: AWS API Gateway integration Helper
Serverless Plugin: AWS Api Gateway integration helper
Overview
The plugin provides the functionality to merge OpenApiSpecification files (formerly known as swagger) with one or multiple YML files containing the the x-amazon-apigateway extensions. There are several use-cases to keep both information separated, e.g. it is needed to deploy different api gateway integrations depending on a stage environment.
When dealing with functional tests you do not want to test the production environment, but only a mocking response.
The plugin supports YML based OpenApi3 specification files only
Features
- deploy stage dependent x-amazon-apigateway integrations
- separate infrastructure (aws) from openapi specification
- use mock integrations for functional testing
- auto-generating CORS methods, headers and api gateway mocking response
- hook into package & deploy lifeCycle and generate combined openApi files on the fly during deployment
- auto-inject generated openApi file into the Body property of specified API Gateway
- generate mocking responses without specifying x-amazon-apigateway-integration objects
- generate request-validation blocks
- generate all required x-amazon-apigateway-integration objects automatically
- full proxy generation support with [NEW]: feature: PROXY Manager
See the examples folder for a full working example
Installation & Setup
Run npm install in your Serverless project.
$ npm install --save-dev serverless-openapi-integration-helper
Add the plugin to your serverless.yml file
plugins:
- serverless-openapi-integration-helper
Plugin configuration
You can configure the plugin under the key openApiIntegration. See See Configuration Reference for a list of available options
The mapping array must be used to configure where the files containing the x-amazon-apigateway-integration blocks are located.
openApiIntegration:
package: true #New feature! Hook into the package & deploy process
inputFile: schema.yml
mapping:
- stage: [dev, prod] #multiple stages
path: integrations
- stage: test #single stage
path: mocks
In the above example all YML files inside the integrations directory will be processed and merged with the schema.yml file when deploying the dev stage
serverless deploy --stage=dev
To use a different x-amazon-apigateway to perform functional tests (with mocking responses e.g) the directory mock is processed and merged with the schema.yml file when deploying the test stage
serverless deploy --stage=test
Usage
You can setup a fully working API GATEWAY with any openApi 3.0 specification file First create the input file containing the OpenApiSpecification
# ./schema.yml
openapi: 3.0.0
info:
description: User Registration
version: 1.0.0
title: UserRegistration
paths:
/api/v1/user:
post:
summary: adds a user
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/Customer'
responses:
'201':
description: user created
components:
schemas:
Customer:
type: object
required:
- email_address
- password
properties:
email_address:
type: string
example: test@example.com
password:
type: string
format: password
example: someStrongPassword#
The plugin will generate the x-amazon-apigateway integrations objects for all methods that do not have an integration.
#generate a file containing a gateway mock integration in the directory /mocks
serverless integration create --output mocks --type mock --stage=test
#generate a file containing the production integration in the directory integrations/
serverless integration create --output integrations --type http --stage=prod
Supported types are
- http_proxy
- http
- aws
- aws_proxy
- mock
The plugin now generates a merged file during deployment that is automatically injected in your serverless resources
#Create OpenApi File containing mocking responses (usable in functional tests) and deploy to ApiGateway
serverless deploy --stage==test
#Create OpenApi File containing the production integration and deploy to ApiGateway
serverless deploy --stage=prod
The generated output is automatically injected in the resources.Resources.YOUR_API_GATEWAY.Properties.Body property
resources:
Resources:
ApiGatewayRestApi:
Type: AWS::ApiGateway::RestApi
Properties:
ApiKeySourceType: HEADER
Body: ~ #autogenerated by plugin
Description: "Some Description"
FailOnWarnings: false
Name: ${opt:stage, self:provider.stage}-some-name
EndpointConfiguration:
Types:
- REGIONAL
ApiGatewayDeployment:
Type: AWS::ApiGateway::Deployment
Properties:
RestApiId:
Ref: ApiGatewayRestApi
StageName: ${opt:stage, self:provider.stage}
Commands
Manual merge
The generate command can be used independently with
serverless integration merge --stage=dev
Of course then the API Gateway Body property has to be specified manually
resources:
Resources:
ApiGatewayRestApi:
Type: AWS::ApiGateway::RestApi
Properties:
ApiKeySourceType: HEADER
Body: ${file(openapi-integration/api.yml)}
CORS generator
The plugin can generate full CORS support out of the box.
openApiIntegration:
cors: true
...
If enabled, the plugin generates all required OPTIONS methods as well as the required header informations and adds a mocking response to API Gateway. You can customize the CORS templates by placing your own files inside a directory openapi-integration (in your project root). The following files can be overwritten:
| Filename | Description |
|---|---|
| headers.yml | All headers required for CORS support |
| integration.yml | Contains the x-amazon-apigateway-integration block |
| path.yml | OpenApi specification for the OPTIONS method |
| response-parameters.yml | The response Parameters of the x-amazon-apigateway-integration responses |
See the EXAMPLES directory for detailed instructions.
Auto Mock Generator
If enabled, the plugin generates mocking responses for all methods that do not have an x-amazon-apigateway-integration block defined. It takes the first 2xx response defined in the openApi specification and generates a simple mocking response on the fly
openApiIntegration:
autoMock: true
...
When using the autoMock feature, you do not need to specify inputPath mappings, since all endpoints are mocked automatically
openApiIntegration:
package: true
inputFile: schema.yml
mapping: ~
VALIDATION generator
The plugin supports full request validation out of the box
openApiIntegration:
validation: true
...
If enabled, the plugin generates the x-amazon-apigateway-request-validators blocks and adds a basic request validation to all methods. You can customize the VALIDATION template by placing your own files inside a directory openapi-integration (in your project root). The following files can be overwritten:
| Filename | Description |
|---|---|
| request-validator.yml | The x-amazon-apigateway-request-validators block |
See the EXAMPLES directory for detailed instructions.
Proxy Manager
The proxymanager feature automates the complete generation of an HTTP proxy integration. You only have to define the target URL and all necessary AWS integration blocks are generated on-the-fly during deployment.
openApiIntegration:
cors: true
validation: true
mapping:
- stage: [dev, prod]
proxyManager:
type: http_proxy
baseUrl: https://www.example.com
pattern: "(?<=api\/v1)\/.+"
...
With this setting, no separate integration files need to be created
A combination of your own and auto-generated files is still possible without any problems
Proxy Manager configuration
type
at the moment only http_proxy supported
baseUrl
The base url is required to map the path variable from the openapi specification to the URI from the aws integration.
Example:
#original openapi specification
paths:
/api/v1/user:
post:
...
will be translated to
#generated openapi specification output
paths:
/api/v1/user:
post:
...
x-amazon-apigateway-integration:
type: http_proxy
passthroughBehavior: when_no_match
httpMethod: POST
uri: https://www.example.com/api/v1/user
pattern
The pattern can be used to adapt the mapping of the base url using regexp, to remove a prefix, or a version string
Example:
baseUrl: https://www.example.com
pattern: "(?<=api\/v1)\/.+"
will translate the route /api/v1/user to https://www.example.com/user
Configuration Reference
configure the plugin under the key openApiIntegration
openApiIntegration:
inputFile: schema.yml #required
package: true #optional, defaults to false
inputDirectory: ./ #optional, defaults to ./
cors: true #optional, defaults to false
autoMock: true #optional, defaults to false
validation: true #optional, defaults to false
mapping: #optional, can be completely blank if autoMock option is enabled
- stage: [dev, prod] #multiple stages
path: integrations
proxyManager: #optional
type: http_proxy
baseUrl: https://example.com
pattern: "(?<=v1)\/.+"
- stage: test #single stage
path: mocks/customer.yml
outputFile: api.yml #optional, defaults to api.yml
outputDirectory: openapi-integration #optional, defaults to ./openapi-integration
Known Issues
Stage deployment
When using serverless framework only to deploy your aws resources without having any lambda functions or triggers, the AWS Gateway deploymemt does not behave as expected. Any deployment to an existing stage will be ignored, since CloudFormation does not redeploy a stage if the DeploymentIdentifier has not changed.
The plugin serverless-random-gateway-deployment-id solves this problem by adding a random id to the deployment-name and all references to it on every deploy
See the examples folder for a full working example
Variable Resolving
Serverless variables inside the openapi integration files are not resolved correctly when using the package & deploy hooks. This problem can be solved by using the api gateway STAGE VARIABLES.
See the examples folder for a full working example
Example
service:
name: user-registration
provider:
name: aws
stage: dev
region: eu-central-1
plugins:
- serverless-openapi-integration-helper
openApiIntegration:
inputFile: schema.yml
package: true
mapping:
- path: integrations
stage: [dev, prod]
- path: mocks/customer.yml
stage: test
functions:
resources:
Resources:
ApiGatewayRestApi:
Type: AWS::ApiGateway::RestApi
Properties:
ApiKeySourceType: HEADER
Body: ~
Description: "Some Description"
FailOnWarnings: false
Name: ${opt:stage, self:provider.stage}-some-name
EndpointConfiguration:
Types:
- REGIONAL
ApiGatewayDeployment:
Type: AWS::ApiGateway::Deployment
Properties:
RestApiId:
Ref: ApiGatewayRestApi
StageName: ${opt:stage, self:provider.stage}
serverless deploy --stage=test
serverless deploy --stage=prod
Approach to a functional test of schema validation
The plugin works well in combination with the serverless-plugin-test-helper to automate tests against the deployed api gateway
Install The plugin test helper package
npm install --save-dev serverless-plugin-test-helper
add the plugin as a plugin dependency in your serverless configuration file and configure the plugin according to the Readme
#./serveless.yml
plugins:
- serverless-plugin-test-helper
- serverless-openapi-integration-helper
[...]
resources:
Outputs:
GatewayUrl: # This is the key that will be used in the generated outputs file
Description: This is a helper for functional tests
Value: !Join
- ''
- - 'https://'
- !Ref ApiGatewayRestApi
- '.execute-api.'
- ${opt:region, self:provider.region}
- '.amazonaws.com/'
- ${opt:stage, self:provider.stage}
Resources:
ApiGatewayRestApi:
Type: AWS::ApiGateway::RestApi
Properties:
ApiKeySourceType: HEADER
Body: ~
Description: User Registration (${opt:stage, self:provider.stage})
FailOnWarnings: false
Name: ${opt:stage, self:provider.stage}-gateway
EndpointConfiguration:
Types:
- REGIONAL
ApiGatewayDeployment:
Type: AWS::ApiGateway::Deployment
Properties:
RestApiId:
Ref: ApiGatewayRestApi
StageName: ${opt:stage, self:provider.stage}
Testing the schema validation
Add a functional test (e.g. with jest)
//tests/registration.js
import {getOutput} from 'serverless-plugin-test-helper';
import axios from 'axios';
axios.defaults.adapter = require('axios/lib/adapters/http'); //Todo
const URL = getOutput('GatewayUrl');
test('request validation on registration', async () => {
expect.assertions(1);
const {status} = await axios.post(URL + '/api/v1/user',
{
"email_address": "test@example.com",
"password": "someStrongPassword#"
},
{
headers: {
'Content-Type': 'application/json',
}
});
expect(status).toEqual(201);
});
test('request validation on registration (invalid request)', async () => {
expect.assertions(1);
try {
await axios.post(URL + '/api/v1/user',
{
"email": "test@example.com"
},
{
headers: {
'Content-Type': 'application/json',
}
});
} catch (e) {
expect(e.response).toMatchObject({
statusText: 'Bad Request',
status: 400
});
}
});
Then perform the functional test
serverless deploy --stage=test
npm test
serverless remove --stage=test
The command will
- merge the openapi specification file with the MOCK integration configured before
- deploy to API Gateway in an isolated TEST infrastructure environment (your other environments will not be affected since we are deploying to a separated gateway)
- perform the test and verify that schema validation is correct
- remove all TEST resources if test succeeded
See the examples folder for a full working example
Feedback is appreciated! If you have an idea for how this plugin/library can be improved (or even just a complaint/criticism) then please open an issue.
Author: yndlingsfar
Source Code: https://github.com/yndlingsfar/serverless-openapi-integration-helper
License: MIT license
What is GEEK
Buddha Community
1599364620
API Integration Practices and Patterns
We all hear it so often that we almost stop hearing it: “Integration is critical to meeting users’ needs.”
Integration work consumes 50%-80% of the time and budget of digital transformation projects, or building a digital platform, while innovation gets only the leftovers, according to SAP and Salesforce. And as everyone from legacy enterprises to SaaS startups launches new digital products, they all hit a point at which the product cannot unlock more value for users or continue to grow without making integration a feature.
If I were to sum up the one question behind all of the other questions that I hear from customers, enterprises, partners, and developers, it would be something like: “Is integration a differentiator that we should own? Or an undifferentiated but necessary feature that supports what we’re trying to accomplish?”
This Refcard won’t try to answer that question for you. Rather, no matter what type of development work you do, API integration is a fact of life today, like gravity. Why? Today, experience is paramount. The average enterprise uses more than 1,500 cloud applications (with the number growing by 20% each year). Every app needs to integrate with other systems in a fluid and ever-changing application ecosystem. So instead, I’ll share some of the common practices you’re likely to contend with as well as some patterns to consider.
This is a preview of the API Integrations Practices and Patterns Refcard. To read the entire Refcard, please download the PDF from the link above.
#apis #api integration #integration patterns #api cloud #api patterns #api authentication #api errors #apis and integrations
1601381326
Public ASX100 APIs: The Essential List
We’ve conducted some initial research into the public APIs of the ASX100 because we regularly have conversations about what others are doing with their APIs and what best practices look like. Being able to point to good local examples and explain what is happening in Australia is a key part of this conversation.
Method
The method used for this initial research was to obtain a list of the ASX100 (as of 18 September 2020). Then work through each company looking at the following:
- Whether the company had a public API: this was found by googling “[company name] API” and “[company name] API developer” and “[company name] developer portal”. Sometimes the company’s website was navigated or searched.
- Some data points about the API were noted, such as the URL of the portal/documentation and the method they used to publish the API (portal, documentation, web page).
- Observations were recorded that piqued the interest of the researchers (you will find these below).
- Other notes were made to support future research.
- You will find a summary of the data in the infographic below.
Data
With regards to how the APIs are shared:
- Documentation is where just the Swagger/OpenAPI docs are hosted somewhere. e.g. https://api.linkgroup.com/member/docs/index
- Portal is where an interactive portal has been developed. e.g. https://dev.telstra.com/
- The web page is where the company has a single page that isn’t interactive explaining their Public APIs. e.g. https://www.bendigoadelaide.com.au/banking-products-api.asp
#api #api-development #api-analytics #apis #api-integration #api-testing #api-security #api-gateway
1598408880
How To Unite AWS KMS with Serverless Application Model (SAM)
The Basics
AWS KMS is a Key Management Service that let you create Cryptographic keys that you can use to encrypt and decrypt data and also other keys. You can read more about it here.
Important points about Keys
Please note that the customer master keys(CMK) generated can only be used to encrypt small amount of data like passwords, RSA key. You can use AWS KMS CMKs to generate, encrypt, and decrypt data keys. However, AWS KMS does not store, manage, or track your data keys, or perform cryptographic operations with data keys.
You must use and manage data keys outside of AWS KMS. KMS API uses AWS KMS CMK in the encryption operations and they cannot accept more than 4 KB (4096 bytes) of data. To encrypt application data, use the server-side encryption features of an AWS service, or a client-side encryption library, such as the AWS Encryption SDK or the Amazon S3 encryption client.
Scenario
We want to create signup and login forms for a website.
Passwords should be encrypted and stored in DynamoDB database.
What do we need?
- KMS key to encrypt and decrypt data
- DynamoDB table to store password.
- Lambda functions & APIs to process Login and Sign up forms.
- Sign up/ Login forms in HTML.
Lets Implement it as Serverless Application Model (SAM)!
Lets first create the Key that we will use to encrypt and decrypt password.
KmsKey:
Type: AWS::KMS::Key
Properties:
Description: CMK for encrypting and decrypting
KeyPolicy:
Version: '2012-10-17'
Id: key-default-1
Statement:
- Sid: Enable IAM User Permissions
Effect: Allow
Principal:
AWS: !Sub arn:aws:iam::${AWS::AccountId}:root
Action: kms:*
Resource: '*'
- Sid: Allow administration of the key
Effect: Allow
Principal:
AWS: !Sub arn:aws:iam::${AWS::AccountId}:user/${KeyAdmin}
Action:
- kms:Create*
- kms:Describe*
- kms:Enable*
- kms:List*
- kms:Put*
- kms:Update*
- kms:Revoke*
- kms:Disable*
- kms:Get*
- kms:Delete*
- kms:ScheduleKeyDeletion
- kms:CancelKeyDeletion
Resource: '*'
- Sid: Allow use of the key
Effect: Allow
Principal:
AWS: !Sub arn:aws:iam::${AWS::AccountId}:user/${KeyUser}
Action:
- kms:DescribeKey
- kms:Encrypt
- kms:Decrypt
- kms:ReEncrypt*
- kms:GenerateDataKey
- kms:GenerateDataKeyWithoutPlaintext
Resource: '*'
The important thing in above snippet is the KeyPolicy. KMS requires a Key Administrator and Key User. As a best practice your Key Administrator and Key User should be 2 separate user in your Organisation. We are allowing all permissions to the root users.
So if your key Administrator leaves the organisation, the root user will be able to delete this key. As you can see **KeyAdmin **can manage the key but not use it and KeyUser can only use the key. ${KeyAdmin} and **${KeyUser} **are parameters in the SAM template.
You would be asked to provide values for these parameters during SAM Deploy.
#aws #serverless #aws-sam #aws-key-management-service #aws-certification #aws-api-gateway #tutorial-for-beginners #aws-blogs
1595418900
Serverless Express – Easy APIs On AWS Lambda & AWS HTTP API
TLDR - Take existing Express.js apps and host them easily onto cheap, auto-scaling, serverless infrastructure on AWS Lambda and AWS HTTP API with Serverless Express. It’s packed loads of production-ready features, like custom domains, SSL certificates, canary deployments, and costs ~$0.000003 per request.
If you simply want to host a common Express.js Node.js application, have it auto-scale to billions of requests, and charge you only when it’s used, we have something special for you…
Announcing Serverless Express, a Serverless Framework offering enabling you to easily host and manage Express.js applications on AWS Lambda and the new AWS HTTP API, which is 60% faster and 71% cheaper than their initial API Gateway product.
Serverless Expess is a pure Express.js experience and it’s perfect for those that want to focus on apps, not infrastructure complexity.
Here are the highlights:
- Easy, Safe, Performance - Includes the optimal infrastructure pattern for cost, performance & scale.
- Never Pay For Idle - No API requests? No cost. Averages ~$0.000003 per request.
- Zero Configuration - Add your Express app, then deploy (advanced config options are available).
- Fast Deployments - Deploy changes to the cloud in seconds.
- Real-time Logging - Rapidly develop on the cloud w/ real-time logs and errors in the CLI.
- Canary Deployments - Deploy your app gradually to a subset of your traffic.
- Custom Domain + SSL - Auto-configure a custom domain w/ a free AWS ACM SSL certificate.
- Team Collaboration - Collaborate with your teamates with shared state and outputs.
Here is how to get started and deliver a Serverless Express.js based API with a custom domain, free SSL certificate and much more! You can also check out our Serverless Fullstack Application boilerplate, which includes Serverless Express in a real-world example that features a database, website using React and more.
Set-Up
Serverless Express is a Serverless Framework Component (i.e premium experiences for popular serverless use-cases) and you’ll need to install Node.js and the Serverless Framework CLI to use it.
Install Node.js here.
Then run this command to install Serverless Framework.
npm i -g serverless
Next, install the Serverless Express template:
serverless create --template-url https://github.com/serverless/components/tree/master/templates/express
Lastly, Serverless Express deploys onto your own Amazon Web Services account, so you’ll need Access Keys to an AWS account you own. Follow this guide to create those.
After you have created AWS Access Keys you can add them directly to an .env file, or reference an AWS Profile in a .env file, within the root of the template you installed.
AWS_ACCESS_KEY_ID=123456789
AWS_SECRET_ACCESS_KEY=123456789
You can also reference an AWS Profile in a .env file like this.
AWS_PROFILE=default
If you don’t include a .env file, the Serverless Framework will automatically look for a default AWS Profile in the root folder of your machine.
Also, Serverless Framework has a built-in stages concept. If you change the stage it will deploy a totally separate copy of your serverless application.
# serverless.yml
component: express@1.0.8
name: express-api
stage: prod
Even better, you can use different .env files for each stage by simply using this convention:
.env # all stages
.env.dev # "dev" stage
.env.prod # "prod" stage
One last—often overlooked—step is to install the Express.js dependency, by running npm i in the template.
#serverless #apis #aws #aws lambda #aws http api
1603674601
Integrating API Gateway — Lambda Responses
Probably the most used, versatile and oldest combination between AWS cloud services is REST APIs (public) with Lambda integration on the backend [1]. While RESTful API development is not at all new, if you are relatively new to developing in the cloud, working with these two services may be somewhat challenging. This article requires a basic understanding of these two services. Its focus is beyond the point where you are able to invoke a Lambda function and receive status code 200 from the API. Given the availability of SNS and CloudWatch Logs, why would one bother integrating the API Method & Integration Responses past returning code 200? The answer is that would be a half implemented feature, which lets users speculating about what the result of the invocation was. This article shows how to program an API to return status codes that make sense to your application.
My Use Case
I shall focus on a Lambda function which may throw unexpected exceptions, errors and return a response. To make this more challenging, I am going to use Python. This language is a big hit with developers transitioning from traditional scientific and engineering roles, who have not used any “JS” languages. The intention is to create a foundation on which further API development can be done or at least leave the API in a minimum viable state.
My Solution
I started creating the Lambda function, which I called DanielHTTP. The function uses the latest Python runtime and the IAM policy AWSLambdaBasicExecutionRole which allows CloudWatch logs to be created. I created a role called the same and attached the policy. Alternatively, when this function is created, I could have selected Create New Role to have this done automatically. The function code is available in my GitHub repository [2]. The code is designed to throw an exception by performing an illegal operation, to raise exceptions and errors and to return an Ok response. All cases are individually handled in the API.
#api-gateway #api-development #api-integration #aws-lambda #aws