1615278840
Did you know? You can work with AWS DynamoDB on your local machine. Yes, that’s right. In this article, we are going to set up DynamoDB locally. There are a couple of ways to do this.
Having this local version helps you save on throughput, data storage, and data transfer fees. In addition, you don’t need an internet connection while you develop your application.
Let’s get started.
The downloadable version of Amazon DynamoDB is provided as an executable .jar
file. The application runs on Windows, Linux, macOS, and other platforms that support Java.
After download, extract the content and run the DynamoDB using the following command:
java -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar -sharedDb
Before you can access DynamoDB programmatically or through the AWS Command Line Interface (AWS CLI), you must configure your credentials to enable authorization for your applications. Downloadable DynamoDB requires any credentials to work, as shown in the following example.
AWS Access Key ID: "fakeMyKeyId"
AWS Secret Access Key: "fakeSecretAccessKey"
You can use $aws configure
command to set up the credentials.
Now you can access your local DynamoDB using APIs. For e.g to list local dynamodb tables:
aws dynamodb list-tables --endpoint-url http://localhost:8000
#aws #serverless #database #dynamodb
1651383480
This serverless plugin is a wrapper for amplify-appsync-simulator made for testing AppSync APIs built with serverless-appsync-plugin.
Install
npm install serverless-appsync-simulator
# or
yarn add serverless-appsync-simulator
Usage
This plugin relies on your serverless yml file and on the serverless-offline
plugin.
plugins:
- serverless-dynamodb-local # only if you need dynamodb resolvers and you don't have an external dynamodb
- serverless-appsync-simulator
- serverless-offline
Note: Order is important serverless-appsync-simulator
must go before serverless-offline
To start the simulator, run the following command:
sls offline start
You should see in the logs something like:
...
Serverless: AppSync endpoint: http://localhost:20002/graphql
Serverless: GraphiQl: http://localhost:20002
...
Configuration
Put options under custom.appsync-simulator
in your serverless.yml
file
| option | default | description | | ------------------------ | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | | apiKey | 0123456789
| When using API_KEY
as authentication type, the key to authenticate to the endpoint. | | port | 20002 | AppSync operations port; if using multiple APIs, the value of this option will be used as a starting point, and each other API will have a port of lastPort + 10 (e.g. 20002, 20012, 20022, etc.) | | wsPort | 20003 | AppSync subscriptions port; if using multiple APIs, the value of this option will be used as a starting point, and each other API will have a port of lastPort + 10 (e.g. 20003, 20013, 20023, etc.) | | location | . (base directory) | Location of the lambda functions handlers. | | refMap | {} | A mapping of resource resolutions for the Ref
function | | getAttMap | {} | A mapping of resource resolutions for the GetAtt
function | | importValueMap | {} | A mapping of resource resolutions for the ImportValue
function | | functions | {} | A mapping of external functions for providing invoke url for external fucntions | | dynamoDb.endpoint | http://localhost:8000 | Dynamodb endpoint. Specify it if you're not using serverless-dynamodb-local. Otherwise, port is taken from dynamodb-local conf | | dynamoDb.region | localhost | Dynamodb region. Specify it if you're connecting to a remote Dynamodb intance. | | dynamoDb.accessKeyId | DEFAULT_ACCESS_KEY | AWS Access Key ID to access DynamoDB | | dynamoDb.secretAccessKey | DEFAULT_SECRET | AWS Secret Key to access DynamoDB | | dynamoDb.sessionToken | DEFAULT_ACCESS_TOKEEN | AWS Session Token to access DynamoDB, only if you have temporary security credentials configured on AWS | | dynamoDb.* | | You can add every configuration accepted by DynamoDB SDK | | rds.dbName | | Name of the database | | rds.dbHost | | Database host | | rds.dbDialect | | Database dialect. Possible values (mysql | postgres) | | rds.dbUsername | | Database username | | rds.dbPassword | | Database password | | rds.dbPort | | Database port | | watch | - *.graphql
- *.vtl | Array of glob patterns to watch for hot-reloading. |
Example:
custom:
appsync-simulator:
location: '.webpack/service' # use webpack build directory
dynamoDb:
endpoint: 'http://my-custom-dynamo:8000'
Hot-reloading
By default, the simulator will hot-relad when changes to *.graphql
or *.vtl
files are detected. Changes to *.yml
files are not supported (yet? - this is a Serverless Framework limitation). You will need to restart the simulator each time you change yml files.
Hot-reloading relies on watchman. Make sure it is installed on your system.
You can change the files being watched with the watch
option, which is then passed to watchman as the match expression.
e.g.
custom:
appsync-simulator:
watch:
- ["match", "handlers/**/*.vtl", "wholename"] # => array is interpreted as the literal match expression
- "*.graphql" # => string like this is equivalent to `["match", "*.graphql"]`
Or you can opt-out by leaving an empty array or set the option to false
Note: Functions should not require hot-reloading, unless you are using a transpiler or a bundler (such as webpack, babel or typescript), un which case you should delegate hot-reloading to that instead.
Resource CloudFormation functions resolution
This plugin supports some resources resolution from the Ref
, Fn::GetAtt
and Fn::ImportValue
functions in your yaml file. It also supports some other Cfn functions such as Fn::Join
, Fb::Sub
, etc.
Note: Under the hood, this features relies on the cfn-resolver-lib package. For more info on supported cfn functions, refer to the documentation
You can reference resources in your functions' environment variables (that will be accessible from your lambda functions) or datasource definitions. The plugin will automatically resolve them for you.
provider:
environment:
BUCKET_NAME:
Ref: MyBucket # resolves to `my-bucket-name`
resources:
Resources:
MyDbTable:
Type: AWS::DynamoDB::Table
Properties:
TableName: myTable
...
MyBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: my-bucket-name
...
# in your appsync config
dataSources:
- type: AMAZON_DYNAMODB
name: dynamosource
config:
tableName:
Ref: MyDbTable # resolves to `myTable`
Sometimes, some references cannot be resolved, as they come from an Output from Cloudformation; or you might want to use mocked values in your local environment.
In those cases, you can define (or override) those values using the refMap
, getAttMap
and importValueMap
options.
refMap
takes a mapping of resource name to value pairsgetAttMap
takes a mapping of resource name to attribute/values pairsimportValueMap
takes a mapping of import name to values pairsExample:
custom:
appsync-simulator:
refMap:
# Override `MyDbTable` resolution from the previous example.
MyDbTable: 'mock-myTable'
getAttMap:
# define ElasticSearchInstance DomainName
ElasticSearchInstance:
DomainEndpoint: 'localhost:9200'
importValueMap:
other-service-api-url: 'https://other.api.url.com/graphql'
# in your appsync config
dataSources:
- type: AMAZON_ELASTICSEARCH
name: elasticsource
config:
# endpoint resolves as 'http://localhost:9200'
endpoint:
Fn::Join:
- ''
- - https://
- Fn::GetAtt:
- ElasticSearchInstance
- DomainEndpoint
In some special cases you will need to use key-value mock nottation. Good example can be case when you need to include serverless stage value (${self:provider.stage}
) in the import name.
This notation can be used with all mocks - refMap
, getAttMap
and importValueMap
provider:
environment:
FINISH_ACTIVITY_FUNCTION_ARN:
Fn::ImportValue: other-service-api-${self:provider.stage}-url
custom:
serverless-appsync-simulator:
importValueMap:
- key: other-service-api-${self:provider.stage}-url
value: 'https://other.api.url.com/graphql'
This plugin only tries to resolve the following parts of the yml tree:
provider.environment
functions[*].environment
custom.appSync
If you have the need of resolving others, feel free to open an issue and explain your use case.
For now, the supported resources to be automatically resovled by Ref:
are:
Feel free to open a PR or an issue to extend them as well.
External functions
When a function is not defined withing the current serverless file you can still call it by providing an invoke url which should point to a REST method. Make sure you specify "get" or "post" for the method. Default is "get", but you probably want "post".
custom:
appsync-simulator:
functions:
addUser:
url: http://localhost:3016/2015-03-31/functions/addUser/invocations
method: post
addPost:
url: https://jsonplaceholder.typicode.com/posts
method: post
Supported Resolver types
This plugin supports resolvers implemented by amplify-appsync-simulator
, as well as custom resolvers.
From Aws Amplify:
Implemented by this plugin
#set( $cols = [] )
#set( $vals = [] )
#foreach( $entry in $ctx.args.input.keySet() )
#set( $regex = "([a-z])([A-Z]+)")
#set( $replacement = "$1_$2")
#set( $toSnake = $entry.replaceAll($regex, $replacement).toLowerCase() )
#set( $discard = $cols.add("$toSnake") )
#if( $util.isBoolean($ctx.args.input[$entry]) )
#if( $ctx.args.input[$entry] )
#set( $discard = $vals.add("1") )
#else
#set( $discard = $vals.add("0") )
#end
#else
#set( $discard = $vals.add("'$ctx.args.input[$entry]'") )
#end
#end
#set( $valStr = $vals.toString().replace("[","(").replace("]",")") )
#set( $colStr = $cols.toString().replace("[","(").replace("]",")") )
#if ( $valStr.substring(0, 1) != '(' )
#set( $valStr = "($valStr)" )
#end
#if ( $colStr.substring(0, 1) != '(' )
#set( $colStr = "($colStr)" )
#end
{
"version": "2018-05-29",
"statements": ["INSERT INTO <name-of-table> $colStr VALUES $valStr", "SELECT * FROM <name-of-table> ORDER BY id DESC LIMIT 1"]
}
#set( $update = "" )
#set( $equals = "=" )
#foreach( $entry in $ctx.args.input.keySet() )
#set( $cur = $ctx.args.input[$entry] )
#set( $regex = "([a-z])([A-Z]+)")
#set( $replacement = "$1_$2")
#set( $toSnake = $entry.replaceAll($regex, $replacement).toLowerCase() )
#if( $util.isBoolean($cur) )
#if( $cur )
#set ( $cur = "1" )
#else
#set ( $cur = "0" )
#end
#end
#if ( $util.isNullOrEmpty($update) )
#set($update = "$toSnake$equals'$cur'" )
#else
#set($update = "$update,$toSnake$equals'$cur'" )
#end
#end
{
"version": "2018-05-29",
"statements": ["UPDATE <name-of-table> SET $update WHERE id=$ctx.args.input.id", "SELECT * FROM <name-of-table> WHERE id=$ctx.args.input.id"]
}
{
"version": "2018-05-29",
"statements": ["UPDATE <name-of-table> set deleted_at=NOW() WHERE id=$ctx.args.id", "SELECT * FROM <name-of-table> WHERE id=$ctx.args.id"]
}
#set ( $index = -1)
#set ( $result = $util.parseJson($ctx.result) )
#set ( $meta = $result.sqlStatementResults[1].columnMetadata)
#foreach ($column in $meta)
#set ($index = $index + 1)
#if ( $column["typeName"] == "timestamptz" )
#set ($time = $result["sqlStatementResults"][1]["records"][0][$index]["stringValue"] )
#set ( $nowEpochMillis = $util.time.parseFormattedToEpochMilliSeconds("$time.substring(0,19)+0000", "yyyy-MM-dd HH:mm:ssZ") )
#set ( $isoDateTime = $util.time.epochMilliSecondsToISO8601($nowEpochMillis) )
$util.qr( $result["sqlStatementResults"][1]["records"][0][$index].put("stringValue", "$isoDateTime") )
#end
#end
#set ( $res = $util.parseJson($util.rds.toJsonString($util.toJson($result)))[1][0] )
#set ( $response = {} )
#foreach($mapKey in $res.keySet())
#set ( $s = $mapKey.split("_") )
#set ( $camelCase="" )
#set ( $isFirst=true )
#foreach($entry in $s)
#if ( $isFirst )
#set ( $first = $entry.substring(0,1) )
#else
#set ( $first = $entry.substring(0,1).toUpperCase() )
#end
#set ( $isFirst=false )
#set ( $stringLength = $entry.length() )
#set ( $remaining = $entry.substring(1, $stringLength) )
#set ( $camelCase = "$camelCase$first$remaining" )
#end
$util.qr( $response.put("$camelCase", $res[$mapKey]) )
#end
$utils.toJson($response)
Variable map support is limited and does not differentiate numbers and strings data types, please inject them directly if needed.
Will be escaped properly: null
, true
, and false
values.
{
"version": "2018-05-29",
"statements": [
"UPDATE <name-of-table> set deleted_at=NOW() WHERE id=:ID",
"SELECT * FROM <name-of-table> WHERE id=:ID and unix_timestamp > $ctx.args.newerThan"
],
variableMap: {
":ID": $ctx.args.id,
## ":TIMESTAMP": $ctx.args.newerThan -- This will be handled as a string!!!
}
}
Requires
Author: Serverless-appsync
Source Code: https://github.com/serverless-appsync/serverless-appsync-simulator
License: MIT License
1624352400
Lando is an amazing tool and it’s so handy; just a few commands and you are ready to run your project locally or set up a new project on your machine. It’s open source built on Docker. I assume that many developers know this amazing tool, it is trending and most of the developers use it.
Requirements
#web dev #set up #local dev environment #lando
1651319520
Serverless APIGateway Service Proxy
This Serverless Framework plugin supports the AWS service proxy integration feature of API Gateway. You can directly connect API Gateway to AWS services without Lambda.
Run serverless plugin install
in your Serverless project.
serverless plugin install -n serverless-apigateway-service-proxy
Here is a services list which this plugin supports for now. But will expand to other services in the feature. Please pull request if you are intersted in it.
Define settings of the AWS services you want to integrate under custom > apiGatewayServiceProxies
and run serverless deploy
.
Sample syntax for Kinesis proxy in serverless.yml
.
custom:
apiGatewayServiceProxies:
- kinesis: # partitionkey is set apigateway requestid by default
path: /kinesis
method: post
streamName: { Ref: 'YourStream' }
cors: true
- kinesis:
path: /kinesis
method: post
partitionKey: 'hardcordedkey' # use static partitionkey
streamName: { Ref: 'YourStream' }
cors: true
- kinesis:
path: /kinesis/{myKey} # use path parameter
method: post
partitionKey:
pathParam: myKey
streamName: { Ref: 'YourStream' }
cors: true
- kinesis:
path: /kinesis
method: post
partitionKey:
bodyParam: data.myKey # use body parameter
streamName: { Ref: 'YourStream' }
cors: true
- kinesis:
path: /kinesis
method: post
partitionKey:
queryStringParam: myKey # use query string param
streamName: { Ref: 'YourStream' }
cors: true
- kinesis: # PutRecords
path: /kinesis
method: post
action: PutRecords
streamName: { Ref: 'YourStream' }
cors: true
resources:
Resources:
YourStream:
Type: AWS::Kinesis::Stream
Properties:
ShardCount: 1
Sample request after deploying.
curl https://xxxxxxx.execute-api.us-east-1.amazonaws.com/dev/kinesis -d '{"message": "some data"}' -H 'Content-Type:application/json'
Sample syntax for SQS proxy in serverless.yml
.
custom:
apiGatewayServiceProxies:
- sqs:
path: /sqs
method: post
queueName: { 'Fn::GetAtt': ['SQSQueue', 'QueueName'] }
cors: true
resources:
Resources:
SQSQueue:
Type: 'AWS::SQS::Queue'
Sample request after deploying.
curl https://xxxxxx.execute-api.us-east-1.amazonaws.com/dev/sqs -d '{"message": "testtest"}' -H 'Content-Type:application/json'
If you'd like to pass additional data to the integration request, you can do so by including your custom API Gateway request parameters in serverless.yml
like so:
custom:
apiGatewayServiceProxies:
- sqs:
path: /queue
method: post
queueName: !GetAtt MyQueue.QueueName
cors: true
requestParameters:
'integration.request.querystring.MessageAttribute.1.Name': "'cognitoIdentityId'"
'integration.request.querystring.MessageAttribute.1.Value.StringValue': 'context.identity.cognitoIdentityId'
'integration.request.querystring.MessageAttribute.1.Value.DataType': "'String'"
'integration.request.querystring.MessageAttribute.2.Name': "'cognitoAuthenticationProvider'"
'integration.request.querystring.MessageAttribute.2.Value.StringValue': 'context.identity.cognitoAuthenticationProvider'
'integration.request.querystring.MessageAttribute.2.Value.DataType': "'String'"
The alternative way to pass MessageAttribute
parameters is via a request body mapping template.
See the SQS section under Customizing request body mapping templates
Simplified response template customization
You can get a simple customization of the responses by providing a template for the possible responses. The template is assumed to be application/json
.
custom:
apiGatewayServiceProxies:
- sqs:
path: /queue
method: post
queueName: !GetAtt MyQueue.QueueName
cors: true
response:
template:
# `success` is used when the integration response is 200
success: |-
{ "message: "accepted" }
# `clientError` is used when the integration response is 400
clientError: |-
{ "message": "there is an error in your request" }
# `serverError` is used when the integration response is 500
serverError: |-
{ "message": "there was an error handling your request" }
Full response customization
If you want more control over the integration response, you can provide an array of objects for the response
value:
custom:
apiGatewayServiceProxies:
- sqs:
path: /queue
method: post
queueName: !GetAtt MyQueue.QueueName
cors: true
response:
- statusCode: 200
selectionPattern: '2\\d{2}'
responseParameters: {}
responseTemplates:
application/json: |-
{ "message": "accepted" }
The object keys correspond to the API Gateway integration response object.
Sample syntax for S3 proxy in serverless.yml
.
custom:
apiGatewayServiceProxies:
- s3:
path: /s3
method: post
action: PutObject
bucket:
Ref: S3Bucket
key: static-key.json # use static key
cors: true
- s3:
path: /s3/{myKey} # use path param
method: get
action: GetObject
bucket:
Ref: S3Bucket
key:
pathParam: myKey
cors: true
- s3:
path: /s3
method: delete
action: DeleteObject
bucket:
Ref: S3Bucket
key:
queryStringParam: key # use query string param
cors: true
resources:
Resources:
S3Bucket:
Type: 'AWS::S3::Bucket'
Sample request after deploying.
curl https://xxxxxx.execute-api.us-east-1.amazonaws.com/dev/s3 -d '{"message": "testtest"}' -H 'Content-Type:application/json'
Similar to the SQS support, you can customize the default request parameters serverless.yml
like so:
custom:
apiGatewayServiceProxies:
- s3:
path: /s3
method: post
action: PutObject
bucket:
Ref: S3Bucket
cors: true
requestParameters:
# if requestParameters has a 'integration.request.path.object' property you should remove the key setting
'integration.request.path.object': 'context.requestId'
'integration.request.header.cache-control': "'public, max-age=31536000, immutable'"
If you'd like use custom API Gateway request templates, you can do so like so:
custom:
apiGatewayServiceProxies:
- s3:
path: /s3
method: get
action: GetObject
bucket:
Ref: S3Bucket
request:
template:
application/json: |
#set ($specialStuff = $context.request.header.x-special)
#set ($context.requestOverride.path.object = $specialStuff.replaceAll('_', '-'))
{}
Note that if the client does not provide a Content-Type
header in the request, ApiGateway defaults to application/json
.
Added the new customization parameter that lets the user set a custom Path Override in API Gateway other than the {bucket}/{object}
This parameter is optional and if not set, will fall back to {bucket}/{object}
The Path Override will add {bucket}/
automatically in front
Please keep in mind, that key or path.object still needs to be set at the moment (maybe this will be made optional later on with this)
Usage (With 2 Path Parameters (folder and file and a fixed file extension)):
custom:
apiGatewayServiceProxies:
- s3:
path: /s3/{folder}/{file}
method: get
action: GetObject
pathOverride: '{folder}/{file}.xml'
bucket:
Ref: S3Bucket
cors: true
requestParameters:
# if requestParameters has a 'integration.request.path.object' property you should remove the key setting
'integration.request.path.folder': 'method.request.path.folder'
'integration.request.path.file': 'method.request.path.file'
'integration.request.path.object': 'context.requestId'
'integration.request.header.cache-control': "'public, max-age=31536000, immutable'"
This will result in API Gateway setting the Path Override attribute to {bucket}/{folder}/{file}.xml
So for example if you navigate to the API Gatway endpoint /language/en
it will fetch the file in S3 from {bucket}/language/en.xml
Can use greedy, for deeper Folders
The forementioned example can also be shortened by a greedy approach. Thanks to @taylorreece for mentioning this.
custom:
apiGatewayServiceProxies:
- s3:
path: /s3/{myPath+}
method: get
action: GetObject
pathOverride: '{myPath}.xml'
bucket:
Ref: S3Bucket
cors: true
requestParameters:
# if requestParameters has a 'integration.request.path.object' property you should remove the key setting
'integration.request.path.myPath': 'method.request.path.myPath'
'integration.request.path.object': 'context.requestId'
'integration.request.header.cache-control': "'public, max-age=31536000, immutable'"
This will translate for example /s3/a/b/c
to a/b/c.xml
You can get a simple customization of the responses by providing a template for the possible responses. The template is assumed to be application/json
.
custom:
apiGatewayServiceProxies:
- s3:
path: /s3
method: post
action: PutObject
bucket:
Ref: S3Bucket
key: static-key.json
response:
template:
# `success` is used when the integration response is 200
success: |-
{ "message: "accepted" }
# `clientError` is used when the integration response is 400
clientError: |-
{ "message": "there is an error in your request" }
# `serverError` is used when the integration response is 500
serverError: |-
{ "message": "there was an error handling your request" }
Sample syntax for SNS proxy in serverless.yml
.
custom:
apiGatewayServiceProxies:
- sns:
path: /sns
method: post
topicName: { 'Fn::GetAtt': ['SNSTopic', 'TopicName'] }
cors: true
resources:
Resources:
SNSTopic:
Type: AWS::SNS::Topic
Sample request after deploying.
curl https://xxxxxx.execute-api.us-east-1.amazonaws.com/dev/sns -d '{"message": "testtest"}' -H 'Content-Type:application/json'
Simplified response template customization
You can get a simple customization of the responses by providing a template for the possible responses. The template is assumed to be application/json
.
custom:
apiGatewayServiceProxies:
- sns:
path: /sns
method: post
topicName: { 'Fn::GetAtt': ['SNSTopic', 'TopicName'] }
cors: true
response:
template:
# `success` is used when the integration response is 200
success: |-
{ "message: "accepted" }
# `clientError` is used when the integration response is 400
clientError: |-
{ "message": "there is an error in your request" }
# `serverError` is used when the integration response is 500
serverError: |-
{ "message": "there was an error handling your request" }
Full response customization
If you want more control over the integration response, you can provide an array of objects for the response
value:
custom:
apiGatewayServiceProxies:
- sns:
path: /sns
method: post
topicName: { 'Fn::GetAtt': ['SNSTopic', 'TopicName'] }
cors: true
response:
- statusCode: 200
selectionPattern: '2\d{2}'
responseParameters: {}
responseTemplates:
application/json: |-
{ "message": "accepted" }
The object keys correspond to the API Gateway integration response object.
Content Handling and Pass Through Behaviour customization
If you want to work with binary fata, you can not specify contentHandling
and PassThrough
inside the request
object.
custom:
apiGatewayServiceProxies:
- sns:
path: /sns
method: post
topicName: { 'Fn::GetAtt': ['SNSTopic', 'TopicName'] }
request:
contentHandling: CONVERT_TO_TEXT
passThrough: WHEN_NO_TEMPLATES
The allowed values correspond with the API Gateway Method integration for ContentHandling and PassthroughBehavior
Sample syntax for DynamoDB proxy in serverless.yml
. Currently, the supported DynamoDB Operations are PutItem
, GetItem
and DeleteItem
.
custom:
apiGatewayServiceProxies:
- dynamodb:
path: /dynamodb/{id}/{sort}
method: put
tableName: { Ref: 'YourTable' }
hashKey: # set pathParam or queryStringParam as a partitionkey.
pathParam: id
attributeType: S
rangeKey: # required if also using sort key. set pathParam or queryStringParam.
pathParam: sort
attributeType: S
action: PutItem # specify action to the table what you want
condition: attribute_not_exists(Id) # optional Condition Expressions parameter for the table
cors: true
- dynamodb:
path: /dynamodb
method: get
tableName: { Ref: 'YourTable' }
hashKey:
queryStringParam: id # use query string parameter
attributeType: S
rangeKey:
queryStringParam: sort
attributeType: S
action: GetItem
cors: true
- dynamodb:
path: /dynamodb/{id}
method: delete
tableName: { Ref: 'YourTable' }
hashKey:
pathParam: id
attributeType: S
action: DeleteItem
cors: true
resources:
Resources:
YourTable:
Type: AWS::DynamoDB::Table
Properties:
TableName: YourTable
AttributeDefinitions:
- AttributeName: id
AttributeType: S
- AttributeName: sort
AttributeType: S
KeySchema:
- AttributeName: id
KeyType: HASH
- AttributeName: sort
KeyType: RANGE
ProvisionedThroughput:
ReadCapacityUnits: 1
WriteCapacityUnits: 1
Sample request after deploying.
curl -XPUT https://xxxxxxx.execute-api.us-east-1.amazonaws.com/dev/dynamodb/<hashKey>/<sortkey> \
-d '{"name":{"S":"john"},"address":{"S":"xxxxx"}}' \
-H 'Content-Type:application/json'
Sample syntax for EventBridge proxy in serverless.yml
.
custom:
apiGatewayServiceProxies:
- eventbridge: # source and detailType are hardcoded; detail defaults to POST body
path: /eventbridge
method: post
source: 'hardcoded_source'
detailType: 'hardcoded_detailType'
eventBusName: { Ref: 'YourBusName' }
cors: true
- eventbridge: # source and detailType as path parameters
path: /eventbridge/{detailTypeKey}/{sourceKey}
method: post
detailType:
pathParam: detailTypeKey
source:
pathParam: sourceKey
eventBusName: { Ref: 'YourBusName' }
cors: true
- eventbridge: # source, detail, and detailType as body parameters
path: /eventbridge/{detailTypeKey}/{sourceKey}
method: post
detailType:
bodyParam: data.detailType
source:
bodyParam: data.source
detail:
bodyParam: data.detail
eventBusName: { Ref: 'YourBusName' }
cors: true
resources:
Resources:
YourBus:
Type: AWS::Events::EventBus
Properties:
Name: YourEventBus
Sample request after deploying.
curl https://xxxxxxx.execute-api.us-east-1.amazonaws.com/dev/eventbridge -d '{"message": "some data"}' -H 'Content-Type:application/json'
To set CORS configurations for your HTTP endpoints, simply modify your event configurations as follows:
custom:
apiGatewayServiceProxies:
- kinesis:
path: /kinesis
method: post
streamName: { Ref: 'YourStream' }
cors: true
Setting cors to true assumes a default configuration which is equivalent to:
custom:
apiGatewayServiceProxies:
- kinesis:
path: /kinesis
method: post
streamName: { Ref: 'YourStream' }
cors:
origin: '*'
headers:
- Content-Type
- X-Amz-Date
- Authorization
- X-Api-Key
- X-Amz-Security-Token
- X-Amz-User-Agent
allowCredentials: false
Configuring the cors property sets Access-Control-Allow-Origin, Access-Control-Allow-Headers, Access-Control-Allow-Methods,Access-Control-Allow-Credentials headers in the CORS preflight response. To enable the Access-Control-Max-Age preflight response header, set the maxAge property in the cors object:
custom:
apiGatewayServiceProxies:
- kinesis:
path: /kinesis
method: post
streamName: { Ref: 'YourStream' }
cors:
origin: '*'
maxAge: 86400
If you are using CloudFront or another CDN for your API Gateway, you may want to setup a Cache-Control header to allow for OPTIONS request to be cached to avoid the additional hop.
To enable the Cache-Control header on preflight response, set the cacheControl property in the cors object:
custom:
apiGatewayServiceProxies:
- kinesis:
path: /kinesis
method: post
streamName: { Ref: 'YourStream' }
cors:
origin: '*'
headers:
- Content-Type
- X-Amz-Date
- Authorization
- X-Api-Key
- X-Amz-Security-Token
- X-Amz-User-Agent
allowCredentials: false
cacheControl: 'max-age=600, s-maxage=600, proxy-revalidate' # Caches on browser and proxy for 10 minutes and doesnt allow proxy to serve out of date content
You can pass in any supported authorization type:
custom:
apiGatewayServiceProxies:
- sqs:
path: /sqs
method: post
queueName: { 'Fn::GetAtt': ['SQSQueue', 'QueueName'] }
cors: true
# optional - defaults to 'NONE'
authorizationType: 'AWS_IAM' # can be one of ['NONE', 'AWS_IAM', 'CUSTOM', 'COGNITO_USER_POOLS']
# when using 'CUSTOM' authorization type, one should specify authorizerId
# authorizerId: { Ref: 'AuthorizerLogicalId' }
# when using 'COGNITO_USER_POOLS' authorization type, one can specify a list of authorization scopes
# authorizationScopes: ['scope1','scope2']
resources:
Resources:
SQSQueue:
Type: 'AWS::SQS::Queue'
Source: AWS::ApiGateway::Method docs
You can indicate whether the method requires clients to submit a valid API key using private
flag:
custom:
apiGatewayServiceProxies:
- sqs:
path: /sqs
method: post
queueName: { 'Fn::GetAtt': ['SQSQueue', 'QueueName'] }
cors: true
private: true
resources:
Resources:
SQSQueue:
Type: 'AWS::SQS::Queue'
which is the same syntax used in Serverless framework.
Source: Serverless: Setting API keys for your Rest API
Source: AWS::ApiGateway::Method docs
By default, the plugin will generate a role with the required permissions for each service type that is configured.
You can configure your own role by setting the roleArn
attribute:
custom:
apiGatewayServiceProxies:
- sqs:
path: /sqs
method: post
queueName: { 'Fn::GetAtt': ['SQSQueue', 'QueueName'] }
cors: true
roleArn: # Optional. A default role is created when not configured
Fn::GetAtt: [CustomS3Role, Arn]
resources:
Resources:
SQSQueue:
Type: 'AWS::SQS::Queue'
CustomS3Role:
# Custom Role definition
Type: 'AWS::IAM::Role'
The plugin allows one to specify which parameters the API Gateway method accepts.
A common use case is to pass custom data to the integration request:
custom:
apiGatewayServiceProxies:
- sqs:
path: /sqs
method: post
queueName: { 'Fn::GetAtt': ['SqsQueue', 'QueueName'] }
cors: true
acceptParameters:
'method.request.header.Custom-Header': true
requestParameters:
'integration.request.querystring.MessageAttribute.1.Name': "'custom-Header'"
'integration.request.querystring.MessageAttribute.1.Value.StringValue': 'method.request.header.Custom-Header'
'integration.request.querystring.MessageAttribute.1.Value.DataType': "'String'"
resources:
Resources:
SqsQueue:
Type: 'AWS::SQS::Queue'
Any published SQS message will have the Custom-Header
value added as a message attribute.
If you'd like to add content types or customize the default templates, you can do so by including your custom API Gateway request mapping template in serverless.yml
like so:
# Required for using Fn::Sub
plugins:
- serverless-cloudformation-sub-variables
custom:
apiGatewayServiceProxies:
- kinesis:
path: /kinesis
method: post
streamName: { Ref: 'MyStream' }
request:
template:
text/plain:
Fn::Sub:
- |
#set($msgBody = $util.parseJson($input.body))
#set($msgId = $msgBody.MessageId)
{
"Data": "$util.base64Encode($input.body)",
"PartitionKey": "$msgId",
"StreamName": "#{MyStreamArn}"
}
- MyStreamArn:
Fn::GetAtt: [MyStream, Arn]
It is important that the mapping template will return a valid
application/json
string
Source: How to connect SNS to Kinesis for cross-account delivery via API Gateway
Customizing SQS request templates requires us to force all requests to use an application/x-www-form-urlencoded
style body. The plugin sets the Content-Type
header to application/x-www-form-urlencoded
for you, but API Gateway will still look for the template under the application/json
request template type, so that is where you need to configure you request body in serverless.yml
:
custom:
apiGatewayServiceProxies:
- sqs:
path: /{version}/event/receiver
method: post
queueName: { 'Fn::GetAtt': ['SqsQueue', 'QueueName'] }
request:
template:
application/json: |-
#set ($body = $util.parseJson($input.body))
Action=SendMessage##
&MessageGroupId=$util.urlEncode($body.event_type)##
&MessageDeduplicationId=$util.urlEncode($body.event_id)##
&MessageAttribute.1.Name=$util.urlEncode("X-Custom-Signature")##
&MessageAttribute.1.Value.DataType=String##
&MessageAttribute.1.Value.StringValue=$util.urlEncode($input.params("X-Custom-Signature"))##
&MessageBody=$util.urlEncode($input.body)
Note that the ##
at the end of each line is an empty comment. In VTL this has the effect of stripping the newline from the end of the line (as it is commented out), which makes API Gateway read all the lines in the template as one line.
Be careful when mixing additional requestParameters
into your SQS endpoint as you may overwrite the integration.request.header.Content-Type
and stop the request template from being parsed correctly. You may also unintentionally create conflicts between parameters passed using requestParameters
and those in your request template. Typically you should only use the request template if you need to manipulate the incoming request body in some way.
Your custom template must also set the Action
and MessageBody
parameters, as these will not be added for you by the plugin.
When using a custom request body, headers sent by a client will no longer be passed through to the SQS queue (PassthroughBehavior
is automatically set to NEVER
). You will need to pass through headers sent by the client explicitly in the request body. Also, any custom querystring parameters in the requestParameters
array will be ignored. These also need to be added via the custom request body.
Similar to the Kinesis support, you can customize the default request mapping templates in serverless.yml
like so:
# Required for using Fn::Sub
plugins:
- serverless-cloudformation-sub-variables
custom:
apiGatewayServiceProxies:
- kinesis:
path: /sns
method: post
topicName: { 'Fn::GetAtt': ['SNSTopic', 'TopicName'] }
request:
template:
application/json:
Fn::Sub:
- "Action=Publish&Message=$util.urlEncode('This is a fixed message')&TopicArn=$util.urlEncode('#{MyTopicArn}')"
- MyTopicArn: { Ref: MyTopic }
It is important that the mapping template will return a valid
application/x-www-form-urlencoded
string
Source: Connect AWS API Gateway directly to SNS using a service integration
You can customize the response body by providing mapping templates for success, server errors (5xx) and client errors (4xx).
Templates must be in JSON format. If a template isn't provided, the integration response will be returned as-is to the client.
custom:
apiGatewayServiceProxies:
- kinesis:
path: /kinesis
method: post
streamName: { Ref: 'MyStream' }
response:
template:
success: |
{
"success": true
}
serverError: |
{
"success": false,
"errorMessage": "Server Error"
}
clientError: |
{
"success": false,
"errorMessage": "Client Error"
}
Author: Serverless-operations
Source Code: https://github.com/serverless-operations/serverless-apigateway-service-proxy
License:
1672833558
Bash has many environment variables for various purposes. The set command of Bash is used to modify or display the different attributes and parameters of the shell environment. This command has many options to perform the different types of tasks. The uses of set command for various purposes are described in this tutorial.
set [options] [arguments]
This command can be used with different types of options and arguments for different purposes. If no option or argument is used with this command, the shell variables are printed. The minus sign (-) is used with the command’s option to enable that option and the plus sign (+) is used with the command’s option to disable that option.
Three exit values can be returned by this command which are mentioned in the following:
The purposes of the most commonly used options of the set command are described in this part of the tutorial.
Option | Purpose |
---|---|
-a | It defines those variables or functions which are created or modified or exported. |
-b | It informs the job termination. |
-B | To do the task of the brace expansion. |
-C | It disables the overwriting feature of the existing file. |
-e | It exits for non-zero exit status value. |
-f | It disables the filename generation task. |
-h | It saves the location of the command where it has been used. |
-m | It enables job control. |
-n | It reads the commands. |
-t | It exits from the command after executing a single command. |
-u | It traces the unset variables. |
-v | It prints the shell input lines. |
-x | It displays the commands and their attributes sequentially. It is mainly used to debug the script. |
The uses of set command with different options are shown in this part of this tutorial.
Example 1: Using the Set Command with -a Option
Create a Bash file with the following script that enables the “set –a” command and initialize three variables named $v1, $v2, and $v3. These variables can be accessed after executing the script.
#!/bin/bash
#Enable -a option to read the values of the variables
set -a
#Initialize three variables
v1=78
v2=50
v3=35
Run the script using the following command:
$ bash set1.bash
Read the values of the variable using the “echo” command:
$ echo $v1 $v2 $v3
The following output appears after executing the previous commands:
Example 2: Using the Set Command with -C Option
Run the “cat” command to create a text file named testfile.txt. Next, run the “set –C” command to disable the overwriting feature. Next, run the “cat” command again to overwrite the file to check whether the overwriting feature is disabled or not.
$ cat > testfile.txt
$ set -C
$ cat > testfile.txt
The following output appears after executing the previous commands:
Example 3: Using the Set Command with -x Option
Create a Bash file with the following script that declares a numeric array of 6 elements. The values of the array are printed using for loop.
#!/bin/bash
#Declare an array
arr=(67 3 90 56 2 80)
#iterate the array values
for value in ${arr[@]}
do
echo $value
done
Execute the previous script by the following command:
$ bash set3.bash
Enable the debugging option using the following command:
$ set -x
The following output appears after executing the provided commands:
Example 4: Using the Set Command with -e Option
Create a Bash file with the following script that reads a file using the “cat” command before and after using the “set –e” command.
#!/bin/bash
#Read a non-existing file without setting set -e
cat myfile.txt
echo "Reading a file..."
#Set the set command with -e option
set -e
#Read a non-existing file after setting set -e
cat myfile.txt
echo "Reading a file..."
The following output appears after executing the provided commands. The first error message is shown because the file does not exist in the current location. The next message is then printed. But after executing the “set –e” command, the execution stops after displaying the error message.
Example 5: Using the Set Command with -u Option
Create a Bash file with the following script that initializes a variable but prints the initialized and uninitialized variable before and after using the “set –u” command.
#!/bin/bash
#Assign value to a variable
strvar="Bash Programming"
printf "$strvar $intvar\n"
#Set the set command with -u option
set -u
#Assign value to a variable
strvar="Bash Programming"
printf "\n$strvar $intvar\n"
The following output appears after executing the previous script. Here, the error is printed for the uninitialized variable:
Example 6: Using the Set Command with -f Option
Run the following command to print the list of all text files of the current location:
$ ls *.txt
Run the following command to disable the globbing:
$ set –f
Run the following command again to print the list of all text files of the current location:
$ ls *.txt
The following output appears after executing the previous script. Based on the output, the “ls *.txt” command did not work after setting “set –f” command:
Example 7: Split the String Using the Set Command with Variable
Create a Bash file with the following script that splits the string value based on the space using the “set – variable” command. The split values are printed later.
#!/bin/bash
#Define a string variable
myvar="Learn bash programming"
#Set the set command without option and with variable
set -- $myvar
#Print the split value
printf "$1\n$2\n$3\n"
The following output appears after executing the previous script. The string value is divided into three parts based on the space that is printed:
The uses of the different options of the “set” command are shown in this tutorial using multiple examples to know the basic uses of this command.
Original article source at: https://linuxhint.com/
1593585300
1. Install the package globally from npm.
$ npm install -g dynamodb-admin
2. Run DynamoDB locally inside a Docker container
Make sure you have Docker installed on your system. Instructions are here.
Now pull and run the Docker dynamodb-local image to spin up your very own DynamoDB instance running on port 8000.
$ docker pull amazon/dynamodb-local
$ docker run -p 8000:8000 amazon/dynamodb-local
3. Start dynamodb-admin (with defaults)
MacOS/Linux
$ dynamodb-admin
Windows
> export DYNAMO_ENDPOINT=http://localhost:8000
> dynamodb-admin
After these steps you will have:
The next step is to create a table and start reading/writing to it!
You may need to override regions, endpoints and/or credentials to peek inside local DynamoDB instances you have spun up to replicate a production environment.
If so, just override the defaults when starting the service. You can override some, or all of these, as required.
DYNAMO_ENDPOINT=http://localhost:<PORT> AWS_REGION=<AWS-REGION> AWS_ACCESS_KEY_ID=<YOUR-ACCESS-KEY> AWS_SECRET_ACCESS_KEY=<YOUR-SECRET> dynamodb-admin
Here we are going to create your table using the dynamodb-admin GUI. This is most likely going to be appropriate for your use-case.
Clicking on ‘Create table’ takes us to a screen where we can define how our table should look. Remember that DynamoDB is effectively a key-value store, meaning to get started we only need to define a table name and a hash attribute (the primary key). If you expect that you’ll need to perform lookups based on another attribute of your data, you may want to add some Secondary Indices.
In this example, I’ve named the table ‘Cats’ and given it a primary index ‘name’ and a secondary index ‘owner’. Both indices are of type String and we must give the secondary index a name — which can be different from the name of the attribute.
‘name’ is not a good choice of primary index in practice, as it means only one cat with a given name can be present in the table. Instead, it would be better to give each cat a unique id and use that as the primary index.
Finally, add your first item to the table by clicking the ‘Create item’ link on the top right. This will take you to a new screen where you can enter the json which defines the record. The only requirement for each entry is that the primary key is included.
Using the GUI to set up tables is fine for simple tables, or when you’re just exploring how your data storage might be structured. However, if you are trying to mimic a complex table, or you want to stand-up tables quickly for testing you may want to use the command line to create the table(s) for you.
1. Install the AWS CLI
Instructions for installing (for Mac) via the command line are here. This is a very powerful utility tool. I’d recommend installing it if you work with AWS even if you don’t opt to use it here.
2. Create a table schema
If you already have a table schema you can skip this and move along to Step 3.
$ aws dynamodb create-table --generate-cli-skeleton > dynamo_table_def.json
This will create a basic table schema and pipe it into a file named dynamo_table_def.json
.
You can now open the json file and edit it to fit your desired table schema. This is great because you can now use this same schema file when you need the table, rather than manually setting it up each time via the GUI.
If you’re just curious what the schema should look like, or you need some inspiration for your own — here is the schema for the Cats table.
{
"AttributeDefinitions": [
{
"AttributeName": "name",
"AttributeType": "S"
},
{
"AttributeName": "owner",
"AttributeType": "S"
}
],
"TableName": "Cats",
"KeySchema": [
{
"AttributeName": "name",
"KeyType": "HASH"
}
],
"ProvisionedThroughput": {
"ReadCapacityUnits": 3,
"WriteCapacityUnits": 3
},
"GlobalSecondaryIndexes": [
{
"IndexName": "idx_owner",
"KeySchema": [
{
"AttributeName": "owner",
"KeyType": "HASH"
}
],
"Projection": {
"ProjectionType": "ALL"
},
"ProvisionedThroughput": {
"ReadCapacityUnits": 3,
"WriteCapacityUnits": 3
}
}
]
}
3. Use the schema to create the table
Finally, create the table locally
$ aws dynamodb create-table --cli-input-json file://dynamo_table_def.json --endpoint-url http://localhost:8000
Remember the --endpoint-url
parameter, otherwise a real table will be created in whatever region your AWS CLI defaults to.
After running this command, go back to dynamodb-admin in your browser. You’ll see your table has been created. Now it’s time to use it!
Use Cases
I’ve picked 3 examples to show how dynamodb-admin can help you develop and test your applications.
This example is super simple. Let’s say you’re developing a Python application which reads from a DynamoDB table of movies. You may want run a local DynamoDB instance for development and tests, to avoid standing up unnecessary infrastructure. Dynamodb-local is a godsend for this. However it can be fiddly to put data in the table, from the command line.
You could write code to put the correct items in the table. Indeed for tests this might be ideal, as you absolutely should test the logic you’re using to read and write from the table.
However to quickly test some code path or to build out a feature, when the remote infrastructure or data is not present, it’s typically much easier to put the data into the DynamoDB table manually.
You can set up your table and add some movies, using the GUI, as described above. Then read from the table like so:
from pprint import pprint
import boto3
from botocore.exceptions import ClientError
def get_movie(title, year, dynamodb=None):
if not dynamodb:
dynamodb = boto3.resource('dynamodb', endpoint_url="http://localhost:8000")
table = dynamodb.Table('Movies')
try:
response = table.get_item(Key={'year': year, 'title': title})
except ClientError as e:
print(e.response['Error']['Message'])
else:
return response['Item']
if __name__ == '__main__':
movie = get_movie("The Big New Movie", 2015,)
if movie:
print("Get movie succeeded:")
pprint(movie, sort_dicts=False)
Creating the table and putting items in it, using dynamodb-admin, lets you focus on the business logic. When you’re happy with how your logic looks, you can focus on writing to the table.
Since dynamodb-admin is a Node library, we can use it inside our Node projects. This is again great for local development, as each time you run the service you have what is effectively the AWS console ready to view and manipulate the data.
const AWS = require('aws-sdk');
const {createServer} = require('dynamodb-admin');
const dynamodb = new AWS.DynamoDB();
const dynClient = new AWS.DynamoDB.DocumentClient({service: dynamodb});
const app = createServer(dynamodb, dynClient);
const port = 8001;
const server = app.listen(port);
server.on('listening', () => {
const address = server.address();
console.log(` listening on http://0.0.0.0:${address.port}`);
});
AWS Amplify is a development framework that deals with a lot of the common problems when building a mobile or web application, setting up the infrastructure required for you. Each part of the framework deserves a blog post of its own, but here we are going to be looking at mocking the DynamoDB tables Amplify creates based on your GraphQL API definition.
If you’d like to find out how to use Amplify to create a GraphQL API the documentation is here
Amplify let’s you mock services used by your app with the Amplify CLI tool by running
$ amplify mock <service>
If you have used Amplify to create a GraphQL API to serve as the backend for your project you can run
$ amplify mock api
This will do two things:
We can now use dynamodb-admin to take a peek inside the tables Amplify has created, based on our API’s requirements by running
$ AWS_REGION=us-fake-1 AWS_ACCESS_KEY_ID=fake AWS_SECRET_ACCESS_KEY=fake DYNAMO_ENDPOINT=http://localhost:62224 dynamodb-admin
I’ve personally found this really useful to test locally, before committing to pushing my API changes.
Originally published by https://medium.com/swlh
#dynamodb #aws #code #dynamodb-admin #dynamodb-local