Michio JP

Michio JP


AWS Infrastructure to Ship CloudTrail logs from S3 to Kinesis

This repository contains a Terraform module to ship CloudTrail logs stored in a S3 bucket into a Kinesis stream for further processing and real-time analysis.

Use-cases include shipping CloudTrail logs of an organization trail to a SIEM (such as Splunk) or third-party service provider.


In summary:

A S3 bucket notification is configured to trigger a SNS event every time AWS delivery a new CloudTrail log file to the S3 bucket.

A subscription is set up so that a Lambda function is invoked for every SNS event, i.e. every time AWS delivers a new CloudTrail log file to the S3 bucket.

The Lambda function picks up the information that a new file was uploaded, reads this file from S3 and writes it to Kinesis.


A CloudTrail organization trail must already be configured to ship organization-wide CloudTrail logs to a S3 bucket

This S3 bucket ("CloudTrail S3 bucket") must already exist

The CloudTrail S3 bucket must be in the same AWS account and region as the one the module is instantiated



cloudtrail-bucket-nameName of the S3 bucket in which CloudTrail logs are stored (must exist and properly configured to receive CloudTrail logs prior to calling this module)stringn/ayes
cloudtrail-sns-topic-nameName of the SNS topic where information about newly shipped CloudTrail log files are sentstring"organization-trail-event-notification-topic"no
cloudwatch-logs-retention-time-daysRetention period for the CloudWatch logs of the Lambda function (in days)number7no
kinesis-num-shardsNumber of shards to use in the Kinesis streamnumber4no
kinesis-retention-time-daysRetention period of the Kinesis stream (in days)number7no
kinesis-stream-kms-key-idID of the KMS key to use for encrypting the Kinesis streamstring"alias/aws/kinesis"no
kinesis-stream-nameName of the Kinesis stream used for aggregationstring"cloudtrail-logs-stream"no
lambda-memoryMemory to allocate to the Lambda functionnumber512no


kinesis-stream-arnARN of the newly created Kinesis stream
kinesis-stream-nameName of the newly created Kinesis stream
sns-topic-arnarn of the newly creates SNS topic
sns-topic-nameName of the newly creates SNS topic

Example usage

Sample usage:

provider "aws" {
  region = "eu-west-1"

// Ship CloudTrail logs that AWS ships in "my-organization-trail-s3-bucket" into a Kinesis stream "cloudtrail-logs"
module "cloudtrail-to-kinesis" {
  source                      = "github.com/nexthink/terraform-aws-cloudtrail-s3-to-kinesis?ref=v1.0.0"
  cloudtrail-bucket-name      = "my-organization-trail-s3-bucket"
  kinesis-stream-name         = "cloudtrail-logs"
  kinesis-retention-time-days = 7



Logs of the Lambda function are sent to a CloudWatch log group. Invocation rate and success/error rate can be seen directly from the AWS console, under the "Monitoring" tab of the Lamdba function


The module creates a custom CloudWatch metric NumberOfCloudTrailRecordsShippedToKinesis, allowing to have visibility on the total number of CloudTrail events shipped by the Lambda function. When used conjointly to the default AWS Lambda CloudWatch metrics, it allows to easily build a CloudWatch dashboard to monitor the volume of logs processed by the Lambda function, as well as any potential errors.

CloudWatch dashboard for observability

Unit tests

Run the Lambda function unit tests using:

$ cd lambda
$ make test
AWS_SECRET_ACCESS_KEY= AWS_ACCESS_KEY_ID= python -m unittest discover tests -v
test_extract (test_extract.TestExtract) ... ok
test_entrypoint (test_lambda_entrypoint.TestLambdaEntryPoint) ... 2021-04-09 16:38:33,062 Reading CloudTrail log file s3://my-cloudtrail-bucket/dir/to/cloudtrail.json.gz
2021-04-09 16:38:33,066 Writing CloudTrail log payload to Kinesis (27596 bytes)
test_writes_to_kinesis (test_load.TestLoad) ... 2021-04-09 16:38:33,124 Found credentials in environment variables.
2021-04-09 16:38:33,155 Writing CloudTrail log payload to Kinesis (24 bytes)
test_returns_correct_data_from_s3 (test_transform.TestExtract) ... 2021-04-09 16:38:33,236 Found credentials in environment variables.
2021-04-09 16:38:33,300 Reading CloudTrail log file s3://my-bucket//path/to/cloudtrail-file.json.gz

Ran 4 tests in 0.532s


End-to-end tests

End-to-end tests use Terratest to spin up real infrastructure, create a S3 bucket, ship a sample CloudTrail log file to it and check that the expected record is produced in Kinesis.

# Pre-requisite: ensure you are authenticated against a test/CI AWS account
# $ aws-vault-exec ci-account

$ cd test/
$ go test

End-to-end tests take around 2-3 minutes to run.


See the examples/ folder for complete usage examples.

Download Details:

Author: nexthink
Download Link: Download The Source Code
Official Website: https://github.com/nexthink/terraform-aws-cloudtrail-s3-to-kinesis 
License: BSD-3-Clause License


What is GEEK

Buddha Community

AWS Infrastructure to Ship CloudTrail logs from S3 to Kinesis
Rory  West

Rory West


AWS Lambda with Kinesis Trigger: 6 Pitfalls and How to Fix Them

Kinesis Data Streams are the solution for real-time streaming and analytics at scale. As we learned last November, AWS themselves use it internally to keep, well, AWS working.

Kinesis works very well with AWS Lambda. Creating a function that will process incoming records is easy, especially if we leverage the Serverless Framework or SAM to deploy required resources.

The simplicity of setting up a Kinesis trigger for a Lambda function may be deceptive. There are pitfalls that can cause problems we will spot only later, in the production environment. I learned about some of them the hard way, so let’s say I speak from the experience.

Simple Kinesis trigger

Let’s start with a Kinesis stream and a Lambda function that handles incoming records. I will use the Serverless Framework here, but we can achieve the same with SAM or raw CloudFormation.

#amazon-kinesis #kinesis #aws-lambda #aws #aws-services

Lindsey  Koepp

Lindsey Koepp


The Benefits of Amazon S3 Explained Through a Comic

AWS S3 is one of the most fundamental services of AWS Cloud.

It’s basically your unlimited and safest cloud storage.

Read this comic style conversation between two guys and get to know why some of the biggest companies in the world are using Amazon S3 for their business and why you should use it too.

#aws-s3 #aws #cloud-computing #cloud-storage #data-storage #aws-services #aws-top-story #aws-certification

Seamus  Quitzon

Seamus Quitzon


AWS Cost Allocation Tags and Cost Reduction

Bob had just arrived in the office for his first day of work as the newly hired chief technical officer when he was called into a conference room by the president, Martha, who immediately introduced him to the head of accounting, Amanda. They exchanged pleasantries, and then Martha got right down to business:

“Bob, we have several teams here developing software applications on Amazon and our bill is very high. We think it’s unnecessarily high, and we’d like you to look into it and bring it under control.”

Martha placed a screenshot of the Amazon Web Services (AWS) billing report on the table and pointed to it.

“This is a problem for us: We don’t know what we’re spending this money on, and we need to see more detail.”

Amanda chimed in, “Bob, look, we have financial dimensions that we use for reporting purposes, and I can provide you with some guidance regarding some information we’d really like to see such that the reports that are ultimately produced mirror these dimensions — if you can do this, it would really help us internally.”

“Bob, we can’t stress how important this is right now. These projects are becoming very expensive for our business,” Martha reiterated.

“How many projects do we have?” Bob inquired.

“We have four projects in total: two in the aviation division and two in the energy division. If it matters, the aviation division has 75 developers and the energy division has 25 developers,” the CEO responded.

Bob understood the problem and responded, “I’ll see what I can do and have some ideas. I might not be able to give you retrospective insight, but going forward, we should be able to get a better idea of what’s going on and start to bring the cost down.”

The meeting ended with Bob heading to find his desk. Cost allocation tags should help us, he thought to himself as he looked for someone who might know where his office is.

#aws #aws cloud #node js #cost optimization #aws cli #well architected framework #aws cost report #cost control #aws cost #aws tags

Hire AWS Developer

Looking to Hire Professional AWS Developers?

The technology inventions have demanded all businesses to use and manage cloud-based computing services and Amazon is dominating the cloud computing services provider in the world.

Hire AWS Developer from HourlyDeveloper.io & Get the best amazon web services development. Take your business to excellence with our best AWS developer that will serve you the benefit of different cloud computing tools.

Consult with experts: https://bit.ly/2CWJgHyAWS Development services

#hire aws developer #aws developers #aws development company #aws development services #aws development #aws

Christa  Stehr

Christa Stehr


How To Unite AWS KMS with Serverless Application Model (SAM)

The Basics

AWS KMS is a Key Management Service that let you create Cryptographic keys that you can use to encrypt and decrypt data and also other keys. You can read more about it here.

Important points about Keys

Please note that the customer master keys(CMK) generated can only be used to encrypt small amount of data like passwords, RSA key. You can use AWS KMS CMKs to generate, encrypt, and decrypt data keys. However, AWS KMS does not store, manage, or track your data keys, or perform cryptographic operations with data keys.

You must use and manage data keys outside of AWS KMS. KMS API uses AWS KMS CMK in the encryption operations and they cannot accept more than 4 KB (4096 bytes) of data. To encrypt application data, use the server-side encryption features of an AWS service, or a client-side encryption library, such as the AWS Encryption SDK or the Amazon S3 encryption client.


We want to create signup and login forms for a website.

Passwords should be encrypted and stored in DynamoDB database.

What do we need?

  1. KMS key to encrypt and decrypt data
  2. DynamoDB table to store password.
  3. Lambda functions & APIs to process Login and Sign up forms.
  4. Sign up/ Login forms in HTML.

Lets Implement it as Serverless Application Model (SAM)!

Lets first create the Key that we will use to encrypt and decrypt password.

    Type: AWS::KMS::Key
      Description: CMK for encrypting and decrypting
        Version: '2012-10-17'
        Id: key-default-1
        - Sid: Enable IAM User Permissions
          Effect: Allow
            AWS: !Sub arn:aws:iam::${AWS::AccountId}:root
          Action: kms:*
          Resource: '*'
        - Sid: Allow administration of the key
          Effect: Allow
            AWS: !Sub arn:aws:iam::${AWS::AccountId}:user/${KeyAdmin}
          - kms:Create*
          - kms:Describe*
          - kms:Enable*
          - kms:List*
          - kms:Put*
          - kms:Update*
          - kms:Revoke*
          - kms:Disable*
          - kms:Get*
          - kms:Delete*
          - kms:ScheduleKeyDeletion
          - kms:CancelKeyDeletion
          Resource: '*'
        - Sid: Allow use of the key
          Effect: Allow
            AWS: !Sub arn:aws:iam::${AWS::AccountId}:user/${KeyUser}
          - kms:DescribeKey
          - kms:Encrypt
          - kms:Decrypt
          - kms:ReEncrypt*
          - kms:GenerateDataKey
          - kms:GenerateDataKeyWithoutPlaintext
          Resource: '*'

The important thing in above snippet is the KeyPolicy. KMS requires a Key Administrator and Key User. As a best practice your Key Administrator and Key User should be 2 separate user in your Organisation. We are allowing all permissions to the root users.

So if your key Administrator leaves the organisation, the root user will be able to delete this key. As you can see **KeyAdmin **can manage the key but not use it and KeyUser can only use the key. ${KeyAdmin} and **${KeyUser} **are parameters in the SAM template.

You would be asked to provide values for these parameters during SAM Deploy.

#aws #serverless #aws-sam #aws-key-management-service #aws-certification #aws-api-gateway #tutorial-for-beginners #aws-blogs