Jake Whittaker

Jake Whittaker


Deploy Docker Containers With AWS CodePipeline

Docker containers may be deployed using one of the several cloud platforms, Amazon Elastic Container Service (ECS) being. AWS CodePipeline is a DevOps service for Continuous Integration, Continuous Delivery and Continuous Deployment of applications hosted on various AWS platforms.

Amazon Elastic Container Service (ECS) is an AWS managed service for containerized applications for Docker containers.

Amazon Fargate is a serverless launch type for Amazon Elastic Container Service (ECS).

For the example application deployed to ECS an AWS CodePipelineconsists of a source code repository such as a GitHub repo, AWS CodeBuild for Build and a AWS ECS (Fargate) service for Staging.

The benefit of using an AWS CodePipeline for an AWS ECS service is that the ECS service continues to run while a new Docker image is built and deployed.


A Docker container deployment may need to be updated or modified due to changes in the Docker image source code or code build. Any modifications in the source code for a Docker image would require that a Docker image be rebuilt and the Docker service be redeployed. Without a mechanism to integrate, build, deliver and deploy source code while an AWS ECS deployment is running, it would involve stopping the ECS service tasks and as a result, incurring downtime for an ECS service. High availability of an ECS service being a priority, stopping tasks to redeploy an application is not a suitable option.


AWS CodePipeline is a DevOps service for Continuous Integration, Continuous Delivery and Continuous Deployment of applications hosted on the various AWS platforms, including Amazon ECS and Fargate as a deployment platform. An ECS service may be updated or modified without stopping the ECS service tasks. AWS CodePipeline provides high availability of an ECS service in a dynamic environment in which source code changes for a Docker image are common. A CodePipeline consists of three phases: Source code integration, Source code build, and Deployment, as shown in Figure 1.

Deploy Docker Containers With AWS CodePipeline

Figure 1. CodePipeline Phases

For source code we shall use a Github repository. For source code build we shall use a AWS CodeBuild project. For deployment we shall use an ECS service of launch type Fargate.

Creating and deploying a CodePipeline application to ECS Fargate involves the following procedure:

  1. Create an ECS Fargate Task Definition an Service
  2. Configure connectivity on Task Subnet/s
  3. Create or Configure a S3 Bucket for Output Artifacts from the CodePipeline Build Stage
  4. Create a CodePipeline to deploy a Docker platform application (image) on ECS Fargate
  5. Modify Input/Output Settings for Stage Artifacts
  6. Run the CodePipeline
  7. Make source code modifications to re-run CodePipeline

Setting the Environment

The only prerequisite is an AWS account. The application deployed by a CodePipeline on ECS Fargate is a Docker application. Any Docker image that has source code repo could be used and we have used Docker image dvohra/node-server. Here is the GitHub source code repository for the Docker image dvohra/node-server.

Creating a GitHub Code Repository

If a new GitHub source code repository were to be used, it must include a Dockerfile from which to build Docker image. The Dockerfile for the dvohra/node-server image is based on the Docker image node:4.6. Dockerfile instructions copy a server.js file, which is used to create a Node server, to the current directory, expose port 8080 for the Node server to listen on, and run a node command on the server.js script. The server.js file creates a Node server and handles an HTTP request/response.

Adding a Build Spec for CodeBuild Project

A build spec is a YAML syntax file with build commands and settings used by a CodeBuild project to run a build. The build spec file must be called “buildspec.yml” and must be copied to the root of the source code repository. A buildspec.yml file consists of key/value pairs to describe the various phases of a build. The build phases are represented with the phases sequence, which is a required mapping in a buildspec.yml. The version is the other required mapping in a buildspec.yml. The buildspec.yml file is listed on the GitHub repo.

Adding an Image Definitions File

For deploying Container based applications such as those deployed to ECS, AWS CodePipeline requires an Image definitions file in JSON format. The Image definitions file is called imagedefinitions.json by default but could be given another name. The image definitions file describes the container application and consists of two attributes: name and imageURI. The name specifies the Docker container name and the container must be running prior to running the CodePipeline. The imageURI specifies the Docker image to be run in the Docker container. The Docker image would typically be the same as the Docker image already running in an ECS container. The image could be different and the variation would typically be of the image tag.  The imagedefinitions.json used for the Node server application deployed to ECS Fargate is listed on the GitHub.

Creating a Task Definition

A task definition in an ECS application describes the container/s in the ECS deployment. In this section, we shall create task definition for a Node Server container to be deployed on ECS Fargate. Open this URL and log in if not already logged in.. Click on Get started to access the ECS Console. Click on Task Definitions in the navigation margin. In the Task Definitions click on Create new Task Definition. In the Create new Task Definition select launch type compatibility as Fargate. Click on Next step. Next, configure task and container definitions. In the Add container dialog specify a Container name (node-server) and specify Image as dvohra/node-server. The task definition is shown in Figure 2.

Deploy Docker Containers With AWS CodePipeline

Figure 2. Task Definition

Configuring Connectivity in Task Subnets

Before creating a service we need to configure connectivity to the Internet in the Subnets to be used when configuring the service. The Route Table lists the routes. We need to add a route with a default Internet gateway. Add a route with Destination as, which is an IP address. Select Target as an internet gateway.

Creating and Testing the Container Service

Next, create an ECS container service in the default cluster as shown in Figure 3.

Deploy Docker Containers With AWS CodePipeline

Figure 3. Cluster with 1 Service

With Fargate launch type an Elastic Network Interface (ENI) is provisioned for each task. Copy the Public IP of the task, which is the same as the Public IP of the Elastic Network Interface, from either the task Details page Network section or the ENI Console. Open the URL :8080 in a browser to invoke the Node Service application. The Node Server returns a message as shown in Figure 4.

AWS CodePipeline

Figure 4. Node Server Response

Creating or Configuring an S3 Bucket

The CodePipeline to build the Source code for the Node server application and deploy a Docker image to an ECS service requires that the CodeBuild project generate “Output Artifacts”. The Output artifacts are stored in an S3 bucket, which is selected when creating a CodePipeline. Create a new S3 Bucket in the S3 Console, or alternatively select a S3 bucket that may be created from an earlier run of CodePipeline.

Creating a CodeBuild Project

Next, create a CodeBuild project that is to be used to build the source code into a Docker image. The source code for a Docker image dvohra/node-server and the buildspec.yml file used to build the source code into the Docker image was discussed earlier. After a Docker image is built, it is uploaded to Docker hub by CodeBuild. To create a CodeBuild project, open this URL in a web browser. Select Build projects and click on Create project as shown in Figure 5.

AWS CodePipeline

Figure 5. Build projects>Create project

The Create project wizard gets started. In Configure project specify a Project name (node-server). In Source>Source Provider select GitHub. For Repository select Use a repository in my account and Choose a repository as the _dvohra/docker-node-server _ repo. Keep the default setting for Git clone depth as 1.

Select the options for Webhook, Insecure SSL and Build Badge. Selecting Webhook makes the code get rebuilt every time a code update is made in the GitHub repo. The Insecure SSL option makes the code build SSL warnings when connecting to project source. The Build Badge option makes the project’s status visible and embeddable. In Environment: How to build select Environment image as Use an image managed by AWS CodeBuild. Select Operating System as Ubuntu. For Runtime, select Docker as shown in Figure 6.

AWS CodePipeline

Figure 6. Selecting Runtime as Docker

For Runtime version, select aws:codebuild/docker:17.09.0, which represents the Docker version 17.9. If a later version is available select the later version. The Privileged option gets selected automatically for Docker runtime as it is required to build a Docker image. For Build specification select Use the buildspec.yml in the source code root directory as shown in Figure 7. The Buildspec name is buildspec.yml by default.

Docker Containers

Figure 7. Configuring Environment: How to build

For Certificate select Do not install any certificate. A certificate is not required by CodePipeline, but for a more secure CodeBuild a self-signed certificate could be installed from S3. Next, configure the output artifacts in the Artifacts. Output artifacts are required by CodePipeline Select Type as Amazon S3. Specify a S3 bucket folder name to use. Set the Path as “/”, which creates the folder at the bucket root. Select Namespace type as Node. Select a Bucket name as the Bucket configured earlier as shown in Figure 8. Select Cache as No cache.

Docker Containers

Figure 8. Configuring S3 Bucket for Artifacts

In Service role select the option Create a service role in your account if the CodeBuild project is being created for the first time. If the CodeBuild project was created before, select Choose an existing service role from your account. Select the option Allow AWS CodeBuild to modify this service role so it can be used with this build project as shown in Figure 9. For VPC select No VPC. Click on Continue.

Docker Containers

Figure 9. Configuring Service Role and VPC settings

In Review, review the Source and Build environment. Scroll down and click on Create to create the CodeBuild project. A CodeBuild project gets created and listed in Build projects as shown in Figure 10.

aws codepipeline tutorial

Figure 10. CodeBuild Project

The service role created by default by CodeBuild does not include some of the required permissions. The service role needs to be modified by adding an inline policy that adds permissions s3:GetObject and s3:PutObject. The inline policy to add is listed:


Testing the CodeBuild Project

Test the CodeBuild project before creating and configuring the CodeBuild in a CodePipeline so that if any errors exist they may be fixed. Click on Start build to start the build as shown in Figure 11.

aws codepipeline tutorial

Figure 11. Start build

The Start new build wizard gets started. Click on Start build. The CodeBuild project gets started and the code starts to get built. When the CodeBuild project gets completed, the Phase details indicate the same as shown in Figure 12.

aws codepipeline github

Figure 12. Phase details and Build logs indicate that the CodeBuild has completed

The Docker image dvohra/node-server generated and updated by the CodeBuild project to Docker Hub is shown in Figure 13.

aws codepipeline github

Figure 13. Docker Image generated and uploaded by CodeBuild on Docker Hub

Creating a CodePipeline

Having created projects for each of the CodePipeline stages; GitHub code repository for Source, CodeBuild for Build and ECS Fargate service for Staging, next we shall create a CodePipeline. Open CodePipeline Console with URL and click on Get started as shown in Figure 14.

aws codepipeline github

Figure 14. CodePipeline>Get started

The Create pipeline wizard gets started as shown in Figure 15. First, specify a Pipeline name (node-server-fargate) and click on Next step.

docker containers list

Figure 15. Specifying Pipeline Name

Next, configure the Source location. For Source provider select GitHub as shown in Figure 16.

docker containers list

Figure 16. Selecting Source provider

Next, connect to the GitHub with Connect to GitHub as shown in Figure 17.

docker containers list

Figure 17. Connect to GitHub

Select the GitHub Repository as shown in Figure 18.

docker containers on aws

Figure 18. Selecting GitHub Repo

Select the repo Branch as shown in Figure 19. Click on Next step.

docker containers on aws

Figure 19. Source location>Next step

Next, configure the Build. Select Build provider as AWS CodeBuild as shown in Figure 20.

docker containers on aws

Figure 20. Selecting Build provider as AWS CodeBuild

The subsequent section displayed is based on the Build provider selected. For AWS CodeBuild, a section to provide details about the CodeBuild gets displayed. For Configure your project select Select an existing project as shown in Figure 21. Select Project name as the CodeBuild project node-server created earlier.

docker containers tutorial

Figure 21. Selecting CodeBuild Project

Click on Next step. Next, configure the Deploy stage of the CodePipeline as shown in Figure 22. Select Deployment provider as Amazon ECS.

docker containers tutorial

Figure 22. Selecting Deployment provider as Amazon ECS

The subsequent section displayed is based on the Deployment provider selected as indicated by the Amazon ECS section in Figure 23. In the Amazon ECS section select Cluster name as the cluster in which the ECS service to be deployed to is created; select default.

docker containers tutorial

Figure 23. Selecting ECS Cluster

Select the Service name as the ECS service to deploy to, which is node-server-service, as shown in Figure 24.

docker containers vs images

Figure 24. Selecting ECS Service

Specify the Image filename as imagedefinitions.json as shown in Figure 25. If omitted, the default Image filename is imagedefinitions.json and the file should be available in the Source code GitHub repo. Click on Next step.

docker containers vs images

Figure 25. Specifying Image filename

Next, select the Service Role name as shown in Figure 26. A new service role is created in IAM automatically the first time a CodePipeline is created. Subsequently, the Service role gets listed to be selected.

docker containers vs images

Figure 26. Selecting Service Role

Click on Next step. Review the CodePipeline and click on Create pipeline as shown in Figure 27.

docker containers vs images

Figure 27. Create pipeline

A CodePipeline gets created as shown in Figure 28. After getting created the CodePipeline starts to run automatically as shown by In Progress status for Source stage.

Deploy Docker Containers With AWS CodePipeline

Figure 28. CodePipeline Source stage in Progress

When the Source stage completes, its status becomes Succeeded as shown in Figure 29. And the Build stage starts to run as indicated by the In Progress status.

Deploy Docker Containers With AWS CodePipeline

Figure 29. Source stage completed and Build stage In Progress

The Build stage also gets completed as indicated by the Succeeded status in Figure 30. The Staging stage starts to run.

Deploy Docker Containers With AWS CodePipeline

Figure 30. Build stage succeeded and Staging stage In Progress

Modifying the Input/Output Artifacts

We had already configured all the CodePiepline stages; why do we need to modify the settings for Input/Output Artifacts? Because for the example ECS application deployed by CodePipeline, the default Input/Output artifacts settings are not as required to run the CodePipeline. By default, the input artifacts to the Staging stage are the output artifacts from the Build stage. For a CodeBuild that only builds a Docker image and uploads the Docker image to Docker Hub or Amazon ECR, no output artifacts are generated by the CodeBuild stage. The input artifacts to the Staging stage need to be set to the output artifacts from the Source stage. Without modifying the settings for the Input/Output artifacts, the Staging stage would fail. The CodePipeline that was started automatically after being created fails. To modify the Input/Output artifacts click on Edit as shown in Figure 31.

Deploy Docker With AWS CodePipeline

Figure 31. CodePipeline>Edit

After modifying the Input/Output artifacts, click on Save pipeline changes as shown in Figure 32.

Deploy Docker With AWS CodePipeline

Figure 32. Save pipeline changes

Running the CodePipeline

To run the CodePipeline after modification, click on Release change as shown in Figure 33.

Deploy Docker With AWS CodePipeline

Figure 33. Release change

In the Release change confirmation dialog, click on Release. The modifications made to the CodePipeline get applied and the CodePipeline starts to run from the start as indicated by the In Progress status for the Source stage in Figure 34. The status for the Build and Staging stages are from the previous run and does not indicate the current status of the stages.

Deploy Docker With AWS CodePipeline

Figure 34. Source stage In Progress

All stages of the CodePipeline get completed with status Succeeded as shown in Figure 35.

docker containers and kubernetes

Figure 35. All stages of CodePipeline completed successfully

The previous service task gets stopped and a task based on the revised task definition starts to run as indicated by the RUNNING task status in Figure 36.

docker containers and kubernetes

Figure 36. New task running

To invoke the new task, find its Public IP as before from the Elastic Network interface for the task. Open URL :8080 in a web browser to invoke the task. The Node server application response is shown in Figure 37.

docker containers and kubernetes

Figure 37. Node Server application response

Modifying Source Code to Re-Run CodePipeline

What is the advantage of running a CodePipeline, if the ECS service response is the same as when invoking the service directly without a CodePipeline? Typically the source code for an ECS deployed in production would need to be updated periodically, which implies that the Docker image would need to be rebuilt. The Docker image deployed in an ECS service task would also need to be updated. The Docker image needs to be updated without any discontinuation of the ECS based service. With a CodePipeline, a new run of the CodePipeline is started automatically every time source code modifications are made. Without any user intervention, the ECS deployment gets updated when source code changes are made.

To demonstrate, make a slight modification to source code in the GitHub repo, such as make the Hello message in server.js different. Click on Commit changes in the repo. The CodePipeline starts to re-run automatically as shown by the Source stage having completed successfully and the Build stage in progress in Figure 38.

docker containers azure

Figure 38. CodePipeline restarted automatically

After the Build stage completes successfully, the Staging starts to run as indicated by the status messages in Figure 39.

docker containers azure

Figure 39. Build stage completed and Staging stage started

The Staging also completes successfully. The new task gets started.  Using the Public IP of the new task, open the URL :8080 in a web browser. The new task gets invoked and the Node server response gets displayed as shown in Figure 40. The Node server response is the modified response from server.js.

docker containers azure

Figure 40. Modified Node Server Response from new task


In this article, we discussed deploying a Docker container application to ECS Fargate. We have demonstrated updating source code for a Docker image and deploying the new Docker image to a running container service without stopping the container service, all using an AWS CodePipeline.

#docker #aws #devops

What is GEEK

Buddha Community

Deploy Docker Containers With AWS CodePipeline
Christa  Stehr

Christa Stehr


50+ Useful Kubernetes Tools for 2020 - Part 2


Last year, we provided a list of Kubernetes tools that proved so popular we have decided to curate another list of some useful additions for working with the platform—among which are many tools that we personally use here at Caylent. Check out the original tools list here in case you missed it.

According to a recent survey done by Stackrox, the dominance Kubernetes enjoys in the market continues to be reinforced, with 86% of respondents using it for container orchestration.

(State of Kubernetes and Container Security, 2020)

And as you can see below, more and more companies are jumping into containerization for their apps. If you’re among them, here are some tools to aid you going forward as Kubernetes continues its rapid growth.

(State of Kubernetes and Container Security, 2020)

#blog #tools #amazon elastic kubernetes service #application security #aws kms #botkube #caylent #cli #container monitoring #container orchestration tools #container security #containers #continuous delivery #continuous deployment #continuous integration #contour #developers #development #developments #draft #eksctl #firewall #gcp #github #harbor #helm #helm charts #helm-2to3 #helm-aws-secret-plugin #helm-docs #helm-operator-get-started #helm-secrets #iam #json #k-rail #k3s #k3sup #k8s #keel.sh #keycloak #kiali #kiam #klum #knative #krew #ksniff #kube #kube-prod-runtime #kube-ps1 #kube-scan #kube-state-metrics #kube2iam #kubeapps #kubebuilder #kubeconfig #kubectl #kubectl-aws-secrets #kubefwd #kubernetes #kubernetes command line tool #kubernetes configuration #kubernetes deployment #kubernetes in development #kubernetes in production #kubernetes ingress #kubernetes interfaces #kubernetes monitoring #kubernetes networking #kubernetes observability #kubernetes plugins #kubernetes secrets #kubernetes security #kubernetes security best practices #kubernetes security vendors #kubernetes service discovery #kubernetic #kubesec #kubeterminal #kubeval #kudo #kuma #microsoft azure key vault #mozilla sops #octant #octarine #open source #palo alto kubernetes security #permission-manager #pgp #rafay #rakess #rancher #rook #secrets operations #serverless function #service mesh #shell-operator #snyk #snyk container #sonobuoy #strongdm #tcpdump #tenkai #testing #tigera #tilt #vert.x #wireshark #yaml

Lindsey  Koepp

Lindsey Koepp


AWS Bottlerocket vs. Google Container-Optimized OS: Which Should You Use and When

What’s the difference between popular Container-Centric OS choices, Google’s Container-Optimized OS, and AWS’s Bottlerocket? The concepts underlying containers have been around for many years. Container technologies like Docker, Kubernetes, and an entire ecosystem of products, as well as best practices, have emerged in the last few years. This has enabled different kinds of applications to be containerized.

Web service providers like Amazon AWS and Google are giving a further boost to container innovation, for enterprises to adopt and use containers at scale. This will help them to reap the benefits containers bring, including increased portability and greater efficiency.

Linux-based OS, AWS Bottlerocket is a new option, designed for running containers on virtual machines (VMs) or bare-metal hosts. In this article, you will learn the core uses and differences between the two open-source OS.

**AWS Bottlerocket **

It is an open-source, stripped-down Linux distribution that’s similar to projects like Google’s Container-Optimized OS. This single-step update process helps reduce management overhead.

_It makes OS updates easy to automate using container orchestration services such as Amazon Elastic Container Service (ECS) and Amazon Elastic Kubernetes Service (EKS). _

**Google Container-Optimized OS **

It’s an OS image for Google Compute Engine VMs that’s optimized for running Docker containers. It allows you to bring up your Docker containers on Google Cloud Platform securely, and quickly. It is based on the open-source Chromium OS project and is maintained by Google.

But before diving into the core differences, let us give you a basic overview of containers, VMs, and container-optimized OS, and its underlying challenges to better understand the differences.

If you are already aware of all the underlying processes of containers, then you can skip to the main differences for AWS Bottlerocket vs Google Container-Optimized OS.

#containers #amazon-aws #google-cloud #container-optimized-os #aws-containers #docker-containers #linux-based-os #orchestration

Adnan Malik


Serverless Containers With AWS Fargate and Docker

In AWS there are a few services which seem similar or related when it comes to containerisation. They do have their nuances so let’s begin with an introduction of what these are and their similarities and differences (according to the AWS documentation)

Elastic Container Service (ECS)

  • Fully managed container orchestration service
  • Manages the lifecycle and placement of tasks comprised of one or more containers that work together
  • Does not run or execute your container(s)
  • Containers run using ECS Container Instances (normal EC2 instances with ECS Container Agent) or Fargate
  • ECS Container Agent regularly polls the ECS API to manage state
  • Cluster of container instances run in an auto-scaling group
  • You** manage your own EC2 instance, patching and scaling**
  • ECS is free but pricing is per running EC2 instance
  • Offers integration with Elastic Load Balancing (ELB)
  • Supports Application Load Balancer (ALB), the Network Load Balancer (NLB) and Classic Load Balancer (CLB)
  • Load balancing is efficient
  • Each task has its own Elastic Network Interface (ENI)
  • Supports IAM roles for tasks
  • AWS proprietary

#aws-fargate #aws-ecr #docker #aws-ecs #aws-eks

ICYMI: From Docker Straight to AWS Built-in

In July we announced a new strategic partnership with Amazon to integrate the Docker experience you already know and love with Amazon Elastic Container Service (ECS) with AWS Fargate. Over the last couple of months we have worked with the community on the beta experience in Docker Desktop Edge. Today we are excited to bring this experience to our entire community in Docker Desktop stable, version

You can watch Carmen Puccio (Amazon) and myself (Docker) and view the original demo in the recording of our latest webinar here.

What started off in the beta as a Docker plugin experience docker ecs has been pulled into Docker directly as a familiar docker compose flow. This is just the beginning, and we could use your input so head over to the Docker Roadmap and let us know what you want to see as part of this integration.

There is no better time to try it. Grab the latest Docker Desktop Stable. Then check out my example application which will walk you through everything you need to know to deploy a Python application locally in development and then again directly to Amazon ECS in minutes not hours.

#community #products #aws #containers #docker #docker on aws #ecs #fargate

Hunter  Krajcik

Hunter Krajcik


Deploying AWS Lambda with Docker Containers: I Gave it a Try and Here’s My Review

You can now build your serverless workloads in a Docker container with FaaS. I gave it a try and here’s my review.

Source: https://imgflip.com/i/4q3cq8

Among all the new features and services that AWS announced during the re:Invent 2020, my favorites were definitely the AWS Lambda updates. And there were many! For example, your code execution is no longer rounded up to the nearest 100ms of duration for billing — you are now billed per millisecond [1]. On top of that, AWS increased the Lambda’s memory capacity to 10 GB [2], and correspondingly the CPU capacity up to 6 vCPUs [3]. But today, I want to dig deeper into something even more exciting — from now on, AWS Lambda doesn’t require packaging your code and dependencies to a zip file. Instead, you can now do it with a Docker container image that can be up to 10 GB in size.

Personally, I consider this a game-changer for many serverless use cases. Here’s why.

#serverless #docker-image #containers #aws-lambda #aws