Setup Serverless CI/CD and Pipelines

Setup Serverless CI/CD and Pipelines

I will share my experience on how i was able to setup Serverless CI/CD using powerful CI tool Teamcity, AWS Fargate, AWS S3 and Amazon ECR.

When we think of introducing or implementing CI/CD pipelines for our releases or deployments, One should always think of setting it in such a way that it shouldn’t introduce extra cost, the solution is scalable, future proof and secure by all means.

I will share my experience on how i was able to setup Serverless CI/CD using powerful CI tool Teamcity, AWS Fargate ,AWS S3 and Amazon ECR.

Although these days all cloud providers have their own serverless CI-CD like CodePipeline, Azure DevOps etc, but this is my implementation which can be used for any application that you want.

What Would You Need ?

  1. AWS Account
  2. EFS

Let’s Dive Right In

let’s first create EFS where we would be storing our configuration and data files for our TeamCity server, these are the files which you want to persists on the server, which means that even if our containers restart, we have the data persistence and for that we are using EFS.

This is important for the stateful application like TeamCity, Also we would schedule it to run only from MON-FRI to save costs.

You can do the same for the application which you don’t use often over the weekend or application which have less traffic, you can scale down the container to meet the demand without worrying about loosing data together with some cost saving.

Creating EFS

Head over to Services->EFS

Select the VPC in which you plan to have your container/Application running and follow the steps as below.

Image for post

Image for post

Image for post

Image for post

Image for post

Creating ECS Cluster

Once your EFS is ready you can start creating Cluster for your application to run, in our case TeamCity server. Please follow the below steps to create your own cluster.

An Amazon ECS cluster is a logical grouping of tasks or services. If you are running tasks or services that use the EC2 launch type, a cluster is also a grouping of container instances which we are using in this case.

Image for post

Image for post

Here you specify your VPC and Subnets, remember the cluster has to be created in the same VPC as your EFS otherwise we had to create peering connection and few other steps, which is not part of this exercise.

Image for post

Below you can select Container Insights to get metrics such as CPU and Memory available, this feature is recently added previously you had to add metrics collectors to your containers to get those metrics.

Image for post

Image for post

Once the Cluster is ready, we need to change the launch configuration to update our userdata section to create few directories and install efs client.

Follow the below steps for changing the Launch Configuration.

Image for post

Image for post

Image for post

User Data would be something like this.

#!/bin/bash
echo ECS_CLUSTER=TeamCity-Server >> /etc/ecs/ecs.config;echo ECS_BACKEND_HOST= >> /etc/ecs/ecs.config;
yum install -y amazon-efs-utils
mkdir /opt/teamcity
mkdir -p /data/teamcity_server/datadir

docker devops cloud aws

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

Multi-cloud Spending: 8 Tips To Lower Cost

Mismanagement of multi-cloud expense costs an arm and leg to business and its management has become a major pain point. Here we break down some crucial tips to take some of the management challenges off your plate and help you optimize your cloud spend.

How to Extend your DevOps Strategy For Success in the Cloud?

DevOps and Cloud computing are joined at the hip, now that fact is well appreciated by the organizations that engaged in SaaS cloud and developed applications in the Cloud. During the COVID crisis period, most of the organizations have started using cloud computing services and implementing a cloud-first strategy to establish their remote operations. Similarly, the extended DevOps strategy will make the development process more agile with automated test cases.

Top 10 AWS Cloud Migration Tools and Services

Take a look at some of the top cloud migration services you can use to migrate your services from on-premises to AWS Cloud. You need to understand the general terms of this field.

Ever Wondered Why We Use Containers In DevOps?

At some point we've all said the words, "But it works on my machine." It usually happens during testing or when you're trying to get a new project set up. Sometimes it happens when you pull down changes from an updated branch.

What are the benefits of cloud migration? Reasons you should migrate

To move or not to move? Benefits are multifold when you are migrating to the cloud. Get the correct information to make your decision, with our cloud engineering expertise.