Rory  West

Rory West

1622717820

Setting up Airflow on AWS EKS

Introduction

Airflow is one of my favorite tools that I frequently use to setup and manage data science pipelines. The Airflow UI gives us a clear picture of the DAGS and its current status. I may be wrong here but from my experience, I have seen that Airflow on a single machine is not scalable. Thus, to scale Airflow, we can use Kubernetes.

When I was trying to deploy Airflow on AWS EKS, I had to go through multiple sources, get things clarified from the community. Thus, I am writing this article to make things as easy as possible to deploy airflow on AWS EKS.

The prerequisites for the article are to have aws-cli, kubectl, and helm installed, setup an EKS cluster in AWS. We will be using helm to deploy the airflow helm chart on AWS EKS.

I found Helm to be very useful in setting up and managing the Kubernetes application. Many helm charts are available in Artifacthub.io.

Airflow also has a helm chart that has very good community support. I have to thank the Airflow helm chart community that helped me out a lot! The helm chart is available at https://github.com/airflow-helm/charts/tree/main/charts/airflow.

#aws #aws-eks #airflow #kubernetes

What is GEEK

Buddha Community

Setting up Airflow on AWS EKS
Rory  West

Rory West

1622717820

Setting up Airflow on AWS EKS

Introduction

Airflow is one of my favorite tools that I frequently use to setup and manage data science pipelines. The Airflow UI gives us a clear picture of the DAGS and its current status. I may be wrong here but from my experience, I have seen that Airflow on a single machine is not scalable. Thus, to scale Airflow, we can use Kubernetes.

When I was trying to deploy Airflow on AWS EKS, I had to go through multiple sources, get things clarified from the community. Thus, I am writing this article to make things as easy as possible to deploy airflow on AWS EKS.

The prerequisites for the article are to have aws-cli, kubectl, and helm installed, setup an EKS cluster in AWS. We will be using helm to deploy the airflow helm chart on AWS EKS.

I found Helm to be very useful in setting up and managing the Kubernetes application. Many helm charts are available in Artifacthub.io.

Airflow also has a helm chart that has very good community support. I have to thank the Airflow helm chart community that helped me out a lot! The helm chart is available at https://github.com/airflow-helm/charts/tree/main/charts/airflow.

#aws #aws-eks #airflow #kubernetes

Seamus  Quitzon

Seamus Quitzon

1601341562

AWS Cost Allocation Tags and Cost Reduction

Bob had just arrived in the office for his first day of work as the newly hired chief technical officer when he was called into a conference room by the president, Martha, who immediately introduced him to the head of accounting, Amanda. They exchanged pleasantries, and then Martha got right down to business:

“Bob, we have several teams here developing software applications on Amazon and our bill is very high. We think it’s unnecessarily high, and we’d like you to look into it and bring it under control.”

Martha placed a screenshot of the Amazon Web Services (AWS) billing report on the table and pointed to it.

“This is a problem for us: We don’t know what we’re spending this money on, and we need to see more detail.”

Amanda chimed in, “Bob, look, we have financial dimensions that we use for reporting purposes, and I can provide you with some guidance regarding some information we’d really like to see such that the reports that are ultimately produced mirror these dimensions — if you can do this, it would really help us internally.”

“Bob, we can’t stress how important this is right now. These projects are becoming very expensive for our business,” Martha reiterated.

“How many projects do we have?” Bob inquired.

“We have four projects in total: two in the aviation division and two in the energy division. If it matters, the aviation division has 75 developers and the energy division has 25 developers,” the CEO responded.

Bob understood the problem and responded, “I’ll see what I can do and have some ideas. I might not be able to give you retrospective insight, but going forward, we should be able to get a better idea of what’s going on and start to bring the cost down.”

The meeting ended with Bob heading to find his desk. Cost allocation tags should help us, he thought to himself as he looked for someone who might know where his office is.

#aws #aws cloud #node js #cost optimization #aws cli #well architected framework #aws cost report #cost control #aws cost #aws tags

Rory  West

Rory West

1622634720

Configure AWS ElasticSearch Service with EKS Cluster

AWS provides Elasticsearch service along with Kibana which makes developer’s work easy and simple. This article will guide on how to provision an AWS ElasticSearch Service and connect with an EKS cluster for log monitoring.

Article content

  1. Provision the ES Cluster
  2. Configure Fluent-Bit in EKS
  3. Deploy Fluent-bit
  4. Access Kibana Dashboard

Introduction to Elasticsearch Service

Elasticsearch can be installed on-premise, on Amazon EC2 or AWS Elasticsearch service. When you install Elasticsearch on-premise or on Amazon EC2, you have to be responsible for installing, provision infrastructure and managing the cluster. But with AWS Elasticsearch amazon takes care of everything. It is a fully managed service.

Along with Elasticsearch, AWS provides Kibana. It is deployed automatically with your Elasticsearch Domain as a fully managed service, automatically taking care of all the heavy lifting to manage the cluster.

AWS provides two types of endpoints in ES setups. 1 — Internet endpoint; 2 — VPC endpoint. You can also have a custom endpoint if you want to configure with a domain you have. We’ll discuss more features and configurations while provisioning an ES Domain.

In this article, we’ll provision an internet endpoint ES Domain.

#aws-eks #aws-elasticsearch #aws #elasticsearch-service #eks cluster

Hire AWS Developer

Looking to Hire Professional AWS Developers?

The technology inventions have demanded all businesses to use and manage cloud-based computing services and Amazon is dominating the cloud computing services provider in the world.

Hire AWS Developer from HourlyDeveloper.io & Get the best amazon web services development. Take your business to excellence with our best AWS developer that will serve you the benefit of different cloud computing tools.

Consult with experts: https://bit.ly/2CWJgHyAWS Development services

#hire aws developer #aws developers #aws development company #aws development services #aws development #aws

Christa  Stehr

Christa Stehr

1598408880

How To Unite AWS KMS with Serverless Application Model (SAM)

The Basics

AWS KMS is a Key Management Service that let you create Cryptographic keys that you can use to encrypt and decrypt data and also other keys. You can read more about it here.

Important points about Keys

Please note that the customer master keys(CMK) generated can only be used to encrypt small amount of data like passwords, RSA key. You can use AWS KMS CMKs to generate, encrypt, and decrypt data keys. However, AWS KMS does not store, manage, or track your data keys, or perform cryptographic operations with data keys.

You must use and manage data keys outside of AWS KMS. KMS API uses AWS KMS CMK in the encryption operations and they cannot accept more than 4 KB (4096 bytes) of data. To encrypt application data, use the server-side encryption features of an AWS service, or a client-side encryption library, such as the AWS Encryption SDK or the Amazon S3 encryption client.

Scenario

We want to create signup and login forms for a website.

Passwords should be encrypted and stored in DynamoDB database.

What do we need?

  1. KMS key to encrypt and decrypt data
  2. DynamoDB table to store password.
  3. Lambda functions & APIs to process Login and Sign up forms.
  4. Sign up/ Login forms in HTML.

Lets Implement it as Serverless Application Model (SAM)!

Lets first create the Key that we will use to encrypt and decrypt password.

KmsKey:
    Type: AWS::KMS::Key
    Properties: 
      Description: CMK for encrypting and decrypting
      KeyPolicy:
        Version: '2012-10-17'
        Id: key-default-1
        Statement:
        - Sid: Enable IAM User Permissions
          Effect: Allow
          Principal:
            AWS: !Sub arn:aws:iam::${AWS::AccountId}:root
          Action: kms:*
          Resource: '*'
        - Sid: Allow administration of the key
          Effect: Allow
          Principal:
            AWS: !Sub arn:aws:iam::${AWS::AccountId}:user/${KeyAdmin}
          Action:
          - kms:Create*
          - kms:Describe*
          - kms:Enable*
          - kms:List*
          - kms:Put*
          - kms:Update*
          - kms:Revoke*
          - kms:Disable*
          - kms:Get*
          - kms:Delete*
          - kms:ScheduleKeyDeletion
          - kms:CancelKeyDeletion
          Resource: '*'
        - Sid: Allow use of the key
          Effect: Allow
          Principal:
            AWS: !Sub arn:aws:iam::${AWS::AccountId}:user/${KeyUser}
          Action:
          - kms:DescribeKey
          - kms:Encrypt
          - kms:Decrypt
          - kms:ReEncrypt*
          - kms:GenerateDataKey
          - kms:GenerateDataKeyWithoutPlaintext
          Resource: '*'

The important thing in above snippet is the KeyPolicy. KMS requires a Key Administrator and Key User. As a best practice your Key Administrator and Key User should be 2 separate user in your Organisation. We are allowing all permissions to the root users.

So if your key Administrator leaves the organisation, the root user will be able to delete this key. As you can see **KeyAdmin **can manage the key but not use it and KeyUser can only use the key. ${KeyAdmin} and **${KeyUser} **are parameters in the SAM template.

You would be asked to provide values for these parameters during SAM Deploy.

#aws #serverless #aws-sam #aws-key-management-service #aws-certification #aws-api-gateway #tutorial-for-beginners #aws-blogs