田辺  亮介

田辺 亮介

1660982100

如何創建和管理 AWS MSK 集群和配置

Apache Kafka 允許在分佈式生態系統中進行異步通信。它允許生產者發布有關主題的消息,然後由對這些主題感興趣的消費者攝取。作為一個概念,pub-sub 模型已經存在了很長時間。然而,Kafka 的美妙之處在於——使用分區和消費者組,Kafka 可以以最小的開發和經濟開銷來擴展消息的消費率。在本教程中,我將帶您了解如何使用 AWS Managed Stream for Kafka (MSK) 服務預置託管 Kafka 集群。我們將使用無服務器框架為 MSK 和支持的 VPC、子網等創建和維護基礎設施。

本教程假設您對 Kafka 以及如何配置它有很好的理解。我們將介紹部署和更新 Apache Kafka 服務(集群和配置)。

先決條件

  • 阿帕奇卡夫卡
  • 無服務器框架的基礎知識
  • 活躍的 AWS 賬戶

MSK(Kafka 的託管流)

Amazon 提供名為 MSK(Kafka 託管流)的託管 Apache Kafka 服務。MSK 允許您構建可擴展的生產就緒 Kafka 應用程序。它為集群的擴展和管理抽象了基礎設施的配置。Kafka 集群附加了一個 MSK 配置。

在本教程中,我們將使用無服務器框架進行以下操作:

  • 創建 MSK 集群
  • 創建 MSK 集群配置
  • 將配置修訂附加到集群
  • 將其部署到 AWS 賬戶並進行驗證

1. 從無服務器項目開始

請克隆啟動項目。這提供了對預置 VPC、子網、IGW、路由表等的開箱即用支持。我強烈建議您瀏覽 resources/config 目錄以熟悉如何設置。

克隆存儲庫,並使用 npm 安裝依賴項。

git clone git@github.com:wednesday-solutions/msk-kafka-starter.git
cd msk-starter-template
npm i

💡 專業提示:需要仔細設置 MSK 的子網。如果您位於美國西部(加利福尼亞北部)區域,則需要兩個不同可用區 (AZ) 中的兩個子網。在所有其他區域中,您可以指定兩個或三個子網。您的子網必須都位於不同的 AZ。創建集群時,MSK 會平均分配代理節點。

在本教程中,我們在ap-south-1區域中創建了三個私有子網,每個子網中有 256 個 IP。我還特別注意確保子網位於不同的可用區。我們還有一個公有子網,可將流量路由到 Internet 網關 (IGW),該網關允許 VPC 中的組件與 Internet 通信。

2. 創建 MSK 配置

讓我們從創建 Kafka 配置開始。我們在server.properties文件中創建一個配置。

vi assets/server.properties

將以下內容添加到文件中。

auto.create.topics.enable = true #1
zookeeper.connection.timeout.ms = 1000 #2
log.roll.ms = 604800000 #3

以上表示具有以下設置的簡約配置:

  1. 在服務器上啟用主題的自動創建
  2. 客戶端等待與 zookeeper 建立連接的最長時間
  3. 即使段文件未滿,Kafka 也會強制日誌滾動的時間段

是可以配置的屬性的完整列表。

現在讓我們編寫用於創建集群配置的無服務器資源。

vi resources/apache-kafka/msk-config.ymlResources:
  ServerlessMSKConfiguration:
    Type: AWS::MSK::Configuration
    Properties:
      Description: cluster for msk cluster-${sls:stage}
      Name: msk-cluster-config-${sls:stage}-config
      ServerProperties: ${file('./assets/server.properties')} #1

ServerProperties引用新創建的server.properties文件

而已。這就是創建配置修訂所需的全部內容。配置中的後續更改將創建一個新版本。必須在 MSK 中引用更新的修訂號。

3.創建Kafka集群

讓我們創建 Kafka 集群,它將引用集群配置。打開serverless.yml文件並添加以下內容。

#serverless.yml
resources:
  ...
 - ${file('./resources/apache-kafka/msk.yml')}

接下來,運行以下命令。

vi resources/apache-kafka/msk.yml

將以下部分複製到文件中。

Resources:
  ServerlessMSK:
    Type: AWS::MSK::Cluster
    Properties:
      ClusterName: ${self:service}-${self:provider.stage}-msk #1
      KafkaVersion: 2.6.2 #2
      BrokerNodeGroupInfo:
        InstanceType: kafka.t3.small #3
        ClientSubnets: #4
          - !Ref ServerlessPrivateSubnet1 
          - !Ref ServerlessPrivateSubnet2 
          - !Ref ServerlessPrivateSubnet3
        SecurityGroups: #5
          - !GetAtt ServerlessMSKSecurityGroup.GroupId
        StorageInfo: #6
          EBSStorageInfo:
            VolumeSize: 10
      NumberOfBrokerNodes: 3 #7
      EncryptionInfo:
        EncryptionInTransit:
          ClientBroker: TLS
          InCluster: true
      EnhancedMonitoring: PER_TOPIC_PER_BROKER
      ConfigurationInfo: #8
        Arn: !GetAtt ServerlessMSKConfiguration.Arn
        Revision: 1
  1. 集群名稱以階段為後綴,允許跨環境輕鬆部署
  2. 我們使用的 Kafka 版本是 2.6.2
  3. 在本教程中,我們使用了 kafka.t3.small 實例,請對其進行配置以最適合您的應用程序需求
  4. 之前創建的子網與集群關聯
  5. 之前創建的安全組與集群關聯
  6. 您可以調整使用此參數配置的存儲
  7. 代理節點的數量是高可用性和冗餘的關鍵因素,可以使用NumberoOfBrokerNodes屬性進行調整
  8. ConfigurationInfo 屬性管理默認的 Kafka 配置。我們引用之前創建的配置和相關的修訂號

部署

使用以下命令部署您的應用程序

npm run deploy:development

在運行腳本之前,請確保以下環境變量設置為正確的值

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY

或者,您可以分叉存儲庫,將上述設置為 Github 機密,將其推送到您的主分支,並讓已經設置的 CD 管道處理其餘部分。一旦您的管道成功運行,請在控制台中檢查您新創建的集群!

集群配置屬性

在 AWS 控制台中部署 MSK 集群

然後去哪兒?

如果您在任何時候覺得卡住了,可以在這裡找到一個完整的 CI/CD 示例。

我希望你喜歡閱讀和完成本教程,就像我在編寫它時所做的一樣。

週三解決方案的出色人員使用無服務器框架創建了一個示例應用程序存儲庫。我很想听聽您對它的看法,也許還有關於它如何簡化/改進您的開發工作流程的故事。

鏈接:https ://medium.com/faun/creating-and-managing-an-aws-msk-cluster-and-configuration-2e5be0d6993f

#aws #kappa

What is GEEK

Buddha Community

如何創建和管理 AWS MSK 集群和配置
Seamus  Quitzon

Seamus Quitzon

1601341562

AWS Cost Allocation Tags and Cost Reduction

Bob had just arrived in the office for his first day of work as the newly hired chief technical officer when he was called into a conference room by the president, Martha, who immediately introduced him to the head of accounting, Amanda. They exchanged pleasantries, and then Martha got right down to business:

“Bob, we have several teams here developing software applications on Amazon and our bill is very high. We think it’s unnecessarily high, and we’d like you to look into it and bring it under control.”

Martha placed a screenshot of the Amazon Web Services (AWS) billing report on the table and pointed to it.

“This is a problem for us: We don’t know what we’re spending this money on, and we need to see more detail.”

Amanda chimed in, “Bob, look, we have financial dimensions that we use for reporting purposes, and I can provide you with some guidance regarding some information we’d really like to see such that the reports that are ultimately produced mirror these dimensions — if you can do this, it would really help us internally.”

“Bob, we can’t stress how important this is right now. These projects are becoming very expensive for our business,” Martha reiterated.

“How many projects do we have?” Bob inquired.

“We have four projects in total: two in the aviation division and two in the energy division. If it matters, the aviation division has 75 developers and the energy division has 25 developers,” the CEO responded.

Bob understood the problem and responded, “I’ll see what I can do and have some ideas. I might not be able to give you retrospective insight, but going forward, we should be able to get a better idea of what’s going on and start to bring the cost down.”

The meeting ended with Bob heading to find his desk. Cost allocation tags should help us, he thought to himself as he looked for someone who might know where his office is.

#aws #aws cloud #node js #cost optimization #aws cli #well architected framework #aws cost report #cost control #aws cost #aws tags

Hire AWS Developer

Looking to Hire Professional AWS Developers?

The technology inventions have demanded all businesses to use and manage cloud-based computing services and Amazon is dominating the cloud computing services provider in the world.

Hire AWS Developer from HourlyDeveloper.io & Get the best amazon web services development. Take your business to excellence with our best AWS developer that will serve you the benefit of different cloud computing tools.

Consult with experts: https://bit.ly/2CWJgHyAWS Development services

#hire aws developer #aws developers #aws development company #aws development services #aws development #aws

Christa  Stehr

Christa Stehr

1598408880

How To Unite AWS KMS with Serverless Application Model (SAM)

The Basics

AWS KMS is a Key Management Service that let you create Cryptographic keys that you can use to encrypt and decrypt data and also other keys. You can read more about it here.

Important points about Keys

Please note that the customer master keys(CMK) generated can only be used to encrypt small amount of data like passwords, RSA key. You can use AWS KMS CMKs to generate, encrypt, and decrypt data keys. However, AWS KMS does not store, manage, or track your data keys, or perform cryptographic operations with data keys.

You must use and manage data keys outside of AWS KMS. KMS API uses AWS KMS CMK in the encryption operations and they cannot accept more than 4 KB (4096 bytes) of data. To encrypt application data, use the server-side encryption features of an AWS service, or a client-side encryption library, such as the AWS Encryption SDK or the Amazon S3 encryption client.

Scenario

We want to create signup and login forms for a website.

Passwords should be encrypted and stored in DynamoDB database.

What do we need?

  1. KMS key to encrypt and decrypt data
  2. DynamoDB table to store password.
  3. Lambda functions & APIs to process Login and Sign up forms.
  4. Sign up/ Login forms in HTML.

Lets Implement it as Serverless Application Model (SAM)!

Lets first create the Key that we will use to encrypt and decrypt password.

KmsKey:
    Type: AWS::KMS::Key
    Properties: 
      Description: CMK for encrypting and decrypting
      KeyPolicy:
        Version: '2012-10-17'
        Id: key-default-1
        Statement:
        - Sid: Enable IAM User Permissions
          Effect: Allow
          Principal:
            AWS: !Sub arn:aws:iam::${AWS::AccountId}:root
          Action: kms:*
          Resource: '*'
        - Sid: Allow administration of the key
          Effect: Allow
          Principal:
            AWS: !Sub arn:aws:iam::${AWS::AccountId}:user/${KeyAdmin}
          Action:
          - kms:Create*
          - kms:Describe*
          - kms:Enable*
          - kms:List*
          - kms:Put*
          - kms:Update*
          - kms:Revoke*
          - kms:Disable*
          - kms:Get*
          - kms:Delete*
          - kms:ScheduleKeyDeletion
          - kms:CancelKeyDeletion
          Resource: '*'
        - Sid: Allow use of the key
          Effect: Allow
          Principal:
            AWS: !Sub arn:aws:iam::${AWS::AccountId}:user/${KeyUser}
          Action:
          - kms:DescribeKey
          - kms:Encrypt
          - kms:Decrypt
          - kms:ReEncrypt*
          - kms:GenerateDataKey
          - kms:GenerateDataKeyWithoutPlaintext
          Resource: '*'

The important thing in above snippet is the KeyPolicy. KMS requires a Key Administrator and Key User. As a best practice your Key Administrator and Key User should be 2 separate user in your Organisation. We are allowing all permissions to the root users.

So if your key Administrator leaves the organisation, the root user will be able to delete this key. As you can see **KeyAdmin **can manage the key but not use it and KeyUser can only use the key. ${KeyAdmin} and **${KeyUser} **are parameters in the SAM template.

You would be asked to provide values for these parameters during SAM Deploy.

#aws #serverless #aws-sam #aws-key-management-service #aws-certification #aws-api-gateway #tutorial-for-beginners #aws-blogs

Ananya Gupta

Ananya Gupta

1605514048

How AWS Skills Can Boost Your Career in The IT Industry?

For those that wish to create a promising career within the IT industry, pursuing AWS training is often the simplest option where you’ll develop and validate your cloud skills and learn the simplest of cloud computing technology. Amazon’s cloud platform i.e. Amazon Web Services is one of the highly preferred cloud computing services that provide easy and innovative cloud computing solutions.

The fast facts about the AWS Certification Course reveal that the potential expected marketplace for AWS Solution Architects will grow by $ 307.7 million by the year 2025. Moreover, the demand for AWS certified professionals has grown by 76% within the last two years. consistent with online employment portals like indeed.com, in developed countries just like the US, the entire vacant job profiles for AWS certified IT professionals is around 9728. Does one still need more facts that reflect the importance of AWS training program for a booming career within the IT sector?

Career Opportunities Offered by AWS

Learning the AWS certified training courses can open up engaging career prospects for you during a sort of cloud computing services. Amazon Web Services offers you a chance to find out from 70 diverse courses that affect memory, Networking, Analytics, Management, Database, Internet of Things, Developer tools, and Application services. Around 380000 cloud computing jobs are still vacant in search of qualified and trained AWS professionals.

Thus, AWS training can assist you to discover the brightest job prospects within the IT sector. The greater specialization in adopting technologically focused processes with the help of cloud-based services proves to be a big reason why pursuing AWS training is often the foremost optimum career decision for IT professionals.

AWS Training Can Fulfill Your Major Career Goals

The AWS certified courses and training can assist you to achieve the specified excellence and professionalism in your career. Being trained in AWS can allow you to experience multiple benefits within the sort of a pay hike and grab extra attention from the employer as they appear for professionals who possess the foremost advanced and updated knowledge within the field of cloud computing. The AWS training empowers you by imparting knowledge about the various fields of cloud computing through a comprehensive practice-based approach.

Additionally, to the present, it’s been observed that your chances of employment rise manifold once you complete the training and authorized courses from recognized AWS Training Program in Noida. Thus, you’ll apply for a spread of job profiles that cloud computing offers. Further, the marvelous outcomes are possible only you spend significant money and time for earning the AWS certification which may end up being a life-changing opportunity for you.Learn here more Aws Certification Types: Choose The Right For You

A final upshot

Pursuing AWS certification courses and training provides the simplest career opportunities for those that want to form a successful career within the IT sector. You’ll move ahead for splendid and memorable career growth within the AWS cloud-based services. Further, this will be fruitful for the enterprises and business owners also, because it can help within the effective storage of knowledge through cloud-based services which may help within the efficient running of the complex business process.

Moreover, cloud computing solutions are the necessity of the hour as they’re loaded with features that provide reduced cost and enhanced efficiency as compared to the normal in-house services. Therefore, IT professionals trained in AWS are highly demanded as they will take the organization to a replacement height through better data management.

#aws #aws online training #aws online course #aws course #aws training #aws training in noida

Hire Dedicated AWS Developer

Want to Hire AWS Developer for cloud computing services?

At HourlyDeveloper.io, we leverage maximum benefits from the AWS platform ensuring prominent Solutions for business requirements. Hire Dedicated AWS Developer and make business solutions truly global and accessible through the power of cloud environment and AWS with an extensive set of on-demand global functionality.

Consult with experts:- https://bit.ly/2C5M6cz

#aws development company #hire dedicated aws developer #aws development services #aws development #aws developer #aws