Auto-scale Kafka applications on Kubernetes with KEDA

This tutorial will demonstrate auto-scaling Kafka-based consumer applications on Kubernetes using [KEDA](https://keda.sh/) which stands for Kubernetes-based Event Driven Autoscaler

[_KEDA_](https://cloudblogs.microsoft.com/opensource/2020/04/06/kubernetes-event-driven-autoscaling-keda-cncf-sandbox-project/)_ is currently a CNCF Sandbox project_

_KEDA__ can drive the scaling of any container in Kubernetes based on the number of events needing to be processed. It is a single-purpose and lightweight component that can be added to any Kubernetes cluster. KEDA works alongside standard Kubernetes components like the Horizontal Pod Autoscaler and can extend functionality without overwriting or duplication._

Image for post

It has a built-in Kafka scaler which can auto-scale your Kafka consumer applications (traditional Consumer apps, Kafka Streams etc.) based on the consumer offset lag. I will be using Azure Event Hubs as the Kafka broker (although the concepts apply to any Kafka cluster) and Azure Kubernetes Service for the Kubernetes cluster (feel free to use alternatives such as minikube)

_Code is available on _[_GitHub_](https://github.com/abhirockzz/keda-eventhubs-kafka)

We will go through the following:

  • A quick overview
  • The app and KEDA configuration (mostly YAMLs! to be honest)
  • How to setup KEDA and required Azure services
  • Deploy the solution and watch auto-scaling in action

Overview

Here are the key components:

  • Producer app: This is a simple Go app that produces simulated JSON data to Kafka. It uses the [sarama](https://github.com/Shopify/sarama) library. You can run this as a Docker container or directly as a Go app (details in an upcoming section)
  • Consumer app: This is another Go app that consumes data from Kafka. To add a bit of variety, it uses the Confluent Go Kafka client. You will be running this as a Kubernetes Deployment (details in an upcoming section)
  • KEDA ScaledObject (which defines the auto-scaling criteria based on Kafka) and other supporting manifests

Pre-requisites

kubectl - https://kubernetes.io/docs/tasks/tools/install-kubectl/

If you choose to use Azure Event Hubs, Azure Kubernetes Service (or both) you will need a Microsoft Azure account. Go ahead and sign up for a free one!

I will be using Helm to install KEDA. Here is the documentation to install Helm - https://helm.sh/docs/intro/install/

For alternative ways (_Operator Hub__ or YAML files) of installing _KEDA_, _take a look at the documentation

Here is how you can set up the required Azure services.

I recommend installing the below services as a part of a single Azure Resource Group which makes it easy to clean up these services

Azure Event Hubs

Azure Event Hubs is a data streaming platform and event ingestion service. It can receive and process millions of events per second. It also provides a Kafka endpoint that can be used by existing Kafka based applications as an alternative to running your own Kafka cluster. Event Hubs supports Apache Kafka protocol 1.0 and later, and works with existing Kafka client applications and other tools in the Kafka ecosystem including Kafka Connect (demonstrated in this blog), MirrorMaker etc.

To set up an Azure Event Hubs cluster, you can choose from a variety of options including the Azure portalAzure CLIAzure PowerShell or an ARM template. Once the setup is complete, you will need the connection string (that will be used in subsequent steps) for authenticating to Event Hubs — use this guide to finish this step.

Please ensure that you also create an Event Hub (Kafka topic) to/from which we can send/receive data

Azure Kubernetes Service

Azure Kubernetes Service (AKS) makes it simple to deploy a managed Kubernetes cluster in Azure. It reduces the complexity and operational overhead of managing Kubernetes by offloading much of that responsibility to Azure. Here are examples of how you can set up an AKS cluster using Azure CLIAzure portal or ARM template


Install KEDA

helm repo add kedacore https://kedacore.github.io/charts
helm repo update

kubectl create namespace keda
helm install keda kedacore/keda --namespace keda

This will install the KEDA Operator and the KEDA Metrics API server (as separate Deployments)

kubectl get deployment -n keda

NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
keda-operator                     1/1     1            1           1h
keda-operator-metrics-apiserver   1/1     1            1           1h

To check KEDA Operator logs

kubectl logs -f $(kubectl get pod -l=app=keda-operator -o jsonpath='{.items[0].metadata.name}' -n keda) -n keda

#azure #kafka #tutorial #open-source #kubernetes #keda

What is GEEK

Buddha Community

Auto-scale Kafka applications on Kubernetes with KEDA
Christa  Stehr

Christa Stehr

1602964260

50+ Useful Kubernetes Tools for 2020 - Part 2

Introduction

Last year, we provided a list of Kubernetes tools that proved so popular we have decided to curate another list of some useful additions for working with the platform—among which are many tools that we personally use here at Caylent. Check out the original tools list here in case you missed it.

According to a recent survey done by Stackrox, the dominance Kubernetes enjoys in the market continues to be reinforced, with 86% of respondents using it for container orchestration.

(State of Kubernetes and Container Security, 2020)

And as you can see below, more and more companies are jumping into containerization for their apps. If you’re among them, here are some tools to aid you going forward as Kubernetes continues its rapid growth.

(State of Kubernetes and Container Security, 2020)

#blog #tools #amazon elastic kubernetes service #application security #aws kms #botkube #caylent #cli #container monitoring #container orchestration tools #container security #containers #continuous delivery #continuous deployment #continuous integration #contour #developers #development #developments #draft #eksctl #firewall #gcp #github #harbor #helm #helm charts #helm-2to3 #helm-aws-secret-plugin #helm-docs #helm-operator-get-started #helm-secrets #iam #json #k-rail #k3s #k3sup #k8s #keel.sh #keycloak #kiali #kiam #klum #knative #krew #ksniff #kube #kube-prod-runtime #kube-ps1 #kube-scan #kube-state-metrics #kube2iam #kubeapps #kubebuilder #kubeconfig #kubectl #kubectl-aws-secrets #kubefwd #kubernetes #kubernetes command line tool #kubernetes configuration #kubernetes deployment #kubernetes in development #kubernetes in production #kubernetes ingress #kubernetes interfaces #kubernetes monitoring #kubernetes networking #kubernetes observability #kubernetes plugins #kubernetes secrets #kubernetes security #kubernetes security best practices #kubernetes security vendors #kubernetes service discovery #kubernetic #kubesec #kubeterminal #kubeval #kudo #kuma #microsoft azure key vault #mozilla sops #octant #octarine #open source #palo alto kubernetes security #permission-manager #pgp #rafay #rakess #rancher #rook #secrets operations #serverless function #service mesh #shell-operator #snyk #snyk container #sonobuoy #strongdm #tcpdump #tenkai #testing #tigera #tilt #vert.x #wireshark #yaml

Auto-scale Kafka applications on Kubernetes with KEDA

This tutorial will demonstrate auto-scaling Kafka-based consumer applications on Kubernetes using [KEDA](https://keda.sh/) which stands for Kubernetes-based Event Driven Autoscaler

[_KEDA_](https://cloudblogs.microsoft.com/opensource/2020/04/06/kubernetes-event-driven-autoscaling-keda-cncf-sandbox-project/)_ is currently a CNCF Sandbox project_

_KEDA__ can drive the scaling of any container in Kubernetes based on the number of events needing to be processed. It is a single-purpose and lightweight component that can be added to any Kubernetes cluster. KEDA works alongside standard Kubernetes components like the Horizontal Pod Autoscaler and can extend functionality without overwriting or duplication._

Image for post

It has a built-in Kafka scaler which can auto-scale your Kafka consumer applications (traditional Consumer apps, Kafka Streams etc.) based on the consumer offset lag. I will be using Azure Event Hubs as the Kafka broker (although the concepts apply to any Kafka cluster) and Azure Kubernetes Service for the Kubernetes cluster (feel free to use alternatives such as minikube)

_Code is available on _[_GitHub_](https://github.com/abhirockzz/keda-eventhubs-kafka)

We will go through the following:

  • A quick overview
  • The app and KEDA configuration (mostly YAMLs! to be honest)
  • How to setup KEDA and required Azure services
  • Deploy the solution and watch auto-scaling in action

Overview

Here are the key components:

  • Producer app: This is a simple Go app that produces simulated JSON data to Kafka. It uses the [sarama](https://github.com/Shopify/sarama) library. You can run this as a Docker container or directly as a Go app (details in an upcoming section)
  • Consumer app: This is another Go app that consumes data from Kafka. To add a bit of variety, it uses the Confluent Go Kafka client. You will be running this as a Kubernetes Deployment (details in an upcoming section)
  • KEDA ScaledObject (which defines the auto-scaling criteria based on Kafka) and other supporting manifests

Pre-requisites

kubectl - https://kubernetes.io/docs/tasks/tools/install-kubectl/

If you choose to use Azure Event Hubs, Azure Kubernetes Service (or both) you will need a Microsoft Azure account. Go ahead and sign up for a free one!

I will be using Helm to install KEDA. Here is the documentation to install Helm - https://helm.sh/docs/intro/install/

For alternative ways (_Operator Hub__ or YAML files) of installing _KEDA_, _take a look at the documentation

Here is how you can set up the required Azure services.

I recommend installing the below services as a part of a single Azure Resource Group which makes it easy to clean up these services

Azure Event Hubs

Azure Event Hubs is a data streaming platform and event ingestion service. It can receive and process millions of events per second. It also provides a Kafka endpoint that can be used by existing Kafka based applications as an alternative to running your own Kafka cluster. Event Hubs supports Apache Kafka protocol 1.0 and later, and works with existing Kafka client applications and other tools in the Kafka ecosystem including Kafka Connect (demonstrated in this blog), MirrorMaker etc.

To set up an Azure Event Hubs cluster, you can choose from a variety of options including the Azure portalAzure CLIAzure PowerShell or an ARM template. Once the setup is complete, you will need the connection string (that will be used in subsequent steps) for authenticating to Event Hubs — use this guide to finish this step.

Please ensure that you also create an Event Hub (Kafka topic) to/from which we can send/receive data

Azure Kubernetes Service

Azure Kubernetes Service (AKS) makes it simple to deploy a managed Kubernetes cluster in Azure. It reduces the complexity and operational overhead of managing Kubernetes by offloading much of that responsibility to Azure. Here are examples of how you can set up an AKS cluster using Azure CLIAzure portal or ARM template


Install KEDA

helm repo add kedacore https://kedacore.github.io/charts
helm repo update

kubectl create namespace keda
helm install keda kedacore/keda --namespace keda

This will install the KEDA Operator and the KEDA Metrics API server (as separate Deployments)

kubectl get deployment -n keda

NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
keda-operator                     1/1     1            1           1h
keda-operator-metrics-apiserver   1/1     1            1           1h

To check KEDA Operator logs

kubectl logs -f $(kubectl get pod -l=app=keda-operator -o jsonpath='{.items[0].metadata.name}' -n keda) -n keda

#azure #kafka #tutorial #open-source #kubernetes #keda

Maud  Rosenbaum

Maud Rosenbaum

1601051854

Kubernetes in the Cloud: Strategies for Effective Multi Cloud Implementations

Kubernetes is a highly popular container orchestration platform. Multi cloud is a strategy that leverages cloud resources from multiple vendors. Multi cloud strategies have become popular because they help prevent vendor lock-in and enable you to leverage a wide variety of cloud resources. However, multi cloud ecosystems are notoriously difficult to configure and maintain.

This article explains how you can leverage Kubernetes to reduce multi cloud complexities and improve stability, scalability, and velocity.

Kubernetes: Your Multi Cloud Strategy

Maintaining standardized application deployments becomes more challenging as your number of applications and the technologies they are based on increase. As environments, operating systems, and dependencies differ, management and operations require more effort and extensive documentation.

In the past, teams tried to get around these difficulties by creating isolated projects in the data center. Each project, including its configurations and requirements were managed independently. This required accurately predicting performance and the number of users before deployment and taking down applications to update operating systems or applications. There were many chances for error.

Kubernetes can provide an alternative to the old method, enabling teams to deploy applications independent of the environment in containers. This eliminates the need to create resource partitions and enables teams to operate infrastructure as a unified whole.

In particular, Kubernetes makes it easier to deploy a multi cloud strategy since it enables you to abstract away service differences. With Kubernetes deployments you can work from a consistent platform and optimize services and applications according to your business needs.

The Compelling Attributes of Multi Cloud Kubernetes

Multi cloud Kubernetes can provide multiple benefits beyond a single cloud deployment. Below are some of the most notable advantages.

Stability

In addition to the built-in scalability, fault tolerance, and auto-healing features of Kubernetes, multi cloud deployments can provide service redundancy. For example, you can mirror applications or split microservices across vendors. This reduces the risk of a vendor-related outage and enables you to create failovers.

#kubernetes #multicloud-strategy #kubernetes-cluster #kubernetes-top-story #kubernetes-cluster-install #kubernetes-explained #kubernetes-infrastructure #cloud

Roberta  Ward

Roberta Ward

1595344320

Wondering how to upgrade your skills in the pandemic? Here's a simple way you can do it.

Corona Virus Pandemic has brought the world to a standstill.

Countries are on a major lockdown. Schools, colleges, theatres, gym, clubs, and all other public places are shut down, the country’s economy is suffering, human health is on stake, people are losing their jobs and nobody knows how worse it can get.

Since most of the places are on lockdown, and you are working from home or have enough time to nourish your skills, then you should use this time wisely! We always complain that we want some ‘time’ to learn and upgrade our knowledge but don’t get it due to our ‘busy schedules’. So, now is the time to make a ‘list of skills’ and learn and upgrade your skills at home!

And for the technology-loving people like us, Knoldus Techhub has already helped us a lot in doing it in a short span of time!

If you are still not aware of it, don’t worry as Georgia Byng has well said,

“No time is better than the present”

– Georgia Byng, a British children’s writer, illustrator, actress and film producer.

No matter if you are a developer (be it front-end or back-end) or a data scientisttester, or a DevOps person, or, a learner who has a keen interest in technology, Knoldus Techhub has brought it all for you under one common roof.

From technologies like Scala, spark, elastic-search to angular, go, machine learning, it has a total of 20 technologies with some recently added ones i.e. DAML, test automation, snowflake, and ionic.

How to upgrade your skills?

Every technology in Tech-hub has n number of templates. Once you click on any specific technology you’ll be able to see all the templates of that technology. Since these templates are downloadable, you need to provide your email to get the template downloadable link in your mail.

These templates helps you learn the practical implementation of a topic with so much of ease. Using these templates you can learn and kick-start your development in no time.

Apart from your learning, there are some out of the box templates, that can help provide the solution to your business problem that has all the basic dependencies/ implementations already plugged in. Tech hub names these templates as xlr8rs (pronounced as accelerators).

xlr8rs make your development real fast by just adding your core business logic to the template.

If you are looking for a template that’s not available, you can also request a template may be for learning or requesting for a solution to your business problem and tech-hub will connect with you to provide you the solution. Isn’t this helpful 🙂

Confused with which technology to start with?

To keep you updated, the Knoldus tech hub provides you with the information on the most trending technology and the most downloaded templates at present. This you’ll be informed and learn the one that’s most trending.

Since we believe:

“There’s always a scope of improvement“

If you still feel like it isn’t helping you in learning and development, you can provide your feedback in the feedback section in the bottom right corner of the website.

#ai #akka #akka-http #akka-streams #amazon ec2 #angular 6 #angular 9 #angular material #apache flink #apache kafka #apache spark #api testing #artificial intelligence #aws #aws services #big data and fast data #blockchain #css #daml #devops #elasticsearch #flink #functional programming #future #grpc #html #hybrid application development #ionic framework #java #java11 #kubernetes #lagom #microservices #ml # ai and data engineering #mlflow #mlops #mobile development #mongodb #non-blocking #nosql #play #play 2.4.x #play framework #python #react #reactive application #reactive architecture #reactive programming #rust #scala #scalatest #slick #software #spark #spring boot #sql #streaming #tech blogs #testing #user interface (ui) #web #web application #web designing #angular #coronavirus #daml #development #devops #elasticsearch #golang #ionic #java #kafka #knoldus #lagom #learn #machine learning #ml #pandemic #play framework #scala #skills #snowflake #spark streaming #techhub #technology #test automation #time management #upgrade

Auto-scaling + Kubernetes = KEDA

Prerequisites for this article — Kubernetes knowledge.

If you are a developer or a devops export, then you will definetly be faced with a task to create an auto-scaling job.

A typical use case is when you have a file storage, and you need to process it as soon as new files arrive, and for that you would need parallel jobs to be running. If you have one job, it won’t be fast enough. If you host more than one job, then it could be too expensive, because they would be idle and consume resources.

So the dilemma is “quick & expensive” vs “slow & cheap”.

This is where Kubernetes and specifically KEDA could help you.

KEDA is a Kubernetes-based Event Driven Autoscaler. With KEDA, you can drive the scaling of any container in Kubernetes based on the number of events needing to be processed.

#kubernetes #autoscaling #keda