1596690540
Auto-scale Kafka applications on Kubernetes with KEDA
This tutorial will demonstrate auto-scaling Kafka-based consumer applications on Kubernetes using [KEDA](https://keda.sh/) which stands for Kubernetes-based Event Driven Autoscaler
[_KEDA_](https://cloudblogs.microsoft.com/opensource/2020/04/06/kubernetes-event-driven-autoscaling-keda-cncf-sandbox-project/)_ is currently a CNCF Sandbox project_
_KEDA__ can drive the scaling of any container in Kubernetes based on the number of events needing to be processed. It is a single-purpose and lightweight component that can be added to any Kubernetes cluster. KEDA works alongside standard Kubernetes components like the Horizontal Pod Autoscaler and can extend functionality without overwriting or duplication._
It has a built-in Kafka scaler which can auto-scale your Kafka consumer applications (traditional Consumer apps, Kafka Streams etc.) based on the consumer offset lag. I will be using Azure Event Hubs as the Kafka broker (although the concepts apply to any Kafka cluster) and Azure Kubernetes Service for the Kubernetes cluster (feel free to use alternatives such as minikube)
_Code is available on _
[_GitHub_](https://github.com/abhirockzz/keda-eventhubs-kafka)
We will go through the following:
- A quick overview
- The app and KEDA configuration (mostly YAMLs! to be honest)
- How to setup KEDA and required Azure services
- Deploy the solution and watch auto-scaling in action
Overview
Here are the key components:
- Producer app: This is a simple Go app that produces simulated JSON data to Kafka. It uses the
[sarama](https://github.com/Shopify/sarama)library. You can run this as a Docker container or directly as a Go app (details in an upcoming section) - Consumer app: This is another Go app that consumes data from Kafka. To add a bit of variety, it uses the Confluent Go Kafka client. You will be running this as a Kubernetes
Deployment(details in an upcoming section) KEDAScaledObject(which defines the auto-scaling criteria based on Kafka) and other supporting manifests
Pre-requisites
kubectl - https://kubernetes.io/docs/tasks/tools/install-kubectl/
If you choose to use Azure Event Hubs, Azure Kubernetes Service (or both) you will need a Microsoft Azure account. Go ahead and sign up for a free one!
I will be using Helm to install KEDA. Here is the documentation to install Helm - https://helm.sh/docs/intro/install/
For alternative ways (
_Operator Hub__ or YAML files) of installing_KEDA_, _take a look at the documentation
Here is how you can set up the required Azure services.
I recommend installing the below services as a part of a single Azure Resource Group which makes it easy to clean up these services
Azure Event Hubs
Azure Event Hubs is a data streaming platform and event ingestion service. It can receive and process millions of events per second. It also provides a Kafka endpoint that can be used by existing Kafka based applications as an alternative to running your own Kafka cluster. Event Hubs supports Apache Kafka protocol 1.0 and later, and works with existing Kafka client applications and other tools in the Kafka ecosystem including Kafka Connect (demonstrated in this blog), MirrorMaker etc.
To set up an Azure Event Hubs cluster, you can choose from a variety of options including the Azure portal, Azure CLI, Azure PowerShell or an ARM template. Once the setup is complete, you will need the connection string (that will be used in subsequent steps) for authenticating to Event Hubs — use this guide to finish this step.
Please ensure that you also create an Event Hub (Kafka topic) to/from which we can send/receive data
Azure Kubernetes Service
Azure Kubernetes Service (AKS) makes it simple to deploy a managed Kubernetes cluster in Azure. It reduces the complexity and operational overhead of managing Kubernetes by offloading much of that responsibility to Azure. Here are examples of how you can set up an AKS cluster using Azure CLI, Azure portal or ARM template
Install KEDA
helm repo add kedacore https://kedacore.github.io/charts
helm repo update
kubectl create namespace keda
helm install keda kedacore/keda --namespace keda
This will install the KEDA Operator and the KEDA Metrics API server (as separate Deployments)
kubectl get deployment -n keda
NAME READY UP-TO-DATE AVAILABLE AGE
keda-operator 1/1 1 1 1h
keda-operator-metrics-apiserver 1/1 1 1 1h
To check KEDA Operator logs
kubectl logs -f $(kubectl get pod -l=app=keda-operator -o jsonpath='{.items[0].metadata.name}' -n keda) -n keda
#azure #kafka #tutorial #open-source #kubernetes #keda
What is GEEK
Buddha Community
1602964260
50+ Useful Kubernetes Tools for 2020 - Part 2
Introduction
Last year, we provided a list of Kubernetes tools that proved so popular we have decided to curate another list of some useful additions for working with the platform—among which are many tools that we personally use here at Caylent. Check out the original tools list here in case you missed it.
According to a recent survey done by Stackrox, the dominance Kubernetes enjoys in the market continues to be reinforced, with 86% of respondents using it for container orchestration.
And as you can see below, more and more companies are jumping into containerization for their apps. If you’re among them, here are some tools to aid you going forward as Kubernetes continues its rapid growth.
#blog #tools #amazon elastic kubernetes service #application security #aws kms #botkube #caylent #cli #container monitoring #container orchestration tools #container security #containers #continuous delivery #continuous deployment #continuous integration #contour #developers #development #developments #draft #eksctl #firewall #gcp #github #harbor #helm #helm charts #helm-2to3 #helm-aws-secret-plugin #helm-docs #helm-operator-get-started #helm-secrets #iam #json #k-rail #k3s #k3sup #k8s #keel.sh #keycloak #kiali #kiam #klum #knative #krew #ksniff #kube #kube-prod-runtime #kube-ps1 #kube-scan #kube-state-metrics #kube2iam #kubeapps #kubebuilder #kubeconfig #kubectl #kubectl-aws-secrets #kubefwd #kubernetes #kubernetes command line tool #kubernetes configuration #kubernetes deployment #kubernetes in development #kubernetes in production #kubernetes ingress #kubernetes interfaces #kubernetes monitoring #kubernetes networking #kubernetes observability #kubernetes plugins #kubernetes secrets #kubernetes security #kubernetes security best practices #kubernetes security vendors #kubernetes service discovery #kubernetic #kubesec #kubeterminal #kubeval #kudo #kuma #microsoft azure key vault #mozilla sops #octant #octarine #open source #palo alto kubernetes security #permission-manager #pgp #rafay #rakess #rancher #rook #secrets operations #serverless function #service mesh #shell-operator #snyk #snyk container #sonobuoy #strongdm #tcpdump #tenkai #testing #tigera #tilt #vert.x #wireshark #yaml
1596690540
Auto-scale Kafka applications on Kubernetes with KEDA
This tutorial will demonstrate auto-scaling Kafka-based consumer applications on Kubernetes using [KEDA](https://keda.sh/) which stands for Kubernetes-based Event Driven Autoscaler
[_KEDA_](https://cloudblogs.microsoft.com/opensource/2020/04/06/kubernetes-event-driven-autoscaling-keda-cncf-sandbox-project/)_ is currently a CNCF Sandbox project_
_KEDA__ can drive the scaling of any container in Kubernetes based on the number of events needing to be processed. It is a single-purpose and lightweight component that can be added to any Kubernetes cluster. KEDA works alongside standard Kubernetes components like the Horizontal Pod Autoscaler and can extend functionality without overwriting or duplication._
It has a built-in Kafka scaler which can auto-scale your Kafka consumer applications (traditional Consumer apps, Kafka Streams etc.) based on the consumer offset lag. I will be using Azure Event Hubs as the Kafka broker (although the concepts apply to any Kafka cluster) and Azure Kubernetes Service for the Kubernetes cluster (feel free to use alternatives such as minikube)
_Code is available on _
[_GitHub_](https://github.com/abhirockzz/keda-eventhubs-kafka)
We will go through the following:
- A quick overview
- The app and KEDA configuration (mostly YAMLs! to be honest)
- How to setup KEDA and required Azure services
- Deploy the solution and watch auto-scaling in action
Overview
Here are the key components:
- Producer app: This is a simple Go app that produces simulated JSON data to Kafka. It uses the
[sarama](https://github.com/Shopify/sarama)library. You can run this as a Docker container or directly as a Go app (details in an upcoming section) - Consumer app: This is another Go app that consumes data from Kafka. To add a bit of variety, it uses the Confluent Go Kafka client. You will be running this as a Kubernetes
Deployment(details in an upcoming section) KEDAScaledObject(which defines the auto-scaling criteria based on Kafka) and other supporting manifests
Pre-requisites
kubectl - https://kubernetes.io/docs/tasks/tools/install-kubectl/
If you choose to use Azure Event Hubs, Azure Kubernetes Service (or both) you will need a Microsoft Azure account. Go ahead and sign up for a free one!
I will be using Helm to install KEDA. Here is the documentation to install Helm - https://helm.sh/docs/intro/install/
For alternative ways (
_Operator Hub__ or YAML files) of installing_KEDA_, _take a look at the documentation
Here is how you can set up the required Azure services.
I recommend installing the below services as a part of a single Azure Resource Group which makes it easy to clean up these services
Azure Event Hubs
Azure Event Hubs is a data streaming platform and event ingestion service. It can receive and process millions of events per second. It also provides a Kafka endpoint that can be used by existing Kafka based applications as an alternative to running your own Kafka cluster. Event Hubs supports Apache Kafka protocol 1.0 and later, and works with existing Kafka client applications and other tools in the Kafka ecosystem including Kafka Connect (demonstrated in this blog), MirrorMaker etc.
To set up an Azure Event Hubs cluster, you can choose from a variety of options including the Azure portal, Azure CLI, Azure PowerShell or an ARM template. Once the setup is complete, you will need the connection string (that will be used in subsequent steps) for authenticating to Event Hubs — use this guide to finish this step.
Please ensure that you also create an Event Hub (Kafka topic) to/from which we can send/receive data
Azure Kubernetes Service
Azure Kubernetes Service (AKS) makes it simple to deploy a managed Kubernetes cluster in Azure. It reduces the complexity and operational overhead of managing Kubernetes by offloading much of that responsibility to Azure. Here are examples of how you can set up an AKS cluster using Azure CLI, Azure portal or ARM template
Install KEDA
helm repo add kedacore https://kedacore.github.io/charts
helm repo update
kubectl create namespace keda
helm install keda kedacore/keda --namespace keda
This will install the KEDA Operator and the KEDA Metrics API server (as separate Deployments)
kubectl get deployment -n keda
NAME READY UP-TO-DATE AVAILABLE AGE
keda-operator 1/1 1 1 1h
keda-operator-metrics-apiserver 1/1 1 1 1h
To check KEDA Operator logs
kubectl logs -f $(kubectl get pod -l=app=keda-operator -o jsonpath='{.items[0].metadata.name}' -n keda) -n keda
#azure #kafka #tutorial #open-source #kubernetes #keda
1601051854
Kubernetes in the Cloud: Strategies for Effective Multi Cloud Implementations
Kubernetes is a highly popular container orchestration platform. Multi cloud is a strategy that leverages cloud resources from multiple vendors. Multi cloud strategies have become popular because they help prevent vendor lock-in and enable you to leverage a wide variety of cloud resources. However, multi cloud ecosystems are notoriously difficult to configure and maintain.
This article explains how you can leverage Kubernetes to reduce multi cloud complexities and improve stability, scalability, and velocity.
Kubernetes: Your Multi Cloud Strategy
Maintaining standardized application deployments becomes more challenging as your number of applications and the technologies they are based on increase. As environments, operating systems, and dependencies differ, management and operations require more effort and extensive documentation.
In the past, teams tried to get around these difficulties by creating isolated projects in the data center. Each project, including its configurations and requirements were managed independently. This required accurately predicting performance and the number of users before deployment and taking down applications to update operating systems or applications. There were many chances for error.
Kubernetes can provide an alternative to the old method, enabling teams to deploy applications independent of the environment in containers. This eliminates the need to create resource partitions and enables teams to operate infrastructure as a unified whole.
In particular, Kubernetes makes it easier to deploy a multi cloud strategy since it enables you to abstract away service differences. With Kubernetes deployments you can work from a consistent platform and optimize services and applications according to your business needs.
The Compelling Attributes of Multi Cloud Kubernetes
Multi cloud Kubernetes can provide multiple benefits beyond a single cloud deployment. Below are some of the most notable advantages.
Stability
In addition to the built-in scalability, fault tolerance, and auto-healing features of Kubernetes, multi cloud deployments can provide service redundancy. For example, you can mirror applications or split microservices across vendors. This reduces the risk of a vendor-related outage and enables you to create failovers.
#kubernetes #multicloud-strategy #kubernetes-cluster #kubernetes-top-story #kubernetes-cluster-install #kubernetes-explained #kubernetes-infrastructure #cloud
1595344320
Wondering how to upgrade your skills in the pandemic? Here's a simple way you can do it.
Corona Virus Pandemic has brought the world to a standstill.
Countries are on a major lockdown. Schools, colleges, theatres, gym, clubs, and all other public places are shut down, the country’s economy is suffering, human health is on stake, people are losing their jobs and nobody knows how worse it can get.
Since most of the places are on lockdown, and you are working from home or have enough time to nourish your skills, then you should use this time wisely! We always complain that we want some ‘time’ to learn and upgrade our knowledge but don’t get it due to our ‘busy schedules’. So, now is the time to make a ‘list of skills’ and learn and upgrade your skills at home!
And for the technology-loving people like us, Knoldus Techhub has already helped us a lot in doing it in a short span of time!
If you are still not aware of it, don’t worry as Georgia Byng has well said,
“No time is better than the present”
– Georgia Byng, a British children’s writer, illustrator, actress and film producer.
No matter if you are a developer (be it front-end or back-end) or a data scientist, tester, or a DevOps person, or, a learner who has a keen interest in technology, Knoldus Techhub has brought it all for you under one common roof.
From technologies like Scala, spark, elastic-search to angular, go, machine learning, it has a total of 20 technologies with some recently added ones i.e. DAML, test automation, snowflake, and ionic.
How to upgrade your skills?
Every technology in Tech-hub has n number of templates. Once you click on any specific technology you’ll be able to see all the templates of that technology. Since these templates are downloadable, you need to provide your email to get the template downloadable link in your mail.
These templates helps you learn the practical implementation of a topic with so much of ease. Using these templates you can learn and kick-start your development in no time.
Apart from your learning, there are some out of the box templates, that can help provide the solution to your business problem that has all the basic dependencies/ implementations already plugged in. Tech hub names these templates as xlr8rs (pronounced as accelerators).
xlr8rs make your development real fast by just adding your core business logic to the template.
If you are looking for a template that’s not available, you can also request a template may be for learning or requesting for a solution to your business problem and tech-hub will connect with you to provide you the solution. Isn’t this helpful 🙂
Confused with which technology to start with?
To keep you updated, the Knoldus tech hub provides you with the information on the most trending technology and the most downloaded templates at present. This you’ll be informed and learn the one that’s most trending.
Since we believe:
“There’s always a scope of improvement“
If you still feel like it isn’t helping you in learning and development, you can provide your feedback in the feedback section in the bottom right corner of the website.
#ai #akka #akka-http #akka-streams #amazon ec2 #angular 6 #angular 9 #angular material #apache flink #apache kafka #apache spark #api testing #artificial intelligence #aws #aws services #big data and fast data #blockchain #css #daml #devops #elasticsearch #flink #functional programming #future #grpc #html #hybrid application development #ionic framework #java #java11 #kubernetes #lagom #microservices #ml # ai and data engineering #mlflow #mlops #mobile development #mongodb #non-blocking #nosql #play #play 2.4.x #play framework #python #react #reactive application #reactive architecture #reactive programming #rust #scala #scalatest #slick #software #spark #spring boot #sql #streaming #tech blogs #testing #user interface (ui) #web #web application #web designing #angular #coronavirus #daml #development #devops #elasticsearch #golang #ionic #java #kafka #knoldus #lagom #learn #machine learning #ml #pandemic #play framework #scala #skills #snowflake #spark streaming #techhub #technology #test automation #time management #upgrade
1618491960
Auto-scaling + Kubernetes = KEDA
Prerequisites for this article — Kubernetes, Azure Function App, Docker knowledge
Use Case
There is a web app, which uploads files from end users and puts them in a cloud storage. The goal is to process those files with the minimum delays. To do that we want the job to be scaled, based on the queue size.
Components
- Web App to get user inputs (files)
- Azure Blob Storage to keep files
- Kubernetes Cluster with KEDA (really, v 2.0)
- Azure Function App as a JOB, hosted in the Kubernetes Cluster, to process the files from the storage
- Container Registry to host your docker images
The implementation of the web app in this case does not matter. Let’s focus on the azure function app.
#keda #kubernetes