Arno  Bradtke

Arno Bradtke

1597944120

Integrating Prometheus and Grafana on Kubernetes

In this blog I have deployed two monitoring tools namely Prometheus and Grafana on top of Kubernetes. The main issue solved is that when the pod gets deleted the data is also lost to resolve this I used the PVC feature of kubernetes to make data persistent.

Image for post

Let us know something about PV, PVCs, Prometheus, and Grafana:

**_PV: _**A Persistent Volume (PV) is a piece of storage in the cluster that has been provisioned by an administrator or dynamically provisioned using Storage Classes.

PVCs: A PersistentVolumeClaim (PVC) is a request for storage by a user. It is similar to a Pod. Pods consume node resources and PVCs consume PV resources. Pods can request specific levels of resources (CPU and Memory).

_Prometheus: _Prometheus is an open-source systems monitoring and alerting toolkit.Prometheus scrapes metrics from instrumented jobs, either directly or via an intermediary push gateway for short-lived jobs. It stores all scraped samples locally and runs rules over this data to either aggregate and record new time series from existing data or generate alerts.

_Grafana: _Grafana is open-source visualization and analytics software. It allows you to query, visualize, alert on, and explore your metrics no matter where they are stored. It provides you with tools to turn your time-series database (TSDB) data into beautiful graphs and visualizations.

Here’s how it is done…

Images of Prometheus and Grafana are made using Dockerfile and uploaded on Docker Hub.

Image for post

Dockerfile for Prometheus

Image for post

#grafana #deployment #prometheus #pvc #kubernetes

What is GEEK

Buddha Community

Integrating Prometheus and Grafana on Kubernetes
Christa  Stehr

Christa Stehr

1602964260

50+ Useful Kubernetes Tools for 2020 - Part 2

Introduction

Last year, we provided a list of Kubernetes tools that proved so popular we have decided to curate another list of some useful additions for working with the platform—among which are many tools that we personally use here at Caylent. Check out the original tools list here in case you missed it.

According to a recent survey done by Stackrox, the dominance Kubernetes enjoys in the market continues to be reinforced, with 86% of respondents using it for container orchestration.

(State of Kubernetes and Container Security, 2020)

And as you can see below, more and more companies are jumping into containerization for their apps. If you’re among them, here are some tools to aid you going forward as Kubernetes continues its rapid growth.

(State of Kubernetes and Container Security, 2020)

#blog #tools #amazon elastic kubernetes service #application security #aws kms #botkube #caylent #cli #container monitoring #container orchestration tools #container security #containers #continuous delivery #continuous deployment #continuous integration #contour #developers #development #developments #draft #eksctl #firewall #gcp #github #harbor #helm #helm charts #helm-2to3 #helm-aws-secret-plugin #helm-docs #helm-operator-get-started #helm-secrets #iam #json #k-rail #k3s #k3sup #k8s #keel.sh #keycloak #kiali #kiam #klum #knative #krew #ksniff #kube #kube-prod-runtime #kube-ps1 #kube-scan #kube-state-metrics #kube2iam #kubeapps #kubebuilder #kubeconfig #kubectl #kubectl-aws-secrets #kubefwd #kubernetes #kubernetes command line tool #kubernetes configuration #kubernetes deployment #kubernetes in development #kubernetes in production #kubernetes ingress #kubernetes interfaces #kubernetes monitoring #kubernetes networking #kubernetes observability #kubernetes plugins #kubernetes secrets #kubernetes security #kubernetes security best practices #kubernetes security vendors #kubernetes service discovery #kubernetic #kubesec #kubeterminal #kubeval #kudo #kuma #microsoft azure key vault #mozilla sops #octant #octarine #open source #palo alto kubernetes security #permission-manager #pgp #rafay #rakess #rancher #rook #secrets operations #serverless function #service mesh #shell-operator #snyk #snyk container #sonobuoy #strongdm #tcpdump #tenkai #testing #tigera #tilt #vert.x #wireshark #yaml

Monitoring YugabyteDB in Kubernetes with the Prometheus Operator and Grafana

This blog post is written by guest author Bhavin Gandhi, Software Engineer at InfraCloud Technologies, Inc. InfraCloud helps startups, SMBs, and enterprises scale by adopting cloud-native technologies.

Using the Prometheus Operator has become a common choice when it comes to running Prometheus in a Kubernetes cluster. It can manage Prometheus and Alertmanager for us with the help of CRDs in Kubernetes. The kube-prometheus-stack Helm chart (formerly known as prometheus-operator) comes with Grafana, node_exporter, and more out of the box.

In a previous blog post about Prometheus, we took a look at setting up Prometheus and Grafana using manifest files. We also explored a few of the metrics exposed by YugabyteDB. In this post, we will be setting up Prometheus and Grafana using the kube-prometheus-stack chart. And we will configure Prometheus to scrape YugabyteDB pods. At the end we will take a look at the YugabyteDB Grafana dashboard that can be used to visualize all the collected metrics.

Before we begin

Before we get started with the setup, make sure you have a Kubernetes 1.16+ cluster with kubectl pointing to it. You can create a GKE cluster or an equivalent in other cloud providers or use Minikube to create a local cluster.

We will be using Helm 3 to install the charts, make sure you have it installed on your machine as well.

Installing the Prometheus Operator

The kube-prometheus-stack Helm chart installs the required CRDs as well as the operator itself. The chart creates instances of Prometheus and Alertmanager, which are the custom resources managed by the operator.

It also comes with a set of components which one would expect to have while setting up monitoring in a Kubernetes cluster. These components include: node_exporter for collecting node level metrics, kube-state-metrics which exposes Kubernetes related metrics, and Grafana for visualizing the metrics collected by Prometheus. The chart also comes with default Grafana dashboards and Prometheus alerts.

To setup Helm with the prometheus-community and yugabytedb repositories, run the following command:

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
"prometheus-community" has been added to your repositories

$ helm repo add yugabytedb https://charts.yugabyte.com
"yugabytedb" has been added to your repositories

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "yugabytedb" chart repository
...Successfully got an update from the "prometheus-community" chart repository
Update Complete. ⎈ Happy Helming!⎈ 

To create a values file for kube-prometheus-stack, execute:

cat <<EOF > kube-prom-stack-values.yaml
grafana:
  dashboardProviders:
    dashboardproviders.yaml:
      apiVersion: 1
      providers:
      - name: 'default'
        orgId: 1
        folder: ''
        type: file
        disableDeletion: false
        editable: true
        options:
          path: /var/lib/grafana/dashboards/default

  dashboards:
    default:
      yugabytedb:
        url: https://raw.githubusercontent.com/yugabyte/yugabyte-db/master/cloud/grafana/YugabyteDB.json
EOF

The above kube-prom-stack-values.yaml file tells Grafana to import the YugabyteDB dashboard from the given GitHub URL.

Now we will create a namespace and install kube-prometheus-stack in it.

$ kubectl create namespace monitoring
namespace/monitoring created

$ helm install prom prometheus-community/kube-prometheus-stack \
    --namespace monitoring \
    --values kube-prom-stack-values.yaml

To check if all the components are running fine, check the status of pods from the monitoring namespace.

$ kubectl get pods --namespace monitoring

NOTE: To keep things simple, we are not providing any other options to the Helm chart. Make sure you go through the README of kube-prometheus-stack and grafana chart to explore other options that you may need.

Installing and scraping YugabyteDB

With the monitoring setup done, let’s install the YugabyteDB Helm chart. It will install YugabyteDB and configure Prometheus to scrape (collect) the metrics from our pods. It uses the ServiceMonitor Custom Resource (CR) to achieve that.

To create a namespace and install the chart, run the following command:

$ kubectl create namespace yb-demo
namespace/yb-demo created

$ helm install cluster-1 yugabytedb/yugabyte \
    --namespace yb-demo \
    --set serviceMonitor.enabled=true \
    --set serviceMonitor.extraLabels.release=prom

If you are using Minikube, refer to this documentation section and add the resource requirements accordingly.

To check the status of all the pods from the yb-demo namespace, execute the following command. Wait till all the pods have a Running status.

#databases #distributed sql #kubernetes #grafana #prometheus #prometheus operator

Arno  Bradtke

Arno Bradtke

1597944120

Integrating Prometheus and Grafana on Kubernetes

In this blog I have deployed two monitoring tools namely Prometheus and Grafana on top of Kubernetes. The main issue solved is that when the pod gets deleted the data is also lost to resolve this I used the PVC feature of kubernetes to make data persistent.

Image for post

Let us know something about PV, PVCs, Prometheus, and Grafana:

**_PV: _**A Persistent Volume (PV) is a piece of storage in the cluster that has been provisioned by an administrator or dynamically provisioned using Storage Classes.

PVCs: A PersistentVolumeClaim (PVC) is a request for storage by a user. It is similar to a Pod. Pods consume node resources and PVCs consume PV resources. Pods can request specific levels of resources (CPU and Memory).

_Prometheus: _Prometheus is an open-source systems monitoring and alerting toolkit.Prometheus scrapes metrics from instrumented jobs, either directly or via an intermediary push gateway for short-lived jobs. It stores all scraped samples locally and runs rules over this data to either aggregate and record new time series from existing data or generate alerts.

_Grafana: _Grafana is open-source visualization and analytics software. It allows you to query, visualize, alert on, and explore your metrics no matter where they are stored. It provides you with tools to turn your time-series database (TSDB) data into beautiful graphs and visualizations.

Here’s how it is done…

Images of Prometheus and Grafana are made using Dockerfile and uploaded on Docker Hub.

Image for post

Dockerfile for Prometheus

Image for post

#grafana #deployment #prometheus #pvc #kubernetes

Maud  Rosenbaum

Maud Rosenbaum

1601051854

Kubernetes in the Cloud: Strategies for Effective Multi Cloud Implementations

Kubernetes is a highly popular container orchestration platform. Multi cloud is a strategy that leverages cloud resources from multiple vendors. Multi cloud strategies have become popular because they help prevent vendor lock-in and enable you to leverage a wide variety of cloud resources. However, multi cloud ecosystems are notoriously difficult to configure and maintain.

This article explains how you can leverage Kubernetes to reduce multi cloud complexities and improve stability, scalability, and velocity.

Kubernetes: Your Multi Cloud Strategy

Maintaining standardized application deployments becomes more challenging as your number of applications and the technologies they are based on increase. As environments, operating systems, and dependencies differ, management and operations require more effort and extensive documentation.

In the past, teams tried to get around these difficulties by creating isolated projects in the data center. Each project, including its configurations and requirements were managed independently. This required accurately predicting performance and the number of users before deployment and taking down applications to update operating systems or applications. There were many chances for error.

Kubernetes can provide an alternative to the old method, enabling teams to deploy applications independent of the environment in containers. This eliminates the need to create resource partitions and enables teams to operate infrastructure as a unified whole.

In particular, Kubernetes makes it easier to deploy a multi cloud strategy since it enables you to abstract away service differences. With Kubernetes deployments you can work from a consistent platform and optimize services and applications according to your business needs.

The Compelling Attributes of Multi Cloud Kubernetes

Multi cloud Kubernetes can provide multiple benefits beyond a single cloud deployment. Below are some of the most notable advantages.

Stability

In addition to the built-in scalability, fault tolerance, and auto-healing features of Kubernetes, multi cloud deployments can provide service redundancy. For example, you can mirror applications or split microservices across vendors. This reduces the risk of a vendor-related outage and enables you to create failovers.

#kubernetes #multicloud-strategy #kubernetes-cluster #kubernetes-top-story #kubernetes-cluster-install #kubernetes-explained #kubernetes-infrastructure #cloud

Nat  Kutch

Nat Kutch

1597891260

Monitoring Kubernetes Clusters Through Prometheus & Grafana

Monitoring is vital whether it is web application databases or kubernetes clusters. It’s about knowing how’s and what’s of application or cluster performance.

As today’s talk is specifically about monitoring kubernetes clusters, when there are thousands or millions of services running inside the cluster it’s not viable or not possible to monitor clusters by the subsequent command or manually. In this article, we will deploy Prometheus and Grafana to kubernetes cluster and monitor our cluster,

Prometheus


Prometheus is free and an open-source event monitoring tool for containers or microservices. Prometheus collects numerical data based on time series. The Prometheus server works on the principle of scraping. This invokes the metric endpoint of the various nodes that have been configured to monitor. These metrics are collected in regular timestamps and stored locally. The endpoint that was used to discard is exposed on the node.

1. Prometheus Data Retention

Prometheus data retention time is 15 days by default. The lowest retention period is 2hour. If you retain the data for the highest period more disk space will be used as there will be more data. The lowest retention period can be used when configuring remote storage for Prometheus.

2. Prometheus With Grafana

Grafana is a multi-platform visualization software that provides us a graph, the chart for a web connected to the data source. Prometheus has it’s own built-in browser expression but Grafana is the industry’s most powerful visualization software. Grafana has out of the box integration with Prometheus.

#devops #kubernetes #monitoring #prometheus #write-for-cloud-native #grafana