Mireille  Von

Mireille Von

1670324640

Explore Kubernetes Logging in Production

In this article, we will learn about Kubernetes Logging in Production. Learn about scalable logging patterns for your production Kubernetes clusters. Historically, in monolithic architectures, logs were stored directly on bare metal or virtual machines. They never left the machine disk and the operations team would check each one for logs as needed.

This worked on long-lived machines, but machines in the cloud are ephemeral. As more companies run their services on containers and orchestrate deployments with Kubernetes, logs can no longer be stored on machines and implementing a log management strategy is of the utmost importance.

Logs are an effective way of debugging and monitoring your applications, and they need to be stored on a separate backend where they can be queried and analyzed in case of pod or node failures. These separate backends include systems like Elasticsearch, GCP’s Stackdriver, and AWS’ Cloudwatch.

Storing logs off of the cluster in a storage backend is called cluster-level logging. In this article we’ll discuss how to implement this approach in your own Kubernetes cluster.

Logging Architectures

In a Kubernetes cluster there are two main log sources, your application and the system components.

Your application runs as a container in the Kubernetes cluster and the container runtime takes care of fetching your application’s logs while Docker redirects those logs to the stdout and stderr streams. In a Kubernetes cluster, both of these streams are written to a JSON file on the cluster node.

These container logs can be fetched anytime with the following command:

kubectl logs podname

The other source of logs are system components. Some of the system components (namely kube-scheduler and kube-proxy) run as containers and follow the same logging principles as your application.

The other system components (kubelet and container runtime itself) run as a native service. If systemd is available on the machine the components write logs in journald, otherwise they write a .log file in /var/log directory.

Now that we understand which components of your application and cluster generate logs and where they’re stored, let’s look at some common patterns to offload these logs to separate storage systems.

Logging Patterns

The two most prominent patterns for collecting logs are the sidecar pattern and the DaemonSet pattern.

1. DaemonSet pattern

In the DaemonSet pattern, logging agents are deployed as pods via the DaemonSet resource in Kubernetes. Deploying a DaemonSet ensures that each node in the cluster has one pod with a logging agent running. This logging agent is configured to read the logs from /var/logs directory and send them to the storage backend. You can see a diagram of this configuration in figure 1.

DaemonSet pattern

Figure 1: A logging agent running per node via a DaemonSet

2. Sidecar pattern

Alternatively, in the sidecar pattern a dedicated container runs along every application container in the same pod. This sidecar can be of two types, streaming sidecar or logging agent sidecar.

The streaming sidecar is used when you are running an application that writes the logs to a file instead of stdout/stderr streams, or one that writes the logs in a nonstandard format. In that case, you can use a streaming sidecar container to publish the logs from the file to its own stdout/stderr stream, which can then be picked up by Kubernetes itself.

The streaming sidecar can also bring parity to the log structure by transforming the log messages to standard log format. You can see this pattern in figure 2.

Streaming sidecar pattern

Figure 2: Streaming sidecar pattern

Another approach is the logging agent sidecar, where the sidecar itself ships the logs to the storage backend. Each pod contains a logging agent like Fluentd or Filebeat, which captures the logs from the application container and sends them directly to the storage backend, as illustrated in figure 3.

Logging agent sidecar pattern

Figure 3: Logging agent sidecar pattern

Pros and Cons

Now that we’ve gone over both the DaemonSet and sidecar approaches, let’s get acquainted with the pros and cons of each.

1. DaemonSet (Node Level)

Pros

Node-level logging is easier to implement because it hooks into the existing file based logging and is less resource intensive than a sidecar approach as there are less containers running per node.

The logs are available via the kubectl command for debugging, as the log files are available to kubelet which returns the content of the log file.

Cons

Less flexibility in supporting different log structures or applications that write to log files instead of streams. You would need to modify the application log structure to achieve parity, or handle the difference in your storage backend.

Since they’re stored as JSON files on the node disk, logs can’t be held forever. You need to have a log rotation mechanism in place to recycle old logs. If you are using Container Runtime Interface, kubelet takes care of rotating the logs and no explicit solution needs to be implemented.

2. Sidecar

Pros

You have the flexibility to customize sidecars per application container. For example, an application might not have the ability to write to stdout/stderr, or it might have some different logging format. In these cases, a sidecar container can bring parity to the system.

If you’re using a logging agent sidecar without streaming, you don't need to rotate the logs because no logs are being stored on the node disk.

Cons

Running a sidecar for each application container is quite resource intensive when compared to node-level pods.

Adding a sidecar to each deployment creates an extra layer of complexity.

If you’re using a streaming sidecar for an application that writes its logs to files, you’ll use double the storage for the same logs because you’ll be duplicating the entries.

If you’re using a logging agent sidecar without streaming, you’ll lose the ability to access logs via kubectl. This is because kubelet no longer has access to the JSON logs.

With a logging agent sidecar you also need a node-level agent, otherwise you won’t be able to collect the system component logs.

Putting Theory into Practice

Now that we’ve looked at the possible patterns for logging in a Kubernetes cluster, let’s put them into action. We’ll deploy dummy containers generating logs and create Kubernetes resources to implement the logging patterns we discussed above.

For this example we’ll use Fluentd as a logging agent, and we will install Elasticsearch for logging backend and Kibana for visualization purposes. We will install Elasticsearch and Kibana using Helm charts into the same cluster. Do note however that your storage backend should not be on the same cluster and we are doing it for demo purposes only. Thanks to Fluentd’s pluggable architecture, it supports various different sinks. That’s why the Elasticsearch backend can be replaced by any cloud-native solution, including Stackdriver or Cloudwatch.

1. Installing Elasticsearch and Kibana

We will deploy the Elasticsearch and Kibana using the official Helm charts which can be found here(Elasticsearch, Kibana). For installing via Helm you would need a helm binary on your path but installation of Helm is outside the scope of this post.

Let us start by adding helm repos.

helm repo add elastic https://helm.elastic.co

Next we will install the Elasticsearch and Kibana charts into our cluster.

helm install elasticsearch elastic/elasticsearch
helm install kibana elastic/kibana

This will install the latest version of Elasticsearch and Kibana in your cluster which can then be used as storage backend for your logs.

We have used the default values in our charts but you can change any parameter based on your needs when you are installing this in production.

2. DaemonSet

We will be deploying Fluentd as a DaemonSet. To keep the verbosity low, we won’t be creating a separate ServiceAccount and ClusterRole. But in a production environment, Fluentd pods should run with a separate service account with limited access.

You can deploy Fluentd as a DaemonSet by using following the Kubernetes resource:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
  labels:
    k8s-app: fluentd-logger
spec:
  template:
    metadata:
      labels:
        k8s-app: fluentd-logger
    spec:
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:elasticsearch
        env:
        - name:  FLUENT\_ELASTICSEARCH\_HOST
          value: "elasticsearch-master"
        - name:  FLUENT\_ELASTICSEARCH\_PORT
          value: "9200"
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: dockerlogs
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: dockerlogs
        hostPath:
          path: /var/lib/docker/containers

In this example, we’re mounting two volumes: one at /var/log and another at /var/log/docker/containers, where the system components and Docker runtime put the logs, respectively.

The image we are using is already configured with smart defaults to be used with DaemonSet, but you can change the configuration.

Save the above YAML resource in a file named fluentd-ds.yaml and apply the resource via the following command:

kubectl apply -f fluentd-ds.yaml

This will start a Fluentd pod on each node in your Kubernetes cluster.

Now we’ll see how to implement streaming and logging agent sidecar patterns.

3. Sidecar

First, let’s look at the streaming sidecar pattern when our application is writing logs to a file instead of stream. We’re running a sidecar to read those logs and write it back to the stdout/stderr stream.

apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  containers:
  - name: legacy-app
    image: busybox
    args:
    - /bin/sh
    - -c
    - >
      i=0;
      while true;
      do
        echo "$i: $(date)" >> /var/log/output.log;
        i=$((i+1));
        sleep 1;
      done      
    volumeMounts:
    - name: varlog
      mountPath: /var/log
  - name: streaming-sidecar
    image: busybox
    args: \[/bin/sh, -c, 'tail -n+1 -f /var/log/output.log'\]
    volumeMounts:
    - name: varlog
      mountPath: /var/log
  volumes:
  - name: varlog
    emptyDir: {}

In this example, we have a dummy container writing logs to files in the /var/log directory of the container. Now these logs can’t be fetched by the container runtime, that’s why we implemented a streaming sidecar to tail the logs from the /var/log location and redirect it to the stdout stream.

This log stream will be picked up by the container runtime and stored as a JSON file at the /var/log directory on the node, which will in turn be picked up by the node-level logging agent.

Now, let’s look at the logging agent sidecar. In this pattern we’ll deploy Fluentd as a sidecar, which will directly write to our Elasticsearch storage backend.

Unfortunately, there is no prebuilt image with an Elasticsearch plugin installed, and creating a custom Docker image is out of the scope of this article. Instead, we’ll use the same Fluentd image that we used in the DaemonSet example.

apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  containers:
  - name: count
    image: busybox
    args:
    - /bin/sh
    - -c
    - >
      i=0;
      while true;
      do
        echo "$i: $(date)" >> /var/log/output.log;
        i=$((i+1));
        sleep 1;
      done      
    volumeMounts:
    - name: varlog
      mountPath: /var/log
  - name: logging-agent
    image: fluent/fluentd-kubernetes-daemonset:elasticsearch
     env:
      - name:  FLUENT\_ELASTICSEARCH\_HOST
        value: "elastisearch-master"
      - name:  FLUENT\_ELASTICSEARCH\_PORT
        value: "9200"
    volumeMounts:
    - name: varlog
      mountPath: /var/log
  volumes:
  - name: varlog
    emptyDir: {}

Conclusion

Given the ephemeral nature of pods and nodes, it’s very important to store logs from your Kubernetes cluster in a separate storage backend. There are multiple patterns that you can use to set up the logging architecture that we discussed in this article.

Note that we suggest a mix of both sidecar and node-level patterns for your production systems. This includes setting up cluster-wide, node-level logging using a DaemonSet pattern, and implementing a streaming sidecar container for applications that do not support writing logs to stream (stdout/stderr) or that don’t write in a standard log format. This streaming container will automatically surface logs for node-level agents to be picked up.

For the choice of storage backend, you can choose self-hosted, open-source solutions such as Elasticsearch, or you can go the managed service route with options like cloud-hosted Elasticsearch, Stackdriver, or Cloudwatch. The choice of backend that’s right for you will depend on the cost, query, and log analysis requirements that you want to implement with your architecture.


Original article sourced at: https://codersociety.com

#kubernetes 

What is GEEK

Buddha Community

Explore Kubernetes Logging in Production
Christa  Stehr

Christa Stehr

1602964260

50+ Useful Kubernetes Tools for 2020 - Part 2

Introduction

Last year, we provided a list of Kubernetes tools that proved so popular we have decided to curate another list of some useful additions for working with the platform—among which are many tools that we personally use here at Caylent. Check out the original tools list here in case you missed it.

According to a recent survey done by Stackrox, the dominance Kubernetes enjoys in the market continues to be reinforced, with 86% of respondents using it for container orchestration.

(State of Kubernetes and Container Security, 2020)

And as you can see below, more and more companies are jumping into containerization for their apps. If you’re among them, here are some tools to aid you going forward as Kubernetes continues its rapid growth.

(State of Kubernetes and Container Security, 2020)

#blog #tools #amazon elastic kubernetes service #application security #aws kms #botkube #caylent #cli #container monitoring #container orchestration tools #container security #containers #continuous delivery #continuous deployment #continuous integration #contour #developers #development #developments #draft #eksctl #firewall #gcp #github #harbor #helm #helm charts #helm-2to3 #helm-aws-secret-plugin #helm-docs #helm-operator-get-started #helm-secrets #iam #json #k-rail #k3s #k3sup #k8s #keel.sh #keycloak #kiali #kiam #klum #knative #krew #ksniff #kube #kube-prod-runtime #kube-ps1 #kube-scan #kube-state-metrics #kube2iam #kubeapps #kubebuilder #kubeconfig #kubectl #kubectl-aws-secrets #kubefwd #kubernetes #kubernetes command line tool #kubernetes configuration #kubernetes deployment #kubernetes in development #kubernetes in production #kubernetes ingress #kubernetes interfaces #kubernetes monitoring #kubernetes networking #kubernetes observability #kubernetes plugins #kubernetes secrets #kubernetes security #kubernetes security best practices #kubernetes security vendors #kubernetes service discovery #kubernetic #kubesec #kubeterminal #kubeval #kudo #kuma #microsoft azure key vault #mozilla sops #octant #octarine #open source #palo alto kubernetes security #permission-manager #pgp #rafay #rakess #rancher #rook #secrets operations #serverless function #service mesh #shell-operator #snyk #snyk container #sonobuoy #strongdm #tcpdump #tenkai #testing #tigera #tilt #vert.x #wireshark #yaml

Maud  Rosenbaum

Maud Rosenbaum

1601051854

Kubernetes in the Cloud: Strategies for Effective Multi Cloud Implementations

Kubernetes is a highly popular container orchestration platform. Multi cloud is a strategy that leverages cloud resources from multiple vendors. Multi cloud strategies have become popular because they help prevent vendor lock-in and enable you to leverage a wide variety of cloud resources. However, multi cloud ecosystems are notoriously difficult to configure and maintain.

This article explains how you can leverage Kubernetes to reduce multi cloud complexities and improve stability, scalability, and velocity.

Kubernetes: Your Multi Cloud Strategy

Maintaining standardized application deployments becomes more challenging as your number of applications and the technologies they are based on increase. As environments, operating systems, and dependencies differ, management and operations require more effort and extensive documentation.

In the past, teams tried to get around these difficulties by creating isolated projects in the data center. Each project, including its configurations and requirements were managed independently. This required accurately predicting performance and the number of users before deployment and taking down applications to update operating systems or applications. There were many chances for error.

Kubernetes can provide an alternative to the old method, enabling teams to deploy applications independent of the environment in containers. This eliminates the need to create resource partitions and enables teams to operate infrastructure as a unified whole.

In particular, Kubernetes makes it easier to deploy a multi cloud strategy since it enables you to abstract away service differences. With Kubernetes deployments you can work from a consistent platform and optimize services and applications according to your business needs.

The Compelling Attributes of Multi Cloud Kubernetes

Multi cloud Kubernetes can provide multiple benefits beyond a single cloud deployment. Below are some of the most notable advantages.

Stability

In addition to the built-in scalability, fault tolerance, and auto-healing features of Kubernetes, multi cloud deployments can provide service redundancy. For example, you can mirror applications or split microservices across vendors. This reduces the risk of a vendor-related outage and enables you to create failovers.

#kubernetes #multicloud-strategy #kubernetes-cluster #kubernetes-top-story #kubernetes-cluster-install #kubernetes-explained #kubernetes-infrastructure #cloud

studio52 dubai

studio52 dubai

1621769539

How to find the best video production company in Dubai?

How to find the best video production company in Dubai?We are the best video production company in Dubai, UAE. We offer Corporate Video, event video, animation video, safety video and timelapse video in most engaging and creative ways.

#video production company #video production dubai #video production services #video production services dubai #video production #video production house

Rusty  Shanahan

Rusty Shanahan

1597516440

Logging Architectures, EFK Stack and Kubernetes

LOGS. When it comes to logging, developers have to go an extra mile by inserting those log lines in their application code and also make sure their infrastructure can handle huge log ingestion to maintain scalability. The benefits? Apart from debugging, logs tell us about the flow of transactions, monitor errors, generate alerts, and do Root Cause Analysis when things go haywire.

In this read, I tried to put forward certain ways on how you can set up a reliable and scalable logging architecture for your application deployed either in VMs or in Kubernetes.

Note: We should not log just for the sake of logging data. Take a proactive approach because it will eventually effect your storage and billing!

Centralized logging

We fetch. We process. We ship.

There are multiple benefits of storing logs in a central location. It reduces overhead on your application servers, you can effectively control and exploit log data and it eliminates the conundrum of tracking logs.

Logging architecture — the EFK stack

Elasticsearch — Real time, distributed, and scalable search engine. Can be used to index and search through volumes of log data.

Fluentd — Acts as a shipper or collector. As the shipper, it forwards log data to Elasticsearch and as a collector, it collects and forwards logs to shipper.

Kibana — Provides visualizations and dashboards for Elasticsearch. Also used to query the Elasticsearch.

#log-management-tool #kubernetes #log-management #logging #devops

Mitchel  Carter

Mitchel Carter

1601305200

Microsoft Announces General Availability Of Bridge To Kubernetes

Recently, Microsoft announced the general availability of Bridge to Kubernetes, formerly known as Local Process with Kubernetes. It is an iterative development tool offered in Visual Studio and VS Code, which allows developers to write, test as well as debug microservice code on their development workstations while consuming dependencies and inheriting the existing configuration from a Kubernetes environment.

Nick Greenfield, Program Manager, Bridge to Kubernetes stated in an official blog post, “Bridge to Kubernetes is expanding support to any Kubernetes. Whether you’re connecting to your development cluster running in the cloud, or to your local Kubernetes cluster, Bridge to Kubernetes is available for your end-to-end debugging scenarios.”

Bridge to Kubernetes provides a number of compelling features. Some of them are mentioned below-

#news #bridge to kubernetes #developer tools #kubernetes #kubernetes platform #kubernetes tools #local process with kubernetes #microsoft