Fluentd — Splitting Logs

In most kubernetes deployments we have applications logging into stdout different type of logs. A good example are application logs and access logs, both have very important information, but we have to parse them differently, to do that we could use the power of fluentd and some of its plugins.

In this hands-on post we are going to explain how to split those logs into parallel streams so you can process them further. If you are new to fluentd, you should checkout this post, where we explore the basics of it.

Setup

To help you with the setup, I’ve created this repo, after cloning it you will end up with the following directory structure:

fluentd/
    ├── etc/
    │   └── fluentd.conf
    ├── log/
    │   └── kong.log
    └── output/

In output/ is where fluentd is going to write the files.

In log/kong.log we have some logs from a kong container that I have running in my laptop. Take a look at these logs, they have the docker format:

{
    "log":"2019/07/31 22:19:52 [notice] 1#0: start worker process 32\n",
    "stream":"stderr",
    "time":"2019-07-31T22:19:52.3754634Z"
}

In etc/fluentd.conf is our fluentd configuration, take a look at it, you can see that there’s an input and an output section, we will be takin a closer look to it later, first let’s run the fluentd container:

Running fluentd

docker run -ti --rm \
-v $(pwd)/etc:/fluentd/etc \
-v $(pwd)/log:/var/log/ \
-v $(pwd)/output:/output \
fluent/fluentd:v1.10-debian-1 -c /fluentd/etc/fluentd.conf -v

Pay attention to that run command and the volumes we are mounting:

• etc/ is mounted onto the /fluentd/etc/ directory inside of the container to override fluentd default config.
• log/ onto /var/log/ ending up with /var/log/kong.log inside of the container.
• output/ onto /output to be able to see what fluentd writes to disk.

After running the container you should see a line like:

2020-05-10 17:33:36 +0000 [info]: #0 fluent/log.rb:327:info: fluentd worker is now running worker=0

This means that fluentd is up and running.

#logging #geoip #kubernetes #fluentd

What is GEEK

Buddha Community

Fluentd — Splitting Logs

In most kubernetes deployments we have applications logging into stdout different type of logs. A good example are application logs and access logs, both have very important information, but we have to parse them differently, to do that we could use the power of fluentd and some of its plugins.

In this hands-on post we are going to explain how to split those logs into parallel streams so you can process them further. If you are new to fluentd, you should checkout this post, where we explore the basics of it.

Setup

To help you with the setup, I’ve created this repo, after cloning it you will end up with the following directory structure:

fluentd/
    ├── etc/
    │   └── fluentd.conf
    ├── log/
    │   └── kong.log
    └── output/

In output/ is where fluentd is going to write the files.

In log/kong.log we have some logs from a kong container that I have running in my laptop. Take a look at these logs, they have the docker format:

{
    "log":"2019/07/31 22:19:52 [notice] 1#0: start worker process 32\n",
    "stream":"stderr",
    "time":"2019-07-31T22:19:52.3754634Z"
}

In etc/fluentd.conf is our fluentd configuration, take a look at it, you can see that there’s an input and an output section, we will be takin a closer look to it later, first let’s run the fluentd container:

Running fluentd

docker run -ti --rm \
-v $(pwd)/etc:/fluentd/etc \
-v $(pwd)/log:/var/log/ \
-v $(pwd)/output:/output \
fluent/fluentd:v1.10-debian-1 -c /fluentd/etc/fluentd.conf -v

Pay attention to that run command and the volumes we are mounting:

• etc/ is mounted onto the /fluentd/etc/ directory inside of the container to override fluentd default config.
• log/ onto /var/log/ ending up with /var/log/kong.log inside of the container.
• output/ onto /output to be able to see what fluentd writes to disk.

After running the container you should see a line like:

2020-05-10 17:33:36 +0000 [info]: #0 fluent/log.rb:327:info: fluentd worker is now running worker=0

This means that fluentd is up and running.

#logging #geoip #kubernetes #fluentd

Using Azure Log Analytics Workspaces to collect Custom Logs from your VM

Motivation:

We all have seen the Monitoring Tab Key Metrics on the VM Page. Yes, it is useful to see whether the CPU has been running or not through the **CPU **metrics, to check when the VM is getting the data from the outside world through Network In metrics and if the VM is doing any kind of write operation using **Disk Operations/Sec **metrics but it’s not effective for the custom services that we build on VM. So, in this blog, I will be giving you an example of how to create your own log based on customer service, bring it to Azure Log Analytics Workspace using its default agent, and query it according to our needs, even better create an alert on it.

Prerequisites:

1. An Azure Account
2. Azure Virtual Machine Service
3. Azure Log Analytics Workspace Service
4. Azure Alert Service

Azure Virtual Machine:

Let’s start with the VM itself. You already have the service running on the VM but don’t know how to get those logs into the portal or even create a log for your services. So, let’s assume that you have not created the logs for your services. So for this blog, I will be taking a simple flask app as my example for the service. To get it up and running we must download the flask library from pip and then create a flask app similar to this below:

#logs #log-analytics #azure #azure log analytics

How to Install Logstash | Liquid Web

Logstash is a free, open source, server-side data collection and processing engine which uses dynamic pipelining capabilities. It takes in information from multiple data sources, reconstructs it, and then sends it to a destination of our choice. Logstash also cleans and modifies the data for use in an advanced outbound analytics and visualization use cases. It is the L in the ELK stack (Elasticsearch, Logstash and Kibana) and is typically responsible for sending data to Elasticsearch.

Effectively administrating Linux systems requires familiarity with the locations of and possible contents of logs. Application and system-level logs provide insight into an application’s behavior or the system that might not be apparent on the surface. When many cooperating systems each provide their logs, aggregating, and processing that information becomes imperative. This is where Logstash shines.

#tutorials #apt transport https #data filters #data pipeline #data processing #elasticsearch #elk #event pipeline #java #java jdk #kibana #log collector #log collectors #log files #logs #logstash #ubuntu #ubuntu 18.04

To Log, or Not to Log, That is The Question.

Logging is one of the those things are universally present in software projects with their different unique forms, requirements, and flavors.

Logging is everywhere, from small 1-person-startups to large enterprises. Even a simple algorithmic programming question involves some logging along the way.

We rely so much on logging to develop, maintain, and keep our programs up and running. However, not much attention has been paid to how to design logging within our systems.

Often times, logging is treated as a second thought – it’s only sprinkled into source code upon implementation like some magic powder that helps lighten the day-to-day operational abyss in our systems.

#to log #not to log,

Logging Architectures, EFK Stack and Kubernetes

LOGS. When it comes to logging, developers have to go an extra mile by inserting those log lines in their application code and also make sure their infrastructure can handle huge log ingestion to maintain scalability. The benefits? Apart from debugging, logs tell us about the flow of transactions, monitor errors, generate alerts, and do Root Cause Analysis when things go haywire.

In this read, I tried to put forward certain ways on how you can set up a reliable and scalable logging architecture for your application deployed either in VMs or in Kubernetes.

Note: We should not log just for the sake of logging data. Take a proactive approach because it will eventually effect your storage and billing!

Centralized logging

We fetch. We process. We ship.

There are multiple benefits of storing logs in a central location. It reduces overhead on your application servers, you can effectively control and exploit log data and it eliminates the conundrum of tracking logs.

Logging architecture — the EFK stack

Elasticsearch — Real time, distributed, and scalable search engine. Can be used to index and search through volumes of log data.

Fluentd — Acts as a shipper or collector. As the shipper, it forwards log data to Elasticsearch and as a collector, it collects and forwards logs to shipper.

Kibana — Provides visualizations and dashboards for Elasticsearch. Also used to query the Elasticsearch.

#log-management-tool #kubernetes #log-management #logging #devops