How to set up a logging Infrastructureusing using ElasticSearch, Fluentd, and Kibana.Logging is one of those Node.js functions that's easy to take for ... and efficiently troubleshoot infrastructure and application issues. Logging Best Practices. There are a number of best practices to follow when setting up your .
Setting up the right logging infrastructure helps us in finding what happened, debugging and monitoring the application. At a very basic level, we should expect the following from our infrastructure:
statusCodeof all the APIs
Architecture Digram using ElasticSearch, Fluentd and Kibana
We would be using Docker for managing our services.Elastic Search
Let’s get the ElasticSearch up and running with the following command
docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --name myES docker.elastic.co/elasticsearch/elasticsearch:7.4.1
We can check if our container is up and running by the following command
curl -X GET "localhost:9200/_cat/nodes?v&pretty"
We can get our Kibana up and running with another docker run command.
docker run —-link myES:elasticsearch -p 5601:5601 kibana:7.4.1
Note that we are linking our kibana and elastic search server using the
If we go to
[http://localhost:5601/app/kibana](http://localhost:5601/app/kibana) , we would be able to see our kibana dashboard.
We can now run all the queries against our elastic search cluster using kibana. We can navigate to
and run the query we ran before (just a little less verbose)
query for elastic cluster nodes using kibanaFluentd
Fluentd is the place where all the data formatting will happen.
Let’s first build our Dockerfile. It does two things:
FROM fluent/fluentd:latest MAINTAINER Abhinav Dhasmana <[email protected]> USER root RUN apk add --no-cache --update --virtual .build-deps \ sudo build-base ruby-dev \ && sudo gem install fluent-plugin-elasticsearch \ && sudo gem install fluent-plugin-record-modifier \ && sudo gem install fluent-plugin-concat \ && sudo gem install fluent-plugin-multi-format-parser \ && sudo gem sources --clear-all \ && apk del .build-deps \ && rm -rf /home/fluent/.gem/ruby/2.5.0/cache/*.gem COPY fluent.conf /fluentd/etc/
# Recieve events over http from port 9880 <source> @type http port 9880 bind 0.0.0.0 </source> # Recieve events from 24224/tcp <source> @type forward port 24224 bind 0.0.0.0 </source> # We need to massage the data before if goes into the ES <filter **> # We parse the input with key "log" (https://docs.fluentd.org/filter/parser) @type parser key_name log # Keep the original key value pair in the result reserve_data true <parse> # Use apache2 parser plugin to parse the data @type multi_format <pattern> format apache2 </pattern> <pattern> format json time_key timestamp </pattern> <pattern> format none </pattern> </parse> </filter> # Fluentd will decide what to do here if the event is matched # In our case, we want all the data to be matched hence ** <match **> # We want all the data to be copied to elasticsearch using inbuilt # copy output plugin https://docs.fluentd.org/output/copy @type copy <store> # We want to store our data to elastic search using out_elasticsearch plugin # https://docs.fluentd.org/output/elasticsearch. See Dockerfile for installation @type elasticsearch time_key timestamp_ms host 0.0.0.0 port 9200 # Use conventional index name format (logstash-%Y.%m.%d) logstash_format true # We will use this when kibana reads logs from ES logstash_prefix fluentd logstash_dateformat %Y-%m-%d flush_interval 1s reload_connections false reconnect_on_error true reload_on_failure true </store> </match>
Config file for the fluent
Let’s spin this docker machine up
docker build -t abhinavdhasmana/fluentd . docker run -p 9880:9880 --network host abhinavdhasmana/fluentd
I have created a small Node.js app for demo purposes which you can find here. It’s a small express app created using express generator. It is using morgan to generate logs in the apache format. You can use your own app in your preferred language. As long as the output remains the same, our infrastructure does not care. Let’s build our docker image and run it.
docker build -t abhinavdhasmana/logging .
Of course we can get all the docker containers up by a single docker compose file given below
version: "3" services: fluentd: build: "./fluentd" ports: - "9880:9880" - "24224:24224" network_mode: "host" web: build: . ports: - "3000:3000" links: - fluentd logging: driver: "fluentd" options: fluentd-address: localhost:24224 elasticsearch: image: elasticsearch:7.4.1 ports: - "9200:9200" - "9300:9300" environment: - discovery.type=single-node kibana: image: kibana:7.4.1 links: - "elasticsearch" ports: - "5601:5601"
docker compose file for the EFK setup
That’s it. Our infrastructure is ready. Now we can generate some logs by going to
We now go to kibana dashboard again and define the index to use
setting up index for use in kibana
Note that in our
fluent.conf , we mentioned
logstash_prefix fluentd and hence we use the same string here. Next are some basic kibana settings
kibana configure settings
Elastic Search using dynamic mapping to guess the
type of the fields that it indexes. The below snapshot shows these
Mapping example of Elastic Search
Let’s check on how are we doing with the requirements we mentioned at the start:
section on the left of the kibana, we can see a fieldpath` . We can apply the filter on this to look for APIs that we are interested in.
**statusCode**of all the APIs: Same as above. Use
codefield and apply filter.
discovery.type=single-node. We can start in a cluster mode, add more nodes or use a hosted solution on any cloud provider of our choice. I have tried AWS and its easy to set it up. AWS also give managed kibana instance for Elasticsearch at no extra cost.
Node.js for Beginners - Learn Node.js from Scratch (Step by Step) - Learn the basics of Node.js. This Node.js tutorial will guide you step by step so that you will learn basics and theory of every part. Learn to use Node.js like a professional. You’ll learn: Basic Of Node, Modules, NPM In Node, Event, Email, Uploading File, Advance Of Node.Node.js for Beginners
Welcome to my course "Node.js for Beginners - Learn Node.js from Scratch". This course will guide you step by step so that you will learn basics and theory of every part. This course contain hands on example so that you can understand coding in Node.js better. If you have no previous knowledge or experience in Node.js, you will like that the course begins with Node.js basics. otherwise if you have few experience in programming in Node.js, this course can help you learn some new information . This course contain hands on practical examples without neglecting theory and basics. Learn to use Node.js like a professional. This comprehensive course will allow to work on the real world as an expert!
What you’ll learn:
Node is fast, scalable, and easy to get started with. Its default package manager is npm, which means it also sports the largest ecosystem of open-source libraries. Node is used by companies such as NASA, Uber, Netflix, and Walmart.
But Node doesn't come alone. It comes with a plethora of frameworks. A Node framework can be pictured as the external scaffolding that you can build your app in. These frameworks are built on top of Node and extend the technology's functionality, mostly by making apps easier to prototype and develop, while also making them faster and more scalable.
Below are 7of the most popular Node frameworks at this point in time (ranked from high to low by GitHub stars).Express
With over 43,000 GitHub stars, Express is the most popular Node framework. It brands itself as a fast, unopinionated, and minimalist framework. Express acts as middleware: it helps set up and configure routes to send and receive requests between the front-end and the database of an app.
Express provides lightweight, powerful tools for HTTP servers. It's a great framework for single-page apps, websites, hybrids, or public HTTP APIs. It supports over fourteen different template engines, so developers aren't forced into any specific ORM.Meteor
The project has over 41,000 GitHub stars and is built to power large projects. Meteor is used by companies such as Mazda, Honeywell, Qualcomm, and IKEA. It has excellent documentation and a strong community behind it.Koa
Koa is built by the same team that built Express. It uses ES6 methods that allow developers to work without callbacks. Developers also have more control over error-handling. Koa has no middleware within its core, which means that developers have more control over configuration, but which means that traditional Node middleware (e.g. req, res, next) won't work with Koa.
Koa already has over 26,000 GitHub stars. The Express developers built Koa because they wanted a lighter framework that was more expressive and more robust than Express. You can find out more about the differences between Koa and Express here.Sails
Sails is a real-time, MVC framework for Node that's built on Express. It supports auto-generated REST APIs and comes with an easy WebSocket integration.
The project has over 20,000 stars on GitHub and is compatible with almost all databases (MySQL, MongoDB, PostgreSQL, Redis). It's also compatible with most front-end technologies (Angular, iOS, Android, React, and even Windows Phone).Nest
Nest is packaged in such a way it serves as a complete development kit for writing enterprise-level apps. The framework uses Express, but is compatible with a wide range of other libraries.LoopBack
LoopBack is a framework that allows developers to quickly create REST APIs. It has an easy-to-use CLI wizard and allows developers to create models either on their schema or dynamically. It also has a built-in API explorer.
LoopBack has over 12,000 GitHub stars and is used by companies such as GoDaddy, Symantec, and the Bank of America. It's compatible with many REST services and a wide variety of databases (MongoDB, Oracle, MySQL, PostgreSQL).Hapi
Similar to Express, hapi serves data by intermediating between server-side and client-side. As such, it's can serve as a substitute for Express. Hapi allows developers to focus on writing reusable app logic in a modular and prescriptive fashion.
The project has over 11,000 GitHub stars. It has built-in support for input validation, caching, authentication, and more. Hapi was originally developed to handle all of Walmart's mobile traffic during Black Friday.
2:32 What is Node.js?
3:22 Client-Server Architecture
4:12 Multi-Threaded Model
6:13 Single-Threaded Model
7:43 Multi-Threaded vs Event-Driven
9:45 Uber Old Architecture
11:10 Uber New Architecture
12:30 What is Node.js?
13:05 Sucess Stories
14:20 Node.js Trend
14:40 Node.js Features
16:25 Node.js Installation
16:50 Node.js First Example
17:30 Blocking vs Non-blocking
23:50 Node.js Modules
25:10 Global Objects
26:55 File System
34:50 Hands On
1:09:45 Node.js Tutorial
1:10:45 What is Node.js?
1:12:10 Features of Node.js
1:13:00 Node.js Architecture
1:14:55 NPM(Node Package Manager)
1:16:20 Node.js Modules
1:16:30 Node.js Modules Types
1:16:35 Core Modules
1:16:55 Local Modules
1:17:10 3rd Party Modules
1:18:35 JSON File
1:23:30 Data Types
1:29:55 File Systems
1:34:20 HTTP Module
1:44:37 HTTP Module
1:45:27 Creating a Web Server using Node.js
1:58:37 Node.js NPM Tutorial
1:59:37 What is NPM?
2:03:12 Main Functions of NPM
2:04:27 Need For NPM
2:08:07 NPM Packages
2:17:42 NPM Installation
2:18:12 JSON File
2:31:32 Node.js Express Tutorial
2:32:02 Introduction to Express.js
2:32:32 Features of Express.js
2:35:27 Getting Started with Express.js
2:39:42 Routing Methods
2:48:12 Building RESTful API with Node.js
2:48:27 What is REST API?
2:49:42 Features of REST API
2:51:12 Principles of REST API
2:56:37 Methods of REST API
2:59:52 Building REST API with Node.js
3:24:07 Node.js MySQL Tutorial
3:24:32 What is MySQL?
3:25:13 Advantages of Using MySQL with Node.js
3:27:38 MySQL Installation
3:44:23 Node.js MongoDB Tutorial
3:44:58 What is NoSQL?
3:47:53 NoSQL Databases
3:48:38 Introduction to MongoDB
3:52:48 Features of MongoDB
3:53:03 MongoDB Installation
4:36:08 Node.js Docker Tutorial
4:36:38 What is Docker?
4:39:13 Docker Working
4:41:43 Docker Basics
4:42:03 Docker Images
4:42:23 Docker Container
4:44:38 Why use Node.js with Docker?
4:45:18 Demo: Node.js with Docker
4:58:38 MEAN Stack Application Tutorial
4:59:18 What is MEAN Application?
5:02:17 RESTful API
5:03:02 Contact List MEAN App
6:17:57 Node.js Interview Questions
Got a question on the topic? Please share it in the comment section below and our experts will answer it for you.