This article is a project presentation of a viewer for log files.

The project is located on GitHub: https://github.com/sevdokimov/log-viewer

Introduction

There are many tools in the world for analysis logs, but most of my colleagues used the simple text editor or “less” command in the terminal. Log analysis tools can be divided into two groups:

1. Applications working on the local machine. They load log files from remote servers using SFTP or SSH.
2. Log aggregators. A log aggregator is a server-based application that collects logs from other applications and stores them to an index. The index allows quick navigation and search.

The main disadvantage of #1 is the time of downloading logs from a remote machine. If the log weighs about 1G, it’s not usable to download it.

Log aggregator is the right solution for serious production environments, but they require additional resources for storing index and additional configuration to collect logs. In some cases, using a log aggregator is overkill.

I got an idea of how to make a log viewer that has some advantages of log aggregators, but actually, it’s a pure viewer. It doesn’t require additional storage for an index, doesn’t download log files to the local machine, but allows viewing logs on remote servers with nice features like filtering, search, merging events from several log files to one view.

The idea is to run a tiny Web UI on a server that provides access to log files located on the server. LogViewer doesn’t load an entire file to the memory. It loads only the part that the user is watching. This approach allows displaying huge log files without significant memory consumption. If we need monitoring logs on more than one node, LogViewer must be run on all nodes. Each LogViewer instance can connect to other instances and request data located on remote nodes. So each LogViewer can show events from the entire cluster merged into one view by date.

It’s important to make the tool as easy to use as possible. That’s why LogViewer can detect log format automatically, there is no required configuration, and log representation is close to a text editor representation.

#spring boot #log analytics #debugging tools #log viewer #a new tool for monitoring logs