Golang: Tracking 1% Request Timeouts with Sumologic Complex Log Parsing

Introduction

One primary factor that attracts the end-user is how quickly the application responds back to the user request. Applications perform pretty well when the traffic is low but things start failing once the load increases on the application.

Before taking the application to production, there will be rigorous benchmark testing on what’s the max load that the application server or database server can handle. Based on the expected load and fluctuations, the service sets an SLA Timeout(Service Level Agreement) saying if processing any request takes more than this SLA time, the upstream service has to cancel the request with a DeadLine exceeded error.

Problem Statement

The manual benchmarking is just a preliminary check to estimate the capacity of the system but we developers can’t solely depend on the benchmark result. Hence to monitor real-time traffic information, we started using application monitoring tools like NewRelic.

NewRelic is the best in the market in terms of real-time application monitoring, insights, alerts, etc. But what I felt is it lacked in one thing.

The above picture depicts a GRPC Service with the average time taken for each sub-component. We identity the pain points of a service and start a transaction for each of the pain points so we can track them.

NewRelic also provides us with %(percentile) performance. Based on this, we can say the Median is below 50 ms, 95 % of requests are getting served in less than 500 ms and 4% of the requests are taking almost 2 seconds. Assume my service SLA is 250 ms. If we observe carefully, there are a few major things that are missing.

monitoring alerts golang

How to best monitor your external and third party API integrations and hold partners accountable to SLAs

Today I’m going to show you how to use our new official Raygun4Go provider created by Jakob Anders and Christoph Rahles from Käuferportal Raygun is an error reporting service for logging errors in your applications and providing you with all the...

Golang Web Development is the new trend globally. Get all the details about the Golang development here. Hire Golang Developers.

Golang array is a fixed-size collection of items of the same type. The items of an array are stored sequentially and can be accessed using their index.

A few words before we start. You can find the code used in this tutorial in this repository. You can find the full contents of Road to Go Pro here. If you missed the last one, you can find it via this link.