Golang: Tracking 1% Request Timeouts with Sumologic Complex Log Parsing

Golang: Tracking 1% Request Timeouts with Sumologic Complex Log Parsing


One primary factor that attracts the end-user is how quickly the application responds back to the user request. Applications perform pretty well when the traffic is low but things start failing once the load increases on the application.

Before taking the application to production, there will be rigorous benchmark testing on what’s the max load that the application server or database server can handle. Based on the expected load and fluctuations, the service sets an SLA Timeout(Service Level Agreement) saying if processing any request takes more than this SLA time, the upstream service has to cancel the request with a DeadLine exceeded error.

Problem Statement

The manual benchmarking is just a preliminary check to estimate the capacity of the system but we developers can’t solely depend on the benchmark result. Hence to monitor real-time traffic information, we started using application monitoring tools like NewRelic.

NewRelic is the best in the market in terms of real-time application monitoring, insights, alerts, etc. But what I felt is it lacked in one thing.

Image for post

The above picture depicts a GRPC Service with the average time taken for each sub-component. We identity the pain points of a service and start a transaction for each of the pain points so we can track them.

Image for post

NewRelic also provides us with %(percentile) performance. Based on this, we can say the Median is below 50 ms, 95 % of requests are getting served in less than 500 ms and 4% of the requests are taking almost 2 seconds. Assume my service SLA is 250 ms. If we observe carefully, there are a few major things that are missing.

monitoring alerts golang

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

How to Monitor Third Party API Integrations

How to best monitor your external and third party API integrations and hold partners accountable to SLAs

How to monitor panics with Raygun for Golang

Today I’m going to show you how to use our new official Raygun4Go provider created by Jakob Anders and Christoph Rahles from Käuferportal Raygun is an error reporting service for logging errors in your applications and providing you with all the...

Golang Web Development:Th Best Programming Language in 2020

Golang Web Development is the new trend globally. Get all the details about the Golang development here. Hire Golang Developers.

Golang Array Example | Arrays in Go Tutorial Explained

Golang array is a fixed-size collection of items of the same type. The items of an array are stored sequentially and can be accessed using their index.

Road to Golang pro — Pointer & Functions

A few words before we start. You can find the code used in this tutorial in this repository. You can find the full contents of Road to Go Pro here. If you missed the last one, you can find it via this link.