The blog aims at exporting the logs generated in Google Cloud Platform’s (GCP) monitoring and logging tool — Stackdriver to the **ELK stack **on the Elastic Cloud. I will be explaining and designing the complete pipeline from scratch to get the logs to the Elastic Cloud. Before we begin, let us have some of our prerequisites cleared.

What is ELK Stack?

The ELK stack is an acronym used to describe a stack that comprises of three popular open-source projects: Elasticsearch(E), Logstash(L), and Kibana(K).

ElasticSearch: used for storing logs (a database)

Logstash: used for both shipping as well as the processing and storing logs. Kibana: a powerful visualization tool (a web interface) that is hosted through Nginx or Apache.

ELK and their functions

ELK Stack is designed to allow users to take to data from any source, in any format, and to search, analyze, and visualize that data in real-time. ELK stack being an open-source tool, many organizations use it for monitoring and logging purpose. Thus, for organizations that have their ELK stack already being set up, it becomes important to sink the logs from any Cloud-based Monitoring and Logging tool to the ELK stack.

What is Elastic Cloud?

Elastic Cloud is a family of SaaS offerings that make it easy to deploy, operate, and scale ELK stack and other solutions in the cloud. Elastic Stack is quickly overtaking existing proprietary solutions and becoming the first choice for enterprises looking for log analysis and management solutions.

Setting up the ELK Stack is a complicated task that requires a lot of expertise, so many companies these days are instead looking for “hosted Elastic Stack” or “Elastic Stack as a service,” or “cloud-hosted Elastic Stack” solutions whenever possible. Thus, Elastic Cloud becomes the right place for organizations to host their ELK stack without worrying about computing, scalability, log analytics ML tools, and other benefits.

So, with this basic knowledge, let us now implement the end to end pipeline from Stackdriver in GCP to the Kibana dashboard on the Elastic Cloud.

#google-cloud-platform #cloud