Kubernetes Monitoring and Logging — An Apache Spark Example

We'll learn Kubernetes Monitoring and Logging — An Apache Spark Example. I will use Apache Spark on Kubernetes as an example to share what I use as my monitoring and logging stack. I want to give an overview here, I will have another blog to explain the how-to in details.

I have moved almost all my big data and machine learning projects to Kubernetes and Pure Storage. I am very happy with this move so far. With this platform, my life as a data engineer / data scientist becomes easier — much easier to deploy, scale and manage my Spark jobs, Presto queries, TensorFlow trainings and so on. It is also easier to share the infrastructure resource among these projects.

However, when I set up the environment at the beginning, I found it is a little difficult to understand what is actually going there. I realised I need an end-to-end monitoring setup for my Kubernetes cluster and its applications. In this blog, I will use Apache Spark on Kubernetes as an example to share what I use as my monitoring and logging stack. I want to give an overview here, I will have another blog to explain the how-to in details.

kubernetes

Our original Kubernetes tool list was so popular that we've curated another great list of tools to help you improve your functionality with the platform.

This article explains how you can leverage Kubernetes to reduce multi cloud complexities and improve stability, scalability, and velocity.

Get Hands-on experience on Kubernetes and the best comparison of Kubernetes over the DevOps at your place at Kubernetes training

Get Hands-on experience on Kubernetes and the best comparison of Kubernetes over the DevOps at your place at Kubernetes training

Microsoft announced the general availability of Bridge to Kubernetes, formerly known as Local Process with Kubernetes.