For those who do not know — Grafana is a tool to visualize your metrics/logs from multiple data sources in easy way by dashboards. For more info about its capabilities read here. Creating dashboards is easy but monotonic and boring in the long term when you have to manually click every setting for every graph. We are going to make this process less schematic.
Grafana dashboards are provided in json format. Most of us create dashboards and then paste it into a folder where we keep our dashboards or into configMaps if you are using prometheus-operator like me.
This copy-paste process of developing dashboard makes each new dashboard request unwanted. You can often make a mistake trying to manually change some value in json. How to make this process more engaging? Less painful and less “jsonful”? Add more clarity to versioning?
For purposes of the article, we will work with prometheus datasource and prometheus-operator. You can suit the presented solution to your monitoring stack with small adjustments.
Grafanalib is an open-source Python library in which we code dashboards. Coded dashboards are then generated in ready to deploy json format. Let’s start with installation:
pip install grafanalib
Get a basic dashboard written in python with:
curl -o example-core.dashboard.py \
https://gist.github.com/kamilswiec/bd170b3724819fd14531b4bf36fee8da
It is important now to check two places in example-core.dashboard.py
:
from grafanalib.core import (
Dashboard, Graph,
OPS_FORMAT, Row,
single_y_axis, Target, TimeRange, YAxes, YAxis
)
dashboard = Dashboard(
title="Python generated dashboard",
rows=[
Row(panels=[
Graph(
title="Prometheus http requests",
dataSource='default',
targets=[
Target(
expr='rate(prometheus_http_requests_total[5m])',
legendFormat="{{ handler }}",
refId='A',
),
],
yAxes=single_y_axis(format=OPS_FORMAT),
),
]),
],
).auto_panel_ids()
Adjust fields if needed. Then use binary which comes with grafanalib to create json:
generate-dashboard -o test.json example-core.dashboard.py
Now you have a simple dashboard! I use prometheus-operator so I need to do extra steps. I create configMap containing json with specified labels and namespace, and save it to yaml file. Saving to yaml instead of pushing into a cluster is important because we use GitOps to continuous deployment. The problem here is how to add labels with create command? Luckily it is possible with this command:
kubectl create cm test -n prometheus-operator --from-file=test.json -o yaml --dry-run | \
kubectl label -f- --dry-run -o yaml --local grafana_dashboard="1" > test.yaml
Flag --local
allows adding labels to conifgMap without need to communicate with API. After committing changes to Github new dashboard is ready to use!
I thought that there is potential to make this less tight. Create functions which will return Graphs for example. My thoughts resulted in code:
from grafanalib.core import *
def singleQueryGraph(name, exp, legend):
return Graph(
title=name,
targets=[
Target(
expr=exp,
legendFormat=legend,
refId='A',
),
],
)
from grafanalib.core import *
from graphtools import *
rows_code = [Row(panels=[
singleQueryGraph(
"1-graph",
"rate(prometheus_http_requests_total[5m])",
"{{ handler }}"
),
singleQueryGraph(
"2-graph",
"rate(prometheus_http_requests_total[5m])",
"{{ handler }}"
)
],
)
]
dashboard = Dashboard(
version=2,
tags=["Grafanalib"],
title="Test with two graphs",
uid="test",
rows=rows_code,
).auto_panel_ids()
print(dashboard.to_json_data())
Simple idea. Last print
is because generate-dashboard
does not allow to add multiple files without dashboard in them. However, I was disappointed — output form to_json_data()
returned invalid json:
I tried to validate and format output but there were so many errors I gave up. The only way to get correct json is to use generate-dashboard
binary. So we are back to place when we can use only one file. If we put the function inside — then it will work — but produce a lot of redundant code with each new dashboard (unless you keep everything in one file). Another solution can be joining to grafanalib project and finding a way to fix the problem which I found.
After all dashboard definition without functions is quite good. You still have better readability than raw json. Grafanalib comes with readthedocs.io page so it is quite easy to find everything you need. Try it out, maybe you will like it.
To generate valid json use functions provided in grafanalib._gen
. Example:
from grafanalib._gen import print_dashboard
...
print_dashboard(dashboard)
This will produce valid json file. Credits goes to Povilas Balciunas , he mentioned this in comments. Thanks!
Originally published by Kamil Świechowski at Medium
#prometheus #kubernetes #programming #grafana #python