A Journey to Dashboards as Code
Understanding the state of your infrastructure and systems is essential for ensuring the reliability and stability of services. The best way to gain this insight is with a robust dashboard that visualizes data and stable alert rules which alerts when things appear to be broken.
Configuring and modifying dashboards and alerts by hand is error-prone. Versioning dashboards and alerts using “Infrastructure as code” are extremely useful in a fast-paced environment.
Why should we even think of moving to Dashboards as Code?
Creating Dashboards is the simplest thing that can be done, why do we need to complicate it?
Many of us think that alerts are more effective than Dashabords, but still, we love to stare at the dashboard to see the historic data to observe trends and indicators which give insights to make educated decisions.
Grafana, Kibana, Wavefront, and so many dashboards solutions are available in the market now. With awesome UI it makes tasks a lot easier. The ease of creating/modifying dashboards has become very easy that even a non-expert can edit the dashboard and make unindented changes.
This has made maintaining dashboards a real challenge. Increased automation leads to lack of accountability.
As of today, dashboards are created/updated by clicking on the UI adding/ updating 100’s of graphs/charts manually without a peer review, affecting productivity, introducing conflicts, inconsistency and no history of untraceable changes which cannot be easily rolled back.
What could be an ideal scenario?
For large applications, no one will question a requirement to have a version control system/code. Why should one not require the same kind of rigor for dashboards?
We realized that creating a process along with a source control system will be a solution to this problem. We wanted to consider dashboards as another production resource. Best way to achieve this was to move to “Dashboards as Code”.
Once we move the dashboards to code, its then a software. Once its a software, then we can make use of the Software Engineering principles and best practices that we have learned from the past.
There are different ways to solve this.
JSON in Git
Most of the modern dashboards represent the graphs, charts, etc as JSON data in the backend.
The easiest solution is to store the JSON in a Git repo. But, have you seen the JSON data behind those beautiful dashboards?? They are real bloaty.
Don’t even think of managing them. A small change in the dashboard will move the entire JSON data by miles. Which makes it impossible to manage and peer review.
How can we squeeze the elephant into the car trunk?
JSON Templating Language and Git
Arguably, reducing the boilerplate in JSON should be the foremost task. Modern templating languages can be used to achieve this.
JSONNET is a very powerful templating language. JSONNET can reduce a lot of boilerplate code blocks inside the JSON data using loops and variables.
But still, someone has to manually push those JSON data generated by JSONNET to git, and then after any changes, they should be deployed to the Dashboard using some automation.
Solve the problem without hurting the future!
At Intuit, we use terraform extensively, hence terraform was the first obvious choice.
With Dynamic content, rich type, and loops in terraform v12, terraform can manage and provision dashboards very neatly. There are already terraform providers for dashboards available in Github.
Write terraform HCL code to create dashboards and Graphs, specify corresponding metric queries, and run “terraform apply”, terraform will create the dashboards with the graphs.
NB: Terraform implementation of “Dashboards as code” is a separate series of blogs. You can find the link towards the end of this post.
Show me the code!
Below Terraform HCL code snippet creates a Wavefront dashboard with multiple graphs.
Github Repo
Creating Wavefront Dashboards with Terraform 0.12 can be found here.
Wrapping up
Terraform is generally used for Infrastructure as Code, but it worked well for Dashboards as well.
Just like any other terraform resource, you can create, modify, and destroy the dashboards using terraform.
Terraform is not the only solution in the market. Few other mature options are mentioned below.
- Grafana-Lib (JSONNET based solution)
- Grafana python module — from Weave works
- Grafana dashboard generator — from Uber
Learnings
In the beginning, we often end up spending more time writing IaC than for the dashboard, but later we created reusable templates. Which also gave us more consistency over our dashboards.
Creating and maintaining dashboards became a collaborative task. Reviewing changes in the dashboard became meaningful with visual differentiation. Creating dashboards became faster and easy.
Further reading…
Creating Wavefront dashboards with Terraform.
