Grafana API Quirks, Oddities and the K8s future
Grafana's API evolution! If you're building tooling against Grafana, the auto-discoverable /apis endpoint changes the game. Wrote up the whole story + working Go code below👇
Intro to Grafana
If you are the one person left in this world that is both interested in this topic and is unaware what Grafana is, let's do an honorable mention and introduction.
it's a very cool, open source platform that lets you visualize virtually any type of data you want. First, You'll need to connect your datasources to Grafana. These can vary from databases, raw files, project management tools, version control system. At this point it would not even surprise me if they had a fax machine integration. You then create a dashboard, or a graph by querying the data from your connection to create both visually pleasing representation and useful graphs that help you debug an issue.
I'll leave you all to explore this further at your own discretion, if I had to cover every feature that Grafana has we'd be here all day. They also have an excellent marketing department and it would be far better to let everyone explore the fruit of their labor.
APIs Intro
This is going to be a one liner intro. If you're not familiar with what an API is, this blog will likely not be a good read, but here are a few good primers:
API 101
- Intro to Web APIs (Generic)
- Basics of REST , or Another Intro
- gRPC APIs intro and one more
The basic idea of an API is to provide a way to control Grafana (in this case) via a programmer interface (in code or otherwise). Most of the Grafana APIs are currently documented here, but they're going through some growing pains and changes and document some of the patterns that are available as well as how to use the new APIs.
Now, one thing we should highlight while some APIs are publicly available, many of them are only available to you after you provide a key, or a valid authentication that identifies you.
Public API Example:
curl http://localhost:3000/api/health ## will return basic server info and health{
"database": "ok",
"version": "13.2.0",
"commit": "f681b1359f6a0b8ecb9f2c49a88ac72b75bde73b",
"enterpriseCommit": "50fa642a9129991374ed6a19ad771b6f72dbc5c1"
}Now there are two forms of authentication, username/password or token based. If we were to try to call api/org without any auth we'd get a permission denied. On the other hand, if I point to my local instance and do something along these lines:
curl -u "admin:admin" http://localhost:3000/api/orgI would get a proper response identifying the Org I belong to
{
"id": 1,
"name": "Main Org.",
"address": {
"address1": "",
"address2": "",
"city": "",
"zipCode": "",
"state": "",
"country": ""
}
}
Now, if I want to use a service account the pattern is a bit different.
curl -H "Authorization: Bearer glsa_Dp2hnh8hy1zuTdpJnCdzI2vtaYlNjo8O_3fe66c18" "http://localhost:3000/api/org"Then I will get a similar response.
{
"id": 2,
"name": "DiffrentOrg",
"address": {
"address1": "",
"address2": "",
"city": "",
"zipCode": "",
"state": "",
"country": ""
}
}
One thing to note, is that a user can belong to multiple organizations. That's not the case for a service account. They are bound to the Org that it was created on.
The User on the other hand is a bit dependent on the current state. If my admin user is currently looking at DiffrentOrg on the browser, you'll get a response matching that. So if you use basic auth and are in a multi org environment you should try to always pass the Org ID explicitly:
curl -u "admin:admin" -H "X-Grafana-Org-Id: 1" "http://localhost:3000/api/org"Coding and APIs
So now that we have the basics we're going to look at how to use it in code. I will focus on golang mostly because it is what I use.
Now, technically you don't NEED anything else but you will have to spend a lot of time creating all the models for every API you're using so using something pre-built would be nice.
The first attempt I went through was to use an unofficial library sdk made by grafana-tools that you can find here. It provided golang objects that matched the responses but as the objects were changing and Grafana was growing the small project could not keep up with the changes.
The first big innovation that came from Grafana was their adoption of OpenAPI that leads to the dawn of the silver age. (Well, not really but I'm going to call it that)
Drum roll and
Welcome to the Silver age
Two major things happened that heralded the silver age of API usage with Grafana.
- OpenAPI adoption, you can now find the full spec available here, and on any live server under
/public/api-merged.json. The /swagger endpoint was also added that allows you to navigate and explore the documented API - Going along with a new official project by Grafana has been started named grafana-openapi-client-go. The new project compiles the OpenAPI spec into valid golang models as well as providing simple examples that allow a user to connect his application.
Setting it up became as simple as importing the library and wiring it up in your code. For example the code below shows how to retrieve a list of all folders.
import (
goapi "github.com/grafana/grafana-openapi-client-go/client"
"github.com/grafana/grafana-openapi-client-go/client/search"
)
func demo() {
cfg := &goapi.TransportConfig{
// Host is the domain name or IP address of the host that serves the API.
Host: "localhost:3000",
// BasePath is the URL prefix for all API paths, relative to the host root.
BasePath: "/api",
// Schemes are the transfer protocols used by the API (http or https).
Schemes: []string{"http"},
// APIKey is an optional API key or service account token.
APIKey: os.Getenv("API_ACCESS_TOKEN"),
// BasicAuth is optional basic auth credentials.
BasicAuth: url.UserPassword("admin", "admin"),
// OrgID provides an optional organization ID.
// OrgID is only supported with BasicAuth since API keys are already org-scoped.
OrgID: 1,
// TLSConfig provides an optional configuration for a TLS client
TLSConfig: &tls.Config{},
// NumRetries contains the optional number of attempted retries
NumRetries: 3,
// RetryTimeout sets an optional time to wait before retrying a request
RetryTimeout: 0,
// RetryStatusCodes contains the optional list of status codes to retry
// Use "x" as a wildcard for a single digit (default: [429, 5xx])
RetryStatusCodes: []string{"420", "5xx"},
// HTTPHeaders contains an optional map of HTTP headers to add to each request
HTTPHeaders: map[string]string{},
}
client := goapi.NewHTTPClientWithConfig(strfmt.Default, cfg)
//usage...
p := search.NewSearchParams()
p.Type = &SearchTypeFolder
folderRawListing, err := client.Search.Search(p)
}This was great, not perfect but much better. We now had a project that followed the Grafana release cycle. The library was also used by the Terraform provider that Grafana paid attention to, so it's getting a bit more love than the grafana-tools project mentioned earlier. The OpenAPI is documented in the code.
Now, there are bugs and discrepancies between what OpenAPI claims to be the API and what the actual code does. Some of the code is mis-documented including the base API, some of the code mismatches the implementation. But fret not, for Grafana is on a mission to solve all your woes!
With Grafana v13, a new Kubernetes style API was introduced. The legacy code is still there but bit by bit it's being deprecated.
So welcome to the Golden Age? Maybe?
The Golden Age
The new APIs introduce some new patterns that allowed certain tooling like gcx, to be introduced. The new API has an auto discovery mode around it that allows some flexibility.
For example there's not one but multiple APIs defined now. Hitting the endpoint /apis will list all the current APIs available
curl -u "admin:admin" "http://localhost:3000/apis" | jq ".groups.[].name"
"dashboard.grafana.app"
"folder.grafana.app"
"userstorage.grafana.app"
"preferences.grafana.app"
"collections.grafana.app"
"features.grafana.app"
"sandboxsettings.grafana.app"
"provisioning.grafana.app"
"queries.grafana.app"
"playlist.grafana.app"
"plugins.grafana.app"
"example.grafana.app"
"quotas.grafana.app"
"shorturl.grafana.app"
"rules.alerting.grafana.app"
"notifications.alerting.grafana.app"
"advisor.grafana.app"
"dashboardtemplates.grafana.app"Then for each API you can discover all the versions available.
curl -u "admin:admin" "http://localhost:3000/apis/dashboard.grafana.app"Giving you:
{
"kind": "APIGroup",
"apiVersion": "v1",
"name": "dashboard.grafana.app",
"versions": [
{
"groupVersion": "dashboard.grafana.app/v2",
"version": "v2"
},
{
"groupVersion": "dashboard.grafana.app/v2beta1",
"version": "v2beta1"
},
{
"groupVersion": "dashboard.grafana.app/v2alpha1",
"version": "v2alpha1"
},
{
"groupVersion": "dashboard.grafana.app/v0alpha1",
"version": "v0alpha1"
},
{
"groupVersion": "dashboard.grafana.app/v1",
"version": "v1"
},
{
"groupVersion": "dashboard.grafana.app/v1beta1",
"version": "v1beta1"
}
],
"preferredVersion": {
"groupVersion": "dashboard.grafana.app/v2",
"version": "v2"
}
}Then for each one you can keep on going. We'll choose the latest version
curl -u "admin:admin" "http://localhost:3000/apis/dashboard.grafana.app/v2"Resource List:
{
"kind": "APIResourceList",
"apiVersion": "v1",
"groupVersion": "dashboard.grafana.app/v2",
"resources": [
{
"name": "dashboards",
"singularName": "dashboard",
"namespaced": true,
"kind": "Dashboard",
"verbs": [
"create",
"delete",
"get",
"list",
"patch",
"update"
]
},
{
"name": "dashboards/dto",
"singularName": "",
"namespaced": true,
"kind": "DashboardWithAccessInfo",
"verbs": [
"get"
]
}
]
}Then for each one we can build a call based on the action we want.
/apis/<group>/<version>/namespaces/<namespace>/<resource>The only thing that's missing is namespace. It will follow this convention:
defaultfor org1- org-ID for any other org, so if the org ID is 5, then the namespace will be
org-5
all the verbs translate to standard HTTP REST API
So in our case if we want to get all dashboards using the v2 API we'll use:
curl -X 'GET' -u 'admin:admin' 'http://localhost:3000/apis/dashboard.grafana.app/v2/namespaces/default/dashboards'The big advantage of this is that Grafana can provide the answer to what it supports, what version is available and so on. The endpoint parameters are also all standardized to match K8s conventions. Every listing endpoint will have the same parameters for example. I have not explored auto discovery programmatically though in theory it should be feasible and I believe gcx does this already.
Okay, now let's get into the code. There are two main projects that we'll need to use.
- The foundation SDK provides all the models for all known versions. For dashboards we'll use these models.
- We need to import the library that follows the K8s pattern. This one specifically to make our lives easier.
- You also need the metadata for K8s style endpoints that allows you to query the endpoints.
go doc k8s.io/apimachinery/pkg/apis/meta/v1.ListOptions
In the go code we'll need to build an identifier that has the info needed to identify the API, in our case it'll be the code below. We'll also add some convenience functions to get us the URL we want.
// GrafanaResourceRef wraps apimachinery's GroupVersionResource with helpers
// for building Grafana's Kubernetes-style API URLs from it.
type GrafanaResourceRef schema.GroupVersionResource
var dashboardResource = GrafanaResourceRef{
Group: "dashboard.grafana.app",
Version: "v2",
Resource: "dashboards",
}
func (r GrafanaResourceRef) CollectionURL(baseURL, namespace string, opts metav1.ListOptions) string {
base := fmt.Sprintf("%s/apis/%s/%s/namespaces/%s/%s",
baseURL, r.Group, r.Version, namespace, r.Resource)
q := url.Values{}
if opts.Limit > 0 {
q.Set("limit", strconv.FormatInt(opts.Limit, 10))
}
// ...and other options...
if len(q) == 0 {
return base
}
return base + "?" + q.Encode()
}
// ItemURL builds the URL for a single named resource:
//
// <baseURL>/apis/<group>/<version>/namespaces/<namespace>/<resource>/<name>
func (r GrafanaResourceRef) ItemURL(baseURL, namespace, name string) string {
return fmt.Sprintf("%s/%s", r.CollectionURL(baseURL, namespace, metav1.ListOptions{}), name)
}After which it's pretty trivial. We'll import the generic K8s metadata struct to set the options we are interested in.
opts := metav1.ListOptions{Limit: 5}
Then pass it on to our code.
func listDashboards(ctx context.Context, opts metav1.ListOptions) error {
url := dashboardGVR.CollectionURL(grafanaURL, namespace, opts)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url)
if err != nil {
return fmt.Errorf("building request: %w", err)
}
// set auth etc
req.SetBasicAuth(grafanaUser, grafanaPass)
req.Header.Set("Accept", "application/json")
//get the body, ...
//and then get the response
// apimachinery's UnstructuredList knows how to decode any Kubernetes-style
// list response (kind ending in "List", an "items" array) without needing
// a generated Go type for the outer envelope
list := &unstructured.UnstructuredList{}
if err := list.UnmarshalJSON(body); err != nil {
return fmt.Errorf("decoding dashboard list: %w", err)
}This is the part that gets interesting. If you are happy working with UnstructuredList you could in theory only work with that. The downside is there is type safety. If we want to have a concrete datatype then we need to convert it to the SDK.
func toTypedDashboard(obj map[string]interface{}) (*dashboardv2.Dashboard, error) {
specMap, found, err := unstructured.NestedMap(obj, "spec")
if err != nil {
return nil, err
}
if !found {
return nil, fmt.Errorf("no spec field present")
}
var dash dashboardv2.Dashboard
if err := runtime.DefaultUnstructuredConverter.FromUnstructured(specMap, &dash); err != nil {
return nil, err
}
return &dash, nil
}So, this opens the door to any API and allows a cli tool to auto discover any new endpoints, create, update, delete etc.
Closing thoughts
So what do you all think? Are we in the golden age? I would love to be able to build the models dynamically but given a statically typed language, I'm not sure how we can get any better than this. What do you all think?
Full Dashboard Listing Code
// list_dashboards.go
//
// Lists all dashboards in a Grafana org using the new Kubernetes-style API
// (dashboard.grafana.app/v2), decoded with k8s.io/apimachinery and typed
// via the Grafana Foundation SDK's dashboardv2 package.
//
// Setup:
// go mod init grafana-hello
// go get github.com/grafana/grafana-foundation-sdk/go@latest
// go get k8s.io/apimachinery@latest
// go run list_dashboards.go
package main
import (
"context"
"fmt"
"io"
"net/http"
"net/url"
"os"
"strconv"
dashboardv2 "github.com/grafana/grafana-foundation-sdk/go/dashboardv2"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/runtime/schema"
)
const (
grafanaURL = "http://localhost:3000"
grafanaUser = "admin"
grafanaPass = "admin"
// "default" is the namespace for the main (and, on a single-org OSS
// instance, only) org. For multi-org it's "org-<id>"; for Grafana
// Cloud it's "stacks-<stack_id>".
namespace = "default"
)
// GrafanaResourceRef identifies a Grafana Kubernetes-style resource
// (group/version/resource) and knows how to build API URLs from it.
// It's a plain type definition over schema.GroupVersionResource, so field
// literals (Group:/Version:/Resource:) work directly, at the cost of not
// inheriting GroupVersionResource's own methods (String(), Empty(), etc) -
// convert explicitly via schema.GroupVersionResource(ref) if you need those.
type GrafanaResourceRef schema.GroupVersionResource
// CollectionURL builds the URL for listing/creating resources in a
// namespace, encoding the standard Kubernetes ListOptions as query
// parameters (limit, continue, labelSelector, fieldSelector,
// resourceVersion, resourceVersionMatch, timeoutSeconds, watch):
//
// <baseURL>/apis/<group>/<version>/namespaces/<namespace>/<resource>?<query>
func (r GrafanaResourceRef) CollectionURL(baseURL, namespace string, opts metav1.ListOptions) string {
base := fmt.Sprintf("%s/apis/%s/%s/namespaces/%s/%s",
baseURL, r.Group, r.Version, namespace, r.Resource)
q := url.Values{}
if opts.Limit > 0 {
q.Set("limit", strconv.FormatInt(opts.Limit, 10))
}
if opts.Continue != "" {
q.Set("continue", opts.Continue)
}
if opts.LabelSelector != "" {
q.Set("labelSelector", opts.LabelSelector)
}
if opts.FieldSelector != "" {
q.Set("fieldSelector", opts.FieldSelector)
}
if opts.ResourceVersion != "" {
q.Set("resourceVersion", opts.ResourceVersion)
}
if opts.ResourceVersionMatch != "" {
q.Set("resourceVersionMatch", string(opts.ResourceVersionMatch))
}
if opts.TimeoutSeconds != nil {
q.Set("timeoutSeconds", strconv.FormatInt(*opts.TimeoutSeconds, 10))
}
if opts.Watch {
q.Set("watch", "true")
}
if len(q) == 0 {
return base
}
return base + "?" + q.Encode()
}
// ItemURL builds the URL for a single named resource:
//
// <baseURL>/apis/<group>/<version>/namespaces/<namespace>/<resource>/<name>
func (r GrafanaResourceRef) ItemURL(baseURL, namespace, name string) string {
return fmt.Sprintf("%s/apis/%s/%s/namespaces/%s/%s/%s",
baseURL, r.Group, r.Version, namespace, r.Resource, name)
}
// dashboardResource identifies the resource, mirroring what the API
// discovery endpoint returns:
//
// GET /apis/dashboard.grafana.app/v2
// -> groupVersion: "dashboard.grafana.app/v2"
// -> resources[].name: "dashboards"
var dashboardResource = GrafanaResourceRef{
Group: "dashboard.grafana.app",
Version: "v2",
Resource: "dashboards",
}
func main() {
// Only fetch the first 5 dashboards.
opts := metav1.ListOptions{Limit: 5}
if err := listDashboards(context.Background(), opts); err != nil {
fmt.Fprintln(os.Stderr, "error:", err)
os.Exit(1)
}
}
func listDashboards(ctx context.Context, opts metav1.ListOptions) error {
url := dashboardResource.CollectionURL(grafanaURL, namespace, opts)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return fmt.Errorf("building request: %w", err)
}
req.SetBasicAuth(grafanaUser, grafanaPass)
req.Header.Set("Accept", "application/json")
resp, err := http.DefaultClient.Do(req)
if err != nil {
return fmt.Errorf("calling grafana: %w", err)
}
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil {
return fmt.Errorf("reading response: %w", err)
}
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("unexpected status %d: %s", resp.StatusCode, string(body))
}
// apimachinery's UnstructuredList knows how to decode any Kubernetes-style
// list response (kind ending in "List", an "items" array) without needing
// a generated Go type for the outer envelope.
list := &unstructured.UnstructuredList{}
if err := list.UnmarshalJSON(body); err != nil {
return fmt.Errorf("decoding dashboard list: %w", err)
}
fmt.Printf("Found %d dashboard(s) in namespace %q:\n\n", len(list.Items), namespace)
for _, item := range list.Items {
// item is an unstructured.Unstructured (just a map[string]interface{}
// under the hood). GetName()/GetNamespace()/GetLabels() etc. come
// from apimachinery's generic Object accessor methods.
uid := item.GetName() // metadata.name is the dashboard UID
title, _, _ := unstructured.NestedString(item.Object, "spec", "title")
fmt.Printf("- %-40s (uid: %s)\n", title, uid)
// Optional: convert the loose "spec" map into the Foundation SDK's
// strongly typed dashboardv2.Dashboard struct, for compile-time
// checked access to the rest of the spec (layout, elements,
// variables, etc). Verify the exact field names for your installed
// SDK version at:
// https://pkg.go.dev/github.com/grafana/grafana-foundation-sdk/go/dashboardv2
if _, err := toTypedDashboard(item.Object); err != nil {
fmt.Printf(" (couldn't convert to typed dashboardv2.Dashboard: %v)\n", err)
}
}
if c := list.GetContinue(); c != "" {
fmt.Printf("\nMore results available - pass continue: %q to fetch the next page.\n", c)
}
return nil
}
// toTypedDashboard converts the "spec" section of a raw unstructured
// dashboard object into the Foundation SDK's typed Dashboard struct using
// apimachinery's reflection-based converter (matches on json tags, same
// mechanism client-go's dynamic client uses under the hood).
func toTypedDashboard(obj map[string]interface{}) (*dashboardv2.Dashboard, error) {
specMap, found, err := unstructured.NestedMap(obj, "spec")
if err != nil {
return nil, err
}
if !found {
return nil, fmt.Errorf("no spec field present")
}
var dash dashboardv2.Dashboard
if err := runtime.DefaultUnstructuredConverter.FromUnstructured(specMap, &dash); err != nil {
return nil, err
}
return &dash, nil
}