Monitoring Golang Application with Prometheus

A warm Hello to all the readers!

I have posted multiple articles about getting started with Prometheus. It talks about installing, configuring and monitoring system through Prometheus.

In this section, we will learn how to instrument our golang code to export interesting metrics to Prometheus. Today’s world is about white box monitoring than old school days of black box monitoring. We should be aware the application performance in detail and notified if it is not upto the expected benchmarks.

Let’s start in instrumenting our Golang application.

Creating Dummy HTTP Service

I will write a dummy HTTP service that will be instrumented. Below is the code for it.

package main

import (
    "fmt"
    "net/http"
    "time"
)

func main() {
    http.HandleFunc("/speakSlow", speakSlow)
    http.HandleFunc("/", speak)
    http.ListenAndServe(":8080", nil)
}

func speak(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Prom, %s!\n", r.URL.Path[1:])
}


func speakSlow(w http.ResponseWriter, r *http.Request) {
    time.Sleep(time.Second * 7)
    fmt.Fprintf(w, "Prom, %s!\n", r.URL.Path[1:])
}

Above is a very basic Golang HTTP server that speaks to you :).

$ go run simple_http_service.go 

$ curl localhost:8080/
Prom, !

Instrumenting Code for Prometheus

There are two golang packages that will help us to expose metrics for Prometheus. Following are the packages.

github.com/prometheus/client_golang/prometheus
github.com/prometheus/client_golang/prometheus/promhttp

Below is the complete code of simple HTTP service that is modified to expose metrics.

package main

import (
    "fmt"
    "net/http"
    "time"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

func main() {
    histogram := prometheus.NewHistogramVec(prometheus.HistogramOpts{
        Name:    "speak_seconds",
        Help:    "Time take to speak",
        Buckets: []float64{1, 2, 5, 6, 10},
    }, []string{"code"})
    prometheus.Register(histogram)

    http.Handle("/metrics", promhttp.Handler())
    http.HandleFunc("/speakSlow", speakSlow(histogram))
    http.HandleFunc("/", speak(histogram))
    http.ListenAndServe(":8080", nil)
}

func speak(histogram *prometheus.HistogramVec) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        code := 200
        defer func() {
            duration := time.Since(start)
            histogram.WithLabelValues(fmt.Sprintf("%d", code)).Observe(duration.Seconds())
        }()

        fmt.Fprintf(w, "Prom, %s!\n", r.URL.Path[1:])
    }
}

func speakSlow(histogram *prometheus.HistogramVec) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        code := 200
        time.Sleep(7 * time.Second)
        defer func() {
            duration := time.Since(start)
            histogram.WithLabelValues(fmt.Sprintf("%d", code)).Observe(duration.Seconds())
        }()

        fmt.Fprintf(w, "Prom, %s!\n", r.URL.Path[1:])
    }
}

$ go run http_service.go

$ curl localhost:8080/metrics

# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 0
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0

$ curl localhost:8080

# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 1
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0
# HELP speak_seconds Time take to speak
# TYPE speak_seconds histogram
speak_seconds_bucket{code="200",le="1"} 1
speak_seconds_bucket{code="200",le="2"} 1
speak_seconds_bucket{code="200",le="5"} 1
speak_seconds_bucket{code="200",le="6"} 1
speak_seconds_bucket{code="200",le="10"} 1
speak_seconds_bucket{code="200",le="+Inf"} 1
speak_seconds_sum{code="200"} 5.705e-06
speak_seconds_count{code="200"} 1

$ curl localhost:8080/speakSlow

# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 2
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0
# HELP speak_seconds Time take to speak
# TYPE speak_seconds histogram
speak_seconds_bucket{code="200",le="1"} 1
speak_seconds_bucket{code="200",le="2"} 1
speak_seconds_bucket{code="200",le="5"} 1
speak_seconds_bucket{code="200",le="6"} 1
speak_seconds_bucket{code="200",le="10"} 2
speak_seconds_bucket{code="200",le="+Inf"} 2
speak_seconds_sum{code="200"} 7.000108857000001
speak_seconds_count{code="200"} 2

We have now exposed the metrics. Next step is to tell Prometheus to scrape these metrics at regular intervals.

Scrape Golang Application Metrics

Edit prometheus config with the new target and re-run the prometheus binary.

$ vim prometheus.yml

scrape_configs:
  - job_name: 'golang'
    static_configs:
      - targets: ['localhost:8080']

Start visualizing metrics through expression manager.

Likewise we can collect various other metrics from golang application. Hope you enjoyed reading the guide.