These are the notes for Go episode 70. Raju's order service now runs in production, and at 2 A.M. customers say "the app is slow, some orders fail". The servers are running, but nobody can say which requests fail, how many, or which of the three services is slow. It's like driving a car with no dashboard. In this episode we invent the dashboard, piece by piece: logs that machines can search, with a request id in every line; metrics (counters, gauges, histograms) that Prometheus collects from /metrics; the RED and USE methods that say what to measure; traces made of spans that follow one request across services with the traceparent header; and health checks that tell "alive" apart from "ready". Every program below ran for real.
All outputs are from real runs (Go 1.24.7, linux/amd64) with github.com/prometheus/client_golang v1.23.2 and go.opentelemetry.io/otel/sdk v1.39.0 (which brings go.opentelemetry.io/otel and go.opentelemetry.io/otel/trace v1.39.0). In this sandbox the modules come from a local module proxy. Trace ids, span ids, timestamps and timings change on every run, so your ids and milliseconds will differ from the ones printed here.
Facts and sources
From the episode's research, the Go documentation of the installed toolchain (go doc, Go 1.24.7), and the source and READMEs of the exact module versions used (read from the module cache in this run):
$(go env GOROOT)/api/go1.21.txtlistspkg log/slog: slog arrived in Go 1.21.expvaris listed inapi/go1.txt(it has been in the standard library since Go 1).go doc log/slog: "Package slog provides structured logging, in which log records include a message, a severity level, and various other attributes expressed as key-value pairs." and "In addition to Logger.Info, there are methods for Debug, Warn and Error levels."go doc log/slog.Logger.With: "With returns a Logger that includes the given attributes in each output operation."go doc expvar: "Package expvar provides a standardized interface to public variables, such as operation counters in servers. It exposes these variables via HTTP at /debug/vars in JSON format. As of Go 1.22, the /debug/vars request must use GET." It also registerscmdline(os.Args) andmemstats(runtime.Memstats).- client_golang README: "the Go client library for Prometheus. It has two separate parts, one for instrumenting application code, and one for creating clients that talk to the Prometheus HTTP API".
- client_golang v1.23.2 source comments: a Counter "represents a single numerical value that only ever goes up"; a Gauge "represents a single numerical value that can arbitrarily go up and down"; a Histogram "counts individual observations ... in configurable static buckets" and "quantiles can be calculated from a Histogram using the histogram_quantile PromQL function"; for a Summary, "the rank estimations cannot be aggregated in a meaningful way with the Prometheus query language (i.e. you cannot average or add them). If you need aggregatable quantiles (e.g. you want the 99th percentile latency of all queries served across all instances of a service), consider the Histogram metric type." Default histogram buckets:
DefBuckets = []float64{.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10}(seconds). - opentelemetry-go README: "the Go implementation of OpenTelemetry. It provides a set of APIs to directly measure performance and behavior of your software and send this data to observability platforms". The v1.39.0 README's status table lists Traces: Stable, Metrics: Stable, Logs: Beta, and its exporters table lists OTLP (logs, metrics, traces), Prometheus (metrics), stdout (all three) and Zipkin (traces).
- otel
propagation/trace_context.go: "TraceContext is a propagator that supports the W3C Trace Context format" and it propagates "the traceparent and tracestate headers". - otel SDK
trace/sampling.go: "TraceIDRatioBased samples a given fraction of traces." - W3C Trace Context:
traceparent=version-trace-id-parent-id-trace-flags; trace-id is 16 bytes (32 hex characters), parent-id is 8 bytes (16 hex characters), version00, flag01= sampled. - Everything else is from the real runs below. Where the notes mention a tool by name that the video names only as an example (Jaeger, Grafana Tempo, Kubernetes, the
rate()query), it is marked "(named in the video)".
The problem: 2 A.M. and nobody can see inside
Real-life example: a car with no dashboard. No speedometer, no fuel gauge, no warning lights. You only find out something is wrong when the car stops.
Raju's team at 2 A.M.:
- customers: "the app is slow, orders are failing";
- the servers are up, CPU looks normal from outside;
- but which requests fail? How many? Which of the three services (order, database, kitchen) is slow? Nobody can tell.
Observability is the dashboard: being able to understand what is happening inside a running system from the signals it gives out, without stopping it or adding a fmt.Println and redeploying. (Monitoring is watching those signals and raising an alarm when one crosses a limit.)
The three pillars of observability
There are three main kinds of signal, called the three pillars:
| Pillar | Question it answers | Example from this episode |
|---|---|---|
| Logs | what happened, to which request? (one event) | payment failed for req-102: upi timeout |
| Metrics | how many, how fast? (numbers over time) | 1 of 20 requests failed; 2 took over 100 ms |
| Traces | where did the time go? (one request across services) | 120 of 135 ms were spent in the kitchen service |
They work together: metrics raise the alarm ("errors went up"), a trace finds the slow or failing part ("the kitchen"), and logs explain why ("upi timeout after 2s").
Logs: a sentence vs key=value fields
Real-life example: a diary. "Today asha's order 81 failed after a long wait" is easy for a person to read, but try finding every failure for asha longer than 2 seconds in a million diary pages.
- Unstructured logging writes sentences for humans (
log.Printf("order %d failed for user %s ...")). Searching meansgrepand hoping nobody ever changed the wording. - Structured logging writes each event as named fields (key and value), often one JSON object per line. Log tools can then search
user = "asha" AND ms > 2000like a database.
Go has structured logging built in since Go 1.21: the log/slog package ("S-log"). A Logger has methods Debug, Info, Warn, Error; each takes a message and then key, value, key, value, ... pairs. A Handler decides the output format: TextHandler writes key=value, JSONHandler writes JSON.
Every program in these notes uses one of these go.mod files:
module shop
go 1.24
(standard library only), or the same with require github.com/prometheus/client_golang v1.23.2 (Prometheus programs), or require go.opentelemetry.io/otel/sdk v1.39.0 (tracing programs). The mini project needs both.
Printf vs slog, side by side
main.go (standard library only):
package main
import (
"log"
"log/slog"
"os"
)
func main() {
id, user, ms := 81, "asha", 2100
// 1. unstructured: a sentence for humans
log.SetOutput(os.Stdout)
log.Printf("order %d failed for user %s after %dms", id, user, ms)
// 2. slog's default logger: same sentence style, but with key=value pairs
slog.Error("order failed", "order_id", id, "user", user, "ms", ms)
// 3. a TextHandler: key=value fields
text := slog.New(slog.NewTextHandler(os.Stdout, nil))
text.Error("order failed", "order_id", id, "user", user, "ms", ms)
// 4. a JSONHandler: one JSON object per line
js := slog.New(slog.NewJSONHandler(os.Stdout, nil))
js.Error("order failed", "order_id", id, "user", user, "ms", ms)
}
2026/11/13 23:58:10 order 81 failed for user asha after 2100ms
2026/11/13 23:58:10 ERROR order failed order_id=81 user=asha ms=2100
time=2026-11-13T23:58:10.922Z level=ERROR msg="order failed" order_id=81 user=asha ms=2100
{"time":"2026-11-13T23:58:10.922372583Z","level":"ERROR","msg":"order failed","order_id":81,"user":"asha","ms":2100}
(Timestamps are from this run; yours will show the current time.)
log.Printf: a sentence. The numbers are inside the text.slog.Erroron the default logger: goes through thelogpackage, so it looks like a log line, but the fields are alreadykey=value.TextHandler:time=... level=ERROR msg="order failed" order_id=81 ..., every value has a name.JSONHandler: one JSON object per line, the easiest for machines (log collectors, search tools).
Levels and With
A level says how important a line is: Debug (details while developing) < Info (normal events) < Warn (something odd) < Error (something failed). A handler's Level option is the minimum it prints; production usually shows Info and above.
main.go (standard library only):
package main
import (
"log/slog"
"os"
)
func noTime(_ []string, a slog.Attr) slog.Attr {
if a.Key == slog.TimeKey {
return slog.Attr{} // drop the time so the output is the same every run
}
return a
}
func main() {
for _, min := range []slog.Level{slog.LevelInfo, slog.LevelDebug} {
l := slog.New(slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{Level: min, ReplaceAttr: noTime}))
l.Info("--- handler level " + min.String() + " ---")
l.Debug("cache lookup", "key", "menu:chai")
l.Info("order placed", "order_id", 81)
l.Warn("stock low", "item", "samosa", "left", 3)
l.Error("payment failed", "gateway", "upi")
}
// With: a child logger that adds fields to every line
reqLog := slog.New(slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{ReplaceAttr: noTime})).
With("request_id", "req-7")
reqLog.Info("start")
reqLog.Info("done", "status", 201)
// a missing value: slog warns you with !BADKEY
reqLog.Info("oops", "status")
}
level=INFO msg="--- handler level INFO ---"
level=INFO msg="order placed" order_id=81
level=WARN msg="stock low" item=samosa left=3
level=ERROR msg="payment failed" gateway=upi
level=INFO msg="--- handler level DEBUG ---"
level=DEBUG msg="cache lookup" key=menu:chai
level=INFO msg="order placed" order_id=81
level=WARN msg="stock low" item=samosa left=3
level=ERROR msg="payment failed" gateway=upi
level=INFO msg=start request_id=req-7
level=INFO msg=done request_id=req-7 status=201
level=INFO msg=oops request_id=req-7 !BADKEY=status
- With
Level: slog.LevelInfo(the default) theDebugline was dropped; withslog.LevelDebugit appeared. ReplaceAttrlets you change or remove any field; returning an emptyslog.Attr{}removes it. We removetimeonly so outputs repeat exactly (keep it in real apps).With("request_id", "req-7")returns a child logger that adds that field to every line it writes."status"with no value printed!BADKEY=status: arguments must come in pairs.
Structured request logging with a request id
The problem: 100 requests run at the same time, and their log lines are mixed together. Which "payment failed" line belongs to which request?
The fix: give every request a request id (a short unique name such as req-102), put it in every line that request writes, and then searching for req-102 gives the whole story of that one request.
How the id reaches code deep inside the handler: the context. Every *http.Request carries a context.Context (r.Context()), a bag of request-scoped values that travels with the request (episode on context). A middleware (a function that wraps a handler and runs code before and after it) creates a logger with the id, puts it into the context, and the handler takes it back out. This is context-aware logging: the request's details travel in its context, so any function that gets the context can log with them.
The episode's program (LOGS)
main.go (standard library only):
package main
import (
"context"
"fmt"
"log/slog"
"net/http"
"net/http/httptest"
"os"
"sync/atomic"
"time"
)
type ctxKey struct{}
// every log line in this request carries its request_id
func logger(ctx context.Context) *slog.Logger {
if l, ok := ctx.Value(ctxKey{}).(*slog.Logger); ok {
return l
}
return slog.Default()
}
type recorder struct {
http.ResponseWriter
status int
}
func (r *recorder) WriteHeader(code int) { r.status = code; r.ResponseWriter.WriteHeader(code) }
var nextID atomic.Int64
// middleware: one structured line per request
func withLogging(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
l := slog.Default().With("request_id", fmt.Sprint("req-", 100+nextID.Add(1)))
ctx := context.WithValue(r.Context(), ctxKey{}, l)
rec := &recorder{w, 200}
start := time.Now()
next.ServeHTTP(rec, r.WithContext(ctx))
l.Info("request", "method", r.Method, "path", r.URL.Path, "status", rec.status,
"ms", time.Since(start).Milliseconds())
})
}
func placeOrder(w http.ResponseWriter, r *http.Request) {
if r.URL.Query().Get("pay") == "fail" {
logger(r.Context()).Error("payment failed", "gateway", "upi", "err", "timeout after 2s")
http.Error(w, "payment failed", http.StatusBadGateway)
return
}
w.WriteHeader(http.StatusCreated)
}
func main() {
// JSON lines: easy for machines to search ("time" removed here to fit the screen)
slog.SetDefault(slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
ReplaceAttr: func(_ []string, a slog.Attr) slog.Attr {
if a.Key == slog.TimeKey {
return slog.Attr{}
}
return a
},
})))
srv := httptest.NewServer(withLogging(http.HandlerFunc(placeOrder)))
defer srv.Close()
http.Post(srv.URL+"/orders", "", nil)
http.Post(srv.URL+"/orders?pay=fail", "", nil)
}
{"level":"INFO","msg":"request","request_id":"req-101","method":"POST","path":"/orders","status":201,"ms":0}
{"level":"ERROR","msg":"payment failed","request_id":"req-102","gateway":"upi","err":"timeout after 2s"}
{"level":"INFO","msg":"request","request_id":"req-102","method":"POST","path":"/orders","status":502,"ms":0}
- Line 1: request
req-101,POST /orders, status 201 (Created). - Line 2: an ERROR line,
payment failed,gateway: upi, written deep insideplaceOrder, and it carriesrequest_id: req-102even thoughplaceOrdernever saw the id: it came from the context. - Line 3: the request line for
req-102, status 502 (Bad Gateway). msis0because these test requests are very fast; it's the time in milliseconds.
How the pieces work:
type ctxKey struct{}: a private type used as the context key, so no other package can clash with it.logger(ctx)returns the logger stored in the context, orslog.Default()if there is none.recorderwrapshttp.ResponseWriterand remembers the status code passed toWriteHeader(the default is 200, because a handler that never callsWriteHeadersends 200).nextIDis anatomic.Int64: handlers run on many goroutines at once, so the counter must be safe for concurrent use (episode on sync/atomic).slog.SetDefaultmakes the JSON logger the default one.
Logging habits
| Habit | Why |
|---|---|
| one line per request + one line per real problem | enough to debug, not so much that it costs a fortune |
stable key names: request_id, user_id, ms, status |
searches and dashboards keep working |
| never log passwords, tokens, card numbers, OTPs | logs are read by many people and kept for a long time |
| log an error once, where it's handled (episode 25) | the same error logged at every level makes 5 lines for 1 problem |
send the request id back in a response header (X-Request-ID) |
a customer or another service can quote it to support |
Metrics: the car's dashboard
Real-life example: the dashboard shows numbers, not stories.
| In a car | Metric type | Meaning | Example |
|---|---|---|---|
| odometer: total km, only goes up | Counter | a number that only ever goes up (resets to 0 when the program restarts) | http_requests_total |
| fuel gauge: up and down | Gauge | a value right now, can go up and down | requests in flight, queue length |
| trips sorted by length: under 5 km, under 20 km... | Histogram | counts values into ranges called buckets | request durations |
| (no car part) | Summary | computes percentiles inside the app | p50, p99 of payment time |
A metric is a named number measured over time. A time series (or series) is one metric with one exact set of labels (key=value tags such as code="500") and its values over time.
Prometheus is a popular system that collects metrics: every few seconds it scrapes each server, which means it sends GET /metrics, reads the plain-text numbers, and stores them over time. client_golang is its Go library: the part we use here instruments application code and serves /metrics.
The episode's program (METRICS)
main.go (require github.com/prometheus/client_golang v1.23.2):
package main
import (
"bufio"
"fmt"
"net/http"
"net/http/httptest"
"strconv"
"strings"
"time"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
)
var (
requests = prometheus.NewCounterVec( // only goes up
prometheus.CounterOpts{Name: "http_requests_total", Help: "Requests by path and status code."},
[]string{"path", "code"})
inFlight = prometheus.NewGauge( // goes up and down
prometheus.GaugeOpts{Name: "http_in_flight_requests", Help: "Requests being served right now."})
duration = prometheus.NewHistogram( // counts values into buckets
prometheus.HistogramOpts{Name: "http_request_duration_seconds", Help: "How long requests took.",
Buckets: []float64{0.01, 0.05, 0.1, 0.5}})
)
type statusRecorder struct {
http.ResponseWriter
code int
}
func (s *statusRecorder) WriteHeader(c int) { s.code = c; s.ResponseWriter.WriteHeader(c) }
func instrument(path string, next http.HandlerFunc) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
inFlight.Inc()
defer inFlight.Dec()
start := time.Now()
rec := &statusRecorder{w, 200}
next(rec, r)
duration.Observe(time.Since(start).Seconds())
requests.WithLabelValues(path, strconv.Itoa(rec.code)).Inc()
}
}
func orders(w http.ResponseWriter, r *http.Request) {
if r.URL.Query().Get("slow") == "1" {
time.Sleep(120 * time.Millisecond) // a slow database call
}
if r.URL.Query().Get("fail") == "1" {
w.WriteHeader(http.StatusInternalServerError)
}
}
func main() {
reg := prometheus.NewRegistry()
reg.MustRegister(requests, inFlight, duration)
mux := http.NewServeMux()
mux.HandleFunc("/orders", instrument("/orders", orders))
mux.Handle("/metrics", promhttp.HandlerFor(reg, promhttp.HandlerOpts{})) // Prometheus scrapes this
srv := httptest.NewServer(mux)
defer srv.Close()
for i := 0; i < 20; i++ { // 20 requests: 2 slow, 1 failing
q := ""
switch i {
case 5, 12:
q = "?slow=1"
case 9:
q = "?fail=1"
}
http.Get(srv.URL + "/orders" + q)
}
resp, err := http.Get(srv.URL + "/metrics")
if err != nil {
panic(err)
}
defer resp.Body.Close()
sc := bufio.NewScanner(resp.Body)
for sc.Scan() {
if line := sc.Text(); strings.HasPrefix(line, "http_") {
fmt.Println(line)
}
}
}
http_in_flight_requests 0
http_request_duration_seconds_bucket{le="0.01"} 18
http_request_duration_seconds_bucket{le="0.05"} 18
http_request_duration_seconds_bucket{le="0.1"} 18
http_request_duration_seconds_bucket{le="0.5"} 20
http_request_duration_seconds_bucket{le="+Inf"} 20
http_request_duration_seconds_sum 0.24065418900000002
http_request_duration_seconds_count 20
http_requests_total{code="200",path="/orders"} 19
http_requests_total{code="500",path="/orders"} 1
(The _sum value is the total seconds of all 20 requests; it changes slightly on every run.)
prometheus.NewCounterVec: a counter with labels (path,code).WithLabelValues("/orders", "200")picks one series;.Inc()adds 1. Result: 19 with code 200, 1 with code 500.prometheus.NewGauge:inFlight.Inc()when a request starts,defer inFlight.Dec()when it ends. It is 0 now because nothing is running.prometheus.NewHistogramwithBuckets: []float64{0.01, 0.05, 0.1, 0.5}(seconds: 10, 50, 100, 500 ms).Observe(seconds)records one duration.- Histogram buckets are cumulative:
lemeans "less than or equal".le="0.01"} 18= 18 requests took at most 10 ms; still 18 at 100 ms; 20 at 500 ms, so 2 were slow (the two?slow=1requests with a 120 ms sleep).le="+Inf"is "everything" and always equals_count. _sum/_count= the average duration (here about 0.24 s / 20 = 12 ms).prometheus.NewRegistry()+MustRegister: a registry is the list of metrics to expose.promhttp.HandlerFor(reg, ...)is the HTTP handler that writes them: the/metricsendpoint.instrumentis a middleware: it updates all three metrics around every request, so handlers stay clean.
All four types, and what /metrics really looks like
The episode program printed only the http_ lines. Here is a full /metrics page, including a summary:
main.go (require github.com/prometheus/client_golang v1.23.2):
package main
import (
"fmt"
"net/http/httptest"
"io"
"strings"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
)
func main() {
placed := prometheus.NewCounter(prometheus.CounterOpts{Name: "orders_placed_total", Help: "Orders placed since start."})
waiting := prometheus.NewGauge(prometheus.GaugeOpts{Name: "customers_waiting", Help: "Customers in the queue now."})
payHist := prometheus.NewHistogram(prometheus.HistogramOpts{Name: "payment_seconds", Help: "Payment time.",
Buckets: []float64{0.1, 0.5, 1}})
paySum := prometheus.NewSummary(prometheus.SummaryOpts{Name: "payment_summary_seconds", Help: "Payment time.",
Objectives: map[float64]float64{0.5: 0.05, 0.99: 0.001}}) // percentile: allowed error
reg := prometheus.NewRegistry()
reg.MustRegister(placed, waiting, payHist, paySum)
placed.Add(3)
waiting.Set(5)
waiting.Dec() // one customer served
// 100 payments: 98 fast (0.05s), 2 slow (0.9s)
for i := 0; i < 100; i++ {
d := 0.05
if i%50 == 49 {
d = 0.9
}
payHist.Observe(d)
paySum.Observe(d)
}
// placed.Add(-1) would panic: "counter cannot decrease in value"
rec := httptest.NewRecorder()
promhttp.HandlerFor(reg, promhttp.HandlerOpts{}).ServeHTTP(rec, httptest.NewRequest("GET", "/metrics", nil))
body, _ := io.ReadAll(rec.Body)
fmt.Print(strings.TrimSpace(string(body)), "\n")
}
# HELP customers_waiting Customers in the queue now.
# TYPE customers_waiting gauge
customers_waiting 4
# HELP orders_placed_total Orders placed since start.
# TYPE orders_placed_total counter
orders_placed_total 3
# HELP payment_seconds Payment time.
# TYPE payment_seconds histogram
payment_seconds_bucket{le="0.1"} 98
payment_seconds_bucket{le="0.5"} 98
payment_seconds_bucket{le="1"} 100
payment_seconds_bucket{le="+Inf"} 100
payment_seconds_sum 6.699999999999991
payment_seconds_count 100
# HELP payment_summary_seconds Payment time.
# TYPE payment_summary_seconds summary
payment_summary_seconds{quantile="0.5"} 0.05
payment_summary_seconds{quantile="0.99"} 0.9
payment_summary_seconds_sum 6.699999999999991
payment_summary_seconds_count 100
# HELPis the description you passed asHelp;# TYPEsays counter, gauge, histogram or summary. This text format is what Prometheus scrapes.- Counter:
Add(3)→3. Gauge:Set(5)thenDec()→4. - The histogram says 98 payments took ≤ 0.1 s, all 100 took ≤ 1 s. It does not store each value, only the bucket counts.
- The summary computed the percentiles itself:
quantile="0.5"(the median) is0.05,quantile="0.99"is0.9.Objectivesmaps each percentile to the allowed error. - Prefer histograms: a summary's percentiles are computed inside one process and "cannot be aggregated" (you can't add or average p99s from 10 servers), while histogram buckets from all servers can simply be added, and the server computes percentiles from them (
histogram_quantilein PromQL, Prometheus's query language).
A counter really can't go down:
main.go (require github.com/prometheus/client_golang v1.23.2):
package main
import "github.com/prometheus/client_golang/prometheus"
func main() {
placed := prometheus.NewCounter(prometheus.CounterOpts{Name: "orders_placed_total", Help: "Orders placed."})
placed.Add(3)
placed.Add(-1) // a refund? counters only go up
}
panic: counter cannot decrease in value
(followed by the goroutine stack trace). Use a gauge for values that go down, or a second counter (refunds_total).
p99: why averages lie
A percentile answers "how slow were the slowest X%?". p50 (the median): half the requests were faster. p99: 99% of requests were faster than this; it shows the slow tail that an average hides. 1% sounds small, but at 1,000 requests a minute that's 10 unhappy customers every minute.
main.go (standard library only):
package main
import (
"fmt"
"sort"
"time"
)
func main() {
// 1000 requests: 985 take 10ms, 15 get stuck for 2s
var d []time.Duration
for i := 0; i < 1000; i++ {
d = append(d, 10*time.Millisecond)
}
for i := 0; i < 15; i++ {
d[i*60] = 2 * time.Second
}
var sum time.Duration
for _, x := range d {
sum += x
}
sort.Slice(d, func(i, j int) bool { return d[i] < d[j] })
pct := func(p float64) time.Duration { return d[int(p*float64(len(d)))-1] } // the value p of requests are at or below
fmt.Println("average:", sum/time.Duration(len(d)))
fmt.Println("p50: ", pct(0.50))
fmt.Println("p90: ", pct(0.90))
fmt.Println("p99: ", pct(0.99))
fmt.Println("max: ", d[len(d)-1])
}
average: 39.85ms
p50: 10ms
p90: 10ms
p99: 2s
max: 2s
The average says "40 ms, fine". p50 and p90 say "10 ms". Only p99 = 2s shows that 15 customers waited 2 seconds. That's why we measure durations with histograms and look at p99.
Labels and cardinality
A label splits one metric into several series: http_requests_total{code="200"} and {code="500"}. Cardinality is the number of different series a metric has = the number of different label-value combinations. Every series costs memory in your app and in Prometheus.
main.go (require github.com/prometheus/client_golang v1.23.2):
package main
import (
"fmt"
"io"
"net/http/httptest"
"strings"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
)
func series(reg *prometheus.Registry, name string) int {
rec := httptest.NewRecorder()
promhttp.HandlerFor(reg, promhttp.HandlerOpts{}).ServeHTTP(rec, httptest.NewRequest("GET", "/metrics", nil))
b, _ := io.ReadAll(rec.Body)
n := 0
for _, line := range strings.Split(string(b), "\n") {
if strings.HasPrefix(line, name+"{") {
n++
}
}
return n
}
func main() {
good := prometheus.NewCounterVec(prometheus.CounterOpts{Name: "orders_total", Help: "Orders."},
[]string{"status"}) // few, fixed values
bad := prometheus.NewCounterVec(prometheus.CounterOpts{Name: "orders_by_user_total", Help: "Orders."},
[]string{"user_id"}) // one value per customer!
reg := prometheus.NewRegistry()
reg.MustRegister(good, bad)
for user := 1; user <= 5000; user++ { // 5000 customers order once each
status := "ok"
if user%100 == 0 {
status = "failed"
}
good.WithLabelValues(status).Inc()
bad.WithLabelValues(fmt.Sprint("u", user)).Inc()
}
fmt.Println("orders_total series: ", series(reg, "orders_total"))
fmt.Println("orders_by_user_total series:", series(reg, "orders_by_user_total"))
// wrong number of label values: a panic
defer func() { fmt.Println("panic:", recover()) }()
good.WithLabelValues("ok", "upi").Inc()
}
orders_total series: 2
orders_by_user_total series: 5000
panic: inconsistent label cardinality: expected 1 label values but got 2 in []string{"ok", "upi"}
statushas 2 values → 2 series.user_idhas one value per customer → 5000 series, and it grows with every new customer (millions later). That's high cardinality: Prometheus slows down and memory explodes.- Rule: labels only with few, fixed values (route, method, status code). Never user ids, order ids, emails or full URLs with ids in them. Per-user detail belongs in logs and traces.
- The wrong number of label values is a panic:
inconsistent label cardinality: expected 1 label values but got 2 in []string{"ok", "upi"}. (Here we caught it withrecoverto print it.)
How Prometheus uses these numbers
- It scrapes
GET /metricson every server every few seconds and stores each series over time. - Queries turn counters into rates. The video's example (named in the video):
rate(http_requests_total{code="500"}[5m])= errors per second over the last 5 minutes. - An alert fires when a query crosses a limit ("more than 1% errors for 5 minutes", "p99 above 500 ms").
- Keep
/metrics(and/debug/vars) off the public internet: they show internal details.
What to measure: RED and USE
Two checklists so you don't measure everything and still miss what matters.
RED, for every service that answers requests:
| Letter | Means | Our metric |
|---|---|---|
| Rate | requests per second | http_requests_total (all codes) |
| Errors | failed requests per second | http_requests_total{code="500"} |
| Duration | how long they take (p50, p99) | http_request_duration_seconds histogram |
Our three metrics in the METRICS program are exactly RED (plus the in-flight gauge).
USE, for every resource (CPU, memory, disk, a database connection pool, a queue):
| Letter | Means | Example |
|---|---|---|
| Utilization | how busy it is | CPU 80%; 9 of 10 DB connections in use |
| Saturation | how much work is waiting | queue length; the pool's WaitCount (episode 67) |
| Errors | failures of the resource | connection errors, disk errors |
RED tells you users are hurting; USE tells you which resource is the cause.
A USE metric for a pool, using a GaugeFunc (a gauge whose value is read by calling your function at scrape time):
main.go (require github.com/prometheus/client_golang v1.23.2):
package main
import (
"fmt"
"io"
"net/http/httptest"
"strings"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
)
func main() {
pool := make(chan struct{}, 10) // a pool of 10 connections (a slot taken = in use)
waiting := 0 // requests waiting for a free slot
reg := prometheus.NewRegistry()
reg.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{Name: "db_pool_in_use", Help: "Utilization: connections in use."},
func() float64 { return float64(len(pool)) }), // read at scrape time
prometheus.NewGaugeFunc(prometheus.GaugeOpts{Name: "db_pool_waiting", Help: "Saturation: requests waiting."},
func() float64 { return float64(waiting) }),
)
for i := 0; i < 9; i++ {
pool <- struct{}{} // 9 of 10 busy
}
waiting = 4
rec := httptest.NewRecorder()
promhttp.HandlerFor(reg, promhttp.HandlerOpts{}).ServeHTTP(rec, httptest.NewRequest("GET", "/metrics", nil))
b, _ := io.ReadAll(rec.Body)
for _, l := range strings.Split(string(b), "\n") {
if strings.HasPrefix(l, "db_") {
fmt.Println(l)
}
}
}
db_pool_in_use 9
db_pool_waiting 4
Utilization 9 of 10, and 4 requests waiting: the pool is saturated. With database/sql you would return db.Stats().InUse and db.Stats().WaitCount (episode 67).
Traces: tracking a courier parcel
Logs and metrics say "orders are slow", but one order passes through the order service, the database and the kitchen service. Which one is slow?
Real-life example: a courier parcel.
| Courier | Distributed tracing |
|---|---|
| one tracking number for the whole journey | trace id: one id for one request, across every service |
| each hub scans it in and out (in 9:00, out 11:00) | span: one step of work with a name, a start, an end and a parent span |
| the label travels with the parcel | the traceparent HTTP header carries the trace id to the next service (propagation) |
- A trace is the whole journey of one request. It is a tree of spans: the first span (the root span) has no parent; every other span has a parent, so you see what happened inside what.
- Distributed tracing means the trace continues across many services (programs on different machines).
- OpenTelemetry (OTel) is the open standard and set of libraries for traces, metrics and logs. In Go: the API (
go.opentelemetry.io/otel, what your code calls), the SDK (go.opentelemetry.io/otel/sdk, which processes and samples spans) and an exporter (which sends finished spans somewhere).
The episode's program (TRACE)
main.go (require go.opentelemetry.io/otel/sdk v1.39.0):
package main
import (
"context"
"fmt"
"net/http"
"net/http/httptest"
"time"
"go.opentelemetry.io/otel/propagation"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
"go.opentelemetry.io/otel/sdk/trace/tracetest"
)
var (
recorder = tracetest.NewSpanRecorder() // keeps finished spans so we can print them
provider = sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(recorder))
tracer = provider.Tracer("shop")
prop = propagation.TraceContext{} // the W3C traceparent header
)
// the kitchen service: continues the trace it receives in the header
func kitchen(w http.ResponseWriter, r *http.Request) {
ctx := prop.Extract(r.Context(), propagation.HeaderCarrier(r.Header))
fmt.Println(" kitchen got traceparent:", r.Header.Get("traceparent"))
_, span := tracer.Start(ctx, "kitchen.cook")
time.Sleep(120 * time.Millisecond) // the slow part
span.End()
}
func main() {
kitchenSrv := httptest.NewServer(http.HandlerFunc(kitchen))
defer kitchenSrv.Close()
// the order service handles one customer request
ctx, root := tracer.Start(context.Background(), "POST /orders")
_, db := tracer.Start(ctx, "db.insert_order")
time.Sleep(15 * time.Millisecond)
db.End()
req, _ := http.NewRequestWithContext(ctx, "POST", kitchenSrv.URL, nil)
prop.Inject(ctx, propagation.HeaderCarrier(req.Header)) // trace id travels in a header
http.DefaultClient.Do(req)
root.End()
for _, s := range recorder.Ended() {
fmt.Printf("trace %s… span %s parent %-16s %-16s %v\n", s.SpanContext().TraceID().String()[:8],
s.SpanContext().SpanID(), s.Parent().SpanID(), s.Name(), s.EndTime().Sub(s.StartTime()).Round(5*time.Millisecond))
}
}
kitchen got traceparent: 00-8c5d6237cdf64640fdc4338e1ebe0281-991a961f6cc4091d-01
trace 8c5d6237… span 42dc2abf36469ad6 parent 991a961f6cc4091d db.insert_order 15ms
trace 8c5d6237… span 91d81c7e12947fed parent 991a961f6cc4091d kitchen.cook 120ms
trace 8c5d6237… span 991a961f6cc4091d parent 0000000000000000 POST /orders 135ms
(Ids and the root's duration change every run: the video's run and this one both showed 135 ms; an earlier run in this session showed 140 ms.)
sdktrace.NewTracerProvidercreates tracers;WithSpanProcessor(recorder)sends every finished span to atracetest.SpanRecorder, which simply keeps them in memory so we can print them (real apps use an exporter instead).provider.Tracer("shop")returns a tracer, the object that starts spans."shop"names the code doing the tracing.tracer.Start(ctx, "POST /orders")starts a span and returns a new context that contains it. Spans started from that context become its children.span.End()records the end time.propagation.TraceContext{}implements the W3C Trace Context format.prop.Inject(ctx, HeaderCarrier(req.Header))writes thetraceparentheader into the outgoing request;prop.Extract(r.Context(), HeaderCarrier(r.Header))in the kitchen reads it back, so the kitchen's span joins the same trace.- Reading the output: all three spans have the same trace id.
db.insert_order(15 ms) andkitchen.cook(120 ms) both have parent = the span id ofPOST /orders(135 ms). Thetraceparentthe kitchen received ends with that same span id. So 120 of 135 ms were in the kitchen, found in one look. - The root is printed last because it finished last (
recorder.Ended()is in end order). A parent id of all zeros means "no parent": the root.
Forgetting to propagate
The same program with the prop.Inject line commented out:
// prop.Inject(ctx, propagation.HeaderCarrier(req.Header)) // forgotten!
kitchen got traceparent:
trace 13dc4df1… span d719d4e01d428831 parent ee032cf000613dc7 db.insert_order 15ms
trace 88335983… span 2ccc509ac6bb5feb parent 0000000000000000 kitchen.cook 120ms
trace 13dc4df1… span ee032cf000613dc7 parent 0000000000000000 POST /orders 135ms
The kitchen got an empty traceparent, so it started a new trace (88335983…, parent all zeros). Now there are two separate traces and nothing connects the slow kitchen to the customer's request. In real apps, instrumentation libraries do the Inject/Extract for you (see below), but only if every hop uses them.
Reading a traceparent header
traceparent: 00-<trace-id: 32 hex>-<parent span id: 16 hex>-01
| Part | Example | Meaning |
|---|---|---|
| version | 00 |
the format version |
| trace-id | 11b5c5e1aa9367dc12e09556aadc27ca |
16 bytes = 32 hex characters, the same for the whole trace |
| parent-id | 8d5791a2829506ed |
8 bytes = 16 hex characters, the id of the span that made the call |
| trace-flags | 01 |
01 = sampled (this trace is being recorded), 00 = not sampled |
(Hex = hexadecimal, digits 0-9 and a-f; two hex characters are one byte.)
main.go (require go.opentelemetry.io/otel/sdk v1.39.0):
package main
import (
"context"
"fmt"
"net/http"
"strings"
"go.opentelemetry.io/otel/propagation"
"go.opentelemetry.io/otel/trace"
)
func main() {
tp := "00-11b5c5e1aa9367dc12e09556aadc27ca-8d5791a2829506ed-01"
p := strings.Split(tp, "-")
fmt.Printf("version %s | trace id %s (%d hex) | parent id %s (%d hex) | flags %s\n",
p[0], p[1], len(p[1]), p[2], len(p[2]), p[3])
// what Extract makes of it (and of broken headers)
prop := propagation.TraceContext{}
for _, h := range []string{tp,
"00-11b5c5e1aa9367dc12e09556aadc27ca-8d5791a2829506ed-00", // not sampled
"00-11b5c5e1aa9367dc-8d5791a2829506ed-01", // trace id too short
} {
hdr := http.Header{}
hdr.Set("traceparent", h)
sc := trace.SpanContextFromContext(prop.Extract(context.Background(), propagation.HeaderCarrier(hdr)))
fmt.Printf("valid=%-5v remote=%-5v sampled=%-5v trace=%s\n", sc.IsValid(), sc.IsRemote(), sc.IsSampled(), sc.TraceID())
}
}
version 00 | trace id 11b5c5e1aa9367dc12e09556aadc27ca (32 hex) | parent id 8d5791a2829506ed (16 hex) | flags 01
valid=true remote=true sampled=true trace=11b5c5e1aa9367dc12e09556aadc27ca
valid=true remote=true sampled=false trace=11b5c5e1aa9367dc12e09556aadc27ca
valid=false remote=false sampled=false trace=00000000000000000000000000000000
- A good header gives a valid, remote (it came from another service) span context with the same trace id.
- Flags
00: valid, but not sampled. - A trace id that's too short is invalid: Extract ignores it and the trace id is all zeros, so this service would start a new trace.
Sampling
Recording every span of every request costs network, storage and money. Sampling records only some traces. The SDK's TraceIDRatioBased(0.1) keeps about 1 trace in 10, decided from the trace id:
main.go (require go.opentelemetry.io/otel/sdk v1.39.0):
package main
import (
"context"
"fmt"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
"go.opentelemetry.io/otel/sdk/trace/tracetest"
)
func main() {
rec := tracetest.NewSpanRecorder()
provider := sdktrace.NewTracerProvider(
sdktrace.WithSampler(sdktrace.TraceIDRatioBased(0.1)), // keep about 1 trace in 10
sdktrace.WithSpanProcessor(rec))
tracer := provider.Tracer("shop")
for i := 0; i < 1000; i++ {
_, span := tracer.Start(context.Background(), "POST /orders")
span.End()
}
fmt.Println("requests: 1000, traces recorded:", len(rec.Ended()))
}
requests: 1000, traces recorded: 93
About 100 of 1000; the exact number changes every run because trace ids are random (other runs here printed 99 and 104). The decision is written into the traceparent flags (01 / 00) so the next service follows it.
OpenTelemetry in real apps
- Your code calls the API, the SDK processes spans, and an exporter sends them out. The opentelemetry-go README lists the OTLP exporter (OpenTelemetry's own protocol, for logs, metrics and traces), plus Prometheus, stdout and Zipkin exporters.
- Spans are sent to a tracing backend that stores them and draws the timeline (the video names Jaeger and Grafana Tempo as examples).
- You rarely start HTTP spans by hand: instrumentation libraries (named in the video) wrap
net/http, gRPC anddatabase/sqland create spans and do the Inject/Extract for you. You add spans for your own important steps (db.insert_order,kitchen.cook). - Put the trace id in your log lines too (the mini project does): from a log line you can jump to its trace, and back.
Health checks: liveness and readiness
Real-life example: a clinic asks two different questions about a doctor.
| At the clinic | Health check | Endpoint | If it fails |
|---|---|---|---|
| Is the doctor alive and awake? | liveness: is the process alive, or stuck? | /healthz |
restart the instance |
| Can the doctor see patients right now? (not while setting up or leaving) | readiness: should this instance get traffic now? | /readyz |
stop sending it requests (don't restart) |
A health check is an HTTP endpoint that a load balancer (the thing that spreads requests over instances) or an orchestrator (named in the video: Kubernetes) calls every few seconds. An instance (or pod) is one running copy of your service.
The episode's program (HEALTH)
main.go (standard library only):
package main
import (
"expvar"
"fmt"
"io"
"net/http"
"net/http/httptest"
"strings"
"sync/atomic"
)
var (
dbReady atomic.Bool // set once the database answers
draining atomic.Bool // set when SIGTERM arrives during a deploy
orders = expvar.NewInt("orders_placed") // shows up at /debug/vars
)
func main() {
mux := http.NewServeMux()
// liveness: "is the process alive?" Keep it dumb: if this fails, restart me.
mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprint(w, "ok")
})
// readiness: "should I get traffic right now?"
mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
if !dbReady.Load() || draining.Load() {
http.Error(w, "not ready", http.StatusServiceUnavailable)
return
}
fmt.Fprint(w, "ready")
})
mux.HandleFunc("/orders", func(w http.ResponseWriter, r *http.Request) { orders.Add(1) })
mux.Handle("/debug/vars", expvar.Handler())
srv := httptest.NewServer(mux)
defer srv.Close()
get := func(path string) string {
resp, err := http.Get(srv.URL + path)
if err != nil {
return err.Error()
}
defer resp.Body.Close()
b, _ := io.ReadAll(resp.Body)
return fmt.Sprintf("%d %s", resp.StatusCode, strings.TrimSpace(string(b)))
}
fmt.Println("starting up: healthz", get("/healthz"), "| readyz", get("/readyz"))
dbReady.Store(true)
fmt.Println("db connected: healthz", get("/healthz"), "| readyz", get("/readyz"))
get("/orders")
get("/orders")
draining.Store(true)
fmt.Println("SIGTERM: healthz", get("/healthz"), "| readyz", get("/readyz"))
vars := get("/debug/vars")
i := strings.Index(vars, `"orders_placed"`)
fmt.Println("expvar:", vars[i:i+len(`"orders_placed": 2`)], "(plus cmdline and memstats)")
}
starting up: healthz 200 ok | readyz 503 not ready
db connected: healthz 200 ok | readyz 200 ready
SIGTERM: healthz 200 ok | readyz 503 not ready
expvar: "orders_placed": 2 (plus cmdline and memstats)
- Starting up: alive (
200 ok) but not ready (503 not ready): the database isn't connected yet, so no traffic. - DB connected:
200and200. - SIGTERM (the "please stop" signal a deploy sends): still alive, but readiness says 503, so the load balancer stops sending new requests while the old ones finish. This is called draining.
503is Service Unavailable;http.Errorwrites the status and the text.atomic.Boolflags are safe to read from handler goroutines whilemainchanges them.
expvar
expvar is a standard-library package that publishes variables as JSON at /debug/vars. expvar.NewInt("orders_placed") creates a counter you .Add(1) to; expvar.Handler() serves the page (or importing expvar registers it on http.DefaultServeMux automatically). It also publishes cmdline (the program's arguments) and memstats (memory statistics). Above it showed "orders_placed": 2. Quick, no setup, no labels, no histograms: fine for a small tool, while Prometheus metrics are the usual choice for services.
Health check rules
- Liveness: cheap, no database check. If it checked the database, a short database blip would make every instance fail liveness and get restarted at once, making the outage worse.
- Readiness: dependencies OK? still warming up? draining?
- On SIGTERM: fail readiness, finish the requests in progress, then exit (graceful shutdown, episode 74).
- Keep
/debug/varsand/metricsoff the public internet.
Observability in one page
| Signal | Job | Go tool |
|---|---|---|
| Metrics | raise the alarm: RED for services, USE for resources | client_golang: counter ↑, gauge ↕, histogram buckets → p99, few label values |
| Traces | find where the time went | OpenTelemetry: trace id + spans + traceparent propagation |
| Logs | explain why | log/slog JSON, request_id via context, levels, no secrets |
| Health | alive vs ready | /healthz, /readyz, expvar at /debug/vars |
Mini project: observable order API
Make Raju's order API observable:
- a slog JSON middleware with a
request_id(keep the caller'sX-Request-IDheader if it sent one, and send it back); - RED metrics per route: a counter (route, code) and a histogram (route);
/metrics; - spans for the handler, the DB call and the kitchen call (with
traceparentpropagation to the kitchen service); /healthzand/readyzwith a ready flag and a draining flag;- make one route slow on purpose and find it three ways: in the metrics, in a trace, and in the logs.
Full solution
main.go:
package main
import (
"bufio"
"context"
"fmt"
"log/slog"
"net/http"
"net/http/httptest"
"os"
"strconv"
"strings"
"sync/atomic"
"time"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
"go.opentelemetry.io/otel/propagation"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
"go.opentelemetry.io/otel/sdk/trace/tracetest"
)
// ---------- tracing ----------
var (
spans = tracetest.NewSpanRecorder()
provider = sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(spans))
tracer = provider.Tracer("order-api")
prop = propagation.TraceContext{}
)
// ---------- metrics (RED) ----------
var (
reqTotal = prometheus.NewCounterVec(prometheus.CounterOpts{Name: "http_requests_total",
Help: "Requests by route and status code."}, []string{"route", "code"}) // Rate + Errors
reqDur = prometheus.NewHistogramVec(prometheus.HistogramOpts{Name: "http_request_duration_seconds",
Help: "Request duration by route.", Buckets: []float64{0.05, 0.25}}, []string{"route"}) // Duration
)
// ---------- logging ----------
type ctxKey struct{}
func logger(ctx context.Context) *slog.Logger {
if l, ok := ctx.Value(ctxKey{}).(*slog.Logger); ok {
return l
}
return slog.Default()
}
type statusRecorder struct {
http.ResponseWriter
code int
}
func (s *statusRecorder) WriteHeader(c int) { s.code = c; s.ResponseWriter.WriteHeader(c) }
var nextID atomic.Int64
// observe wraps one route with a span, a request id, a log line and RED metrics.
func observe(route string, next http.HandlerFunc) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
id := r.Header.Get("X-Request-ID") // keep the caller's id if it sent one
if id == "" {
id = fmt.Sprint("req-", 100+nextID.Add(1))
}
w.Header().Set("X-Request-ID", id)
ctx := prop.Extract(r.Context(), propagation.HeaderCarrier(r.Header))
ctx, span := tracer.Start(ctx, r.Method+" "+route)
defer span.End()
l := slog.Default().With("request_id", id, "trace_id", span.SpanContext().TraceID().String()[:8])
ctx = context.WithValue(ctx, ctxKey{}, l)
rec := &statusRecorder{w, http.StatusOK}
start := time.Now()
next(rec, r.WithContext(ctx))
took := time.Since(start)
reqTotal.WithLabelValues(route, strconv.Itoa(rec.code)).Inc()
reqDur.WithLabelValues(route).Observe(took.Seconds())
l.Info("request", "method", r.Method, "route", route, "status", rec.code, "ms", took.Milliseconds())
})
}
// ---------- the app ----------
var kitchenURL string
func menu(w http.ResponseWriter, r *http.Request) { fmt.Fprint(w, "chai, samosa") }
func placeOrder(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
if r.URL.Query().Get("qty") == "0" {
logger(ctx).Warn("bad order", "reason", "qty must be 1-20")
http.Error(w, "bad qty", http.StatusBadRequest)
return
}
_, db := tracer.Start(ctx, "db.insert_order")
time.Sleep(10 * time.Millisecond)
db.End()
ctx, call := tracer.Start(ctx, "call kitchen")
req, _ := http.NewRequestWithContext(ctx, "POST", kitchenURL, nil)
prop.Inject(ctx, propagation.HeaderCarrier(req.Header)) // the trace continues in the kitchen
resp, err := http.DefaultClient.Do(req)
call.End()
if err != nil {
logger(ctx).Error("kitchen call failed", "err", err)
http.Error(w, "kitchen down", http.StatusBadGateway)
return
}
resp.Body.Close()
w.WriteHeader(http.StatusCreated)
}
func kitchen(w http.ResponseWriter, r *http.Request) {
ctx := prop.Extract(r.Context(), propagation.HeaderCarrier(r.Header))
_, span := tracer.Start(ctx, "kitchen.cook")
time.Sleep(120 * time.Millisecond) // the slow part, on purpose
span.End()
}
func noTime(_ []string, a slog.Attr) slog.Attr {
if a.Key == slog.TimeKey {
return slog.Attr{}
}
return a
}
func main() {
slog.SetDefault(slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{ReplaceAttr: noTime})))
reg := prometheus.NewRegistry()
reg.MustRegister(reqTotal, reqDur)
kitchenSrv := httptest.NewServer(http.HandlerFunc(kitchen))
defer kitchenSrv.Close()
kitchenURL = kitchenSrv.URL
var ready, draining atomic.Bool
mux := http.NewServeMux()
mux.Handle("GET /menu", observe("/menu", menu))
mux.Handle("POST /orders", observe("/orders", placeOrder))
mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) { fmt.Fprint(w, "ok") })
mux.HandleFunc("GET /readyz", func(w http.ResponseWriter, r *http.Request) {
if !ready.Load() || draining.Load() {
http.Error(w, "not ready", http.StatusServiceUnavailable)
return
}
fmt.Fprint(w, "ready")
})
mux.Handle("GET /metrics", promhttp.HandlerFor(reg, promhttp.HandlerOpts{}))
srv := httptest.NewServer(mux)
defer srv.Close()
status := func(method, path string, hdr ...string) int {
req, _ := http.NewRequest(method, srv.URL+path, nil)
if len(hdr) == 2 {
req.Header.Set(hdr[0], hdr[1])
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return 0
}
resp.Body.Close()
return resp.StatusCode
}
fmt.Println("== health")
fmt.Println("startup healthz", status("GET", "/healthz"), "readyz", status("GET", "/readyz"))
ready.Store(true)
fmt.Println("ready healthz", status("GET", "/healthz"), "readyz", status("GET", "/readyz"))
fmt.Println("== logs")
status("GET", "/menu")
status("GET", "/menu")
status("POST", "/orders")
status("POST", "/orders?qty=0", "X-Request-ID", "mobile-42")
fmt.Println("== metrics (way 1: which route is slow?)")
resp, err := http.Get(srv.URL + "/metrics")
if err != nil {
panic(err)
}
sc := bufio.NewScanner(resp.Body)
for sc.Scan() {
if line := sc.Text(); strings.HasPrefix(line, "http_") && !strings.Contains(line, "_sum") {
fmt.Println(line)
}
}
resp.Body.Close()
fmt.Println("== traces (way 2: where inside the route?)")
for _, s := range spans.Ended() {
fmt.Printf("trace %s… span %s parent %-16s %-16s %v\n", s.SpanContext().TraceID().String()[:8],
s.SpanContext().SpanID(), s.Parent().SpanID(), s.Name(), s.EndTime().Sub(s.StartTime()).Round(5*time.Millisecond))
}
draining.Store(true) // SIGTERM arrives
fmt.Println("== SIGTERM: healthz", status("GET", "/healthz"), "readyz", status("GET", "/readyz"))
}
go.mod:
module shop
go 1.24
require (
github.com/prometheus/client_golang v1.23.2
go.opentelemetry.io/otel/sdk v1.39.0
)
== health
startup healthz 200 readyz 503
ready healthz 200 readyz 200
== logs
{"level":"INFO","msg":"request","request_id":"req-101","trace_id":"c7391cec","method":"GET","route":"/menu","status":200,"ms":0}
{"level":"INFO","msg":"request","request_id":"req-102","trace_id":"dfd97f3c","method":"GET","route":"/menu","status":200,"ms":0}
{"level":"INFO","msg":"request","request_id":"req-103","trace_id":"ec21a898","method":"POST","route":"/orders","status":201,"ms":131}
{"level":"WARN","msg":"bad order","request_id":"mobile-42","trace_id":"32e61c33","reason":"qty must be 1-20"}
{"level":"INFO","msg":"request","request_id":"mobile-42","trace_id":"32e61c33","method":"POST","route":"/orders","status":400,"ms":0}
== metrics (way 1: which route is slow?)
http_request_duration_seconds_bucket{route="/menu",le="0.05"} 2
http_request_duration_seconds_bucket{route="/menu",le="0.25"} 2
http_request_duration_seconds_bucket{route="/menu",le="+Inf"} 2
http_request_duration_seconds_count{route="/menu"} 2
http_request_duration_seconds_bucket{route="/orders",le="0.05"} 1
http_request_duration_seconds_bucket{route="/orders",le="0.25"} 2
http_request_duration_seconds_bucket{route="/orders",le="+Inf"} 2
http_request_duration_seconds_count{route="/orders"} 2
http_requests_total{code="200",route="/menu"} 2
http_requests_total{code="201",route="/orders"} 1
http_requests_total{code="400",route="/orders"} 1
== traces (way 2: where inside the route?)
trace c7391cec… span 20e1cce57f6ba4cd parent 0000000000000000 GET /menu 0s
trace dfd97f3c… span 3148281a422cb468 parent 0000000000000000 GET /menu 0s
trace ec21a898… span f028648e1917fd9b parent 7b636908742f775b db.insert_order 10ms
trace ec21a898… span e2a1bb697f70e773 parent c45f98b40c707945 kitchen.cook 120ms
trace ec21a898… span c45f98b40c707945 parent 7b636908742f775b call kitchen 120ms
trace ec21a898… span 7b636908742f775b parent 0000000000000000 POST /orders 130ms
trace 32e61c33… span f99a0af2d536ac96 parent 0000000000000000 POST /orders 0s
== SIGTERM: healthz 200 readyz 503
(Trace ids, span ids and the milliseconds change every run.)
Finding the slow route three ways:
- Metrics:
/menuhas both requests in thele="0.05"bucket;/ordershas only 1 of 2 under 50 ms (the other, the 400, returned early) and needed the0.25bucket./ordersis slow. - Trace: in trace
ec21a898…,POST /orderstook 130 ms, of whichcall kitchentook 120 ms and its childkitchen.cook(in the kitchen service, same trace id thanks totraceparent) took 120 ms. The kitchen is the slow part. - Logs:
req-103shows"route":"/orders","status":201,"ms":131, and itstrace_idleads straight to the trace. Themobile-42lines show the caller's ownX-Request-IDwas kept, and the WARN line and the request line share it.
Design notes: validation runs first, so a bad order (400) costs nothing; the route label is the pattern (/orders), not the full URL, so cardinality stays low; the logger carries both request_id and trace_id; /healthz, /readyz and /metrics are not wrapped in observe, so health checks don't flood the logs and metrics.
Common mistakes (and the real messages)
| Mistake | What you see | Why | Fix |
|---|---|---|---|
log.Printf sentences |
order 81 failed for user asha after 2100ms (can't query user=asha AND ms>2000) |
no fields | slog with key-value pairs |
| no request id | lines from 100 requests mixed up | nothing ties them together | logger.With("request_id", id) in the context |
| odd number of slog arguments | !BADKEY=status |
keys and values must come in pairs | "status", 201 |
| a counter that goes down | panic: counter cannot decrease in value |
counters only go up | a gauge, or a second counter |
| user id as a metric label | orders_by_user_total series: 5000 (and growing) |
high cardinality | few, fixed label values; ids go in logs and traces |
| wrong number of label values | panic: inconsistent label cardinality: expected 1 label values but got 2 in []string{"ok", "upi"} |
one value per label name | match the label list |
| registering a metric twice | panic: duplicate metrics collector registration attempted |
MustRegister in a function that runs more than once |
register once, at startup |
| watching only averages | average: 39.85ms while p99: 2s |
averages hide the tail | histograms → p99 |
not propagating traceparent |
kitchen got traceparent: (empty), two different trace ids |
the next service starts a new trace | Inject / Extract, or instrumentation libraries |
a malformed traceparent |
valid=false ... trace=00000000000000000000000000000000 |
32 hex trace id, 16 hex parent id required | let the propagator write it |
| liveness checks the database | every instance restarted in a DB blip | wrong question | DB checks go in readiness |
| logging secrets | passwords and tokens in log storage | logs are widely read and kept | never log them |
The duplicate-registration program:
package main
import "github.com/prometheus/client_golang/prometheus"
func main() {
c := prometheus.NewCounter(prometheus.CounterOpts{Name: "orders_total", Help: "Orders."})
reg := prometheus.NewRegistry()
reg.MustRegister(c)
reg.MustRegister(c) // registered twice, e.g. in a handler that runs per request
}
panic: duplicate metrics collector registration attempted
Practice
- Change the LOGS program so the middleware also sends the request id back to the client in an
X-Request-IDresponse header. Where mustw.Header().Setgo, before or afternext.ServeHTTP? - In the METRICS run, how many requests took between 100 ms and 500 ms? Which two lines tell you?
- Your API has routes
/menu,/orders,/orders/{id}. A teammate labelshttp_requests_totalwithr.URL.Path. What goes wrong, and what should the label be? - Which RED letter and which USE letter would show "the DB pool has 9 of 10 connections busy and 4 requests waiting"?
- A service receives
traceparent: 00-11b5c5e1aa9367dc12e09556aadc27ca-8d5791a2829506ed-00. Is this request's trace recorded? Is it the same trace as one with flags01and the same trace id? - Your
/healthzpings the database. The database restarts for 20 seconds. What happens to your 30 instances? Fix it.
Solutions
1. Before. Once the handler writes the status or the body, headers are already sent. Put w.Header().Set("X-Request-ID", id) before next.ServeHTTP, exactly as the mini project's observe does (its run shows request_id: mobile-42 kept from the caller).
2. 2 requests. le="0.1"} 18 and le="0.5"} 20: buckets are cumulative, so 20 − 18 = 2.
3. /orders/1, /orders/2, ... each become a new series: unbounded cardinality, like the user_id run (5000 series for 5000 users). Use the route pattern (/orders/{id}) as the label, as the mini project does with observe("/orders", ...).
4. It's a resource, so USE: Utilization (9 of 10 in use) and Saturation (4 waiting). The GaugeFunc run above printed db_pool_in_use 9 and db_pool_waiting 4. RED's Duration would show the effect on users (slower requests).
5. Not recorded: flags 00 = not sampled (the parse run printed sampled=false). Yes, same trace id, so it's the same trace; the flag only says whether it is being recorded.
6. All 30 fail liveness and get restarted together, in the middle of the outage, and restarting instances can't serve anything. Keep /healthz returning ok without dependencies, and check the database in /readyz: instances stop getting traffic for 20 seconds, then come back without restarting.
Quick quiz
1. Orders placed since the server started; customers waiting in the queue right now; how long each payment took. Which metric types? (From the video.)
A. gauge, counter, histogram · B. counter, gauge, histogram · C. counter, counter, gauge · D. histogram, gauge, counter
Answer
B. Orders placed only goes up: a counter. Customers waiting goes up and down: a gauge. Payment durations are many values to spread over ranges: a histogram, which gives percentiles like p99.
2. A histogram prints _bucket{le="0.1"} 98 and _bucket{le="1"} 100. How many observations were above 0.1 s?
Answer
2. Buckets count values at or below le, so 100 − 98 = 2 (the two 0.9 s payments in the types run).
3. During a deploy, an instance gets SIGTERM. What should /healthz and /readyz return?
Answer
/healthz 200 (still alive, don't restart it) and /readyz 503 (stop sending new traffic while it finishes), exactly as the HEALTH run printed.
4. In the TRACE output, why is POST /orders printed after its children?
Answer
recorder.Ended() lists spans in the order they ended, and a parent ends after its children. The tree comes from the parent ids, not from the print order.
Interview questions
- What is observability, and what are its three pillars? Understanding a running system's inside from its outputs: logs (events), metrics (numbers over time), traces (one request across services). Metrics alert, traces locate, logs explain.
- Why structured logging? Fields can be searched, filtered, counted and alerted on; sentences need fragile grep. In Go:
log/slog(Go 1.21+),JSONHandlerin production. - How do you get a request id into every log line? A middleware creates
slog.Default().With("request_id", id), stores it in the request context, and handlers fetch it fromr.Context(). Return the id in a response header. - Counter vs gauge vs histogram vs summary? Only-up total; up-and-down current value; bucket counts (+ sum, count) from which percentiles are computed on the server and which can be aggregated; in-process percentiles that can't be aggregated across instances.
- What is label cardinality and why does it matter? The number of series = label-value combinations; unbounded values (user ids, raw paths) explode memory in the app and in Prometheus.
- Why p99 and not the average? Averages hide the slow tail; p99 shows what the slowest 1% experience.
- Explain RED and USE. RED for services: Rate, Errors, Duration. USE for resources: Utilization, Saturation, Errors.
- What is a span, and how does a trace cross services? A named, timed unit of work with a parent; the trace context is injected into an outgoing request's
traceparentheader and extracted by the receiver, so its spans join the same trace. - What is in a traceparent header?
version-traceid(32 hex)-parentid(16 hex)-flags, e.g.00-...-...-01, where01= sampled. - Why sample traces? Cost: recording everything is expensive; a ratio sampler keeps a fraction, and the decision travels in the flags.
- Liveness vs readiness? Liveness: alive or restart me (keep it cheap, no dependencies). Readiness: send me traffic now (dependencies, warm-up, draining).
- What does expvar give you? JSON variables at
/debug/varsfrom the standard library, pluscmdlineandmemstats, with no setup.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Observability | being able to understand what happens inside a running system from the signals it gives out | the 2 A.M. dashboard |
| Monitoring | watching signals and raising alarms when they cross a limit | "errors above 1%" alert |
| Signal / telemetry | data a program gives out about itself | logs, metrics, traces |
| Three pillars | the three main kinds of signal | logs, metrics, traces |
| Production | the real system real customers use | Raju's live order API |
| Log | a record of one event | payment failed |
| Unstructured logging | log lines as free sentences | log.Printf("order %d failed...") |
| Structured logging | log lines as named key-value fields | {"msg":"order failed","user":"asha"} |
| Key-value pair | a name and its value | "ms", 2100 → ms=2100 |
| JSON | a text format of {"key": value} objects |
{"level":"ERROR"} |
log/slog |
Go's structured logging package (Go 1.21+) | slog.Info("hello", "count", 3) |
| Logger | the object you call to write log lines | *slog.Logger |
| Handler (slog) | decides the format and destination of log lines | slog.NewJSONHandler(os.Stdout, nil) |
TextHandler |
writes key=value lines |
level=INFO msg=start |
JSONHandler |
writes one JSON object per line | {"level":"INFO",...} |
| Default logger | the logger used by slog.Info etc. |
slog.SetDefault(...) |
| Level | importance of a log line | Debug < Info < Warn < Error |
HandlerOptions.Level |
the minimum level a handler prints | slog.LevelDebug |
ReplaceAttr |
a function that changes or removes fields | drop time |
| Attr | one key-value field in slog | slog.Attr{Key: "time"} |
With |
returns a child logger that adds fields to every line | .With("request_id", "req-7") |
!BADKEY |
slog's marker for a value with no key | !BADKEY=status |
| Request id | a unique name for one request, in every line it writes | req-102 |
X-Request-ID |
a common HTTP header that carries a request id | mobile-42 |
| Middleware | a function wrapping a handler to run code before and after it | withLogging, instrument |
Context (context.Context) |
request-scoped values and deadlines that travel with a request | r.Context() |
| Context-aware logging | taking the logger (with request fields) from the context | logger(r.Context()).Error(...) |
| Context key | a private type used to store a value in a context | type ctxKey struct{} |
| Status code | the HTTP result number | 200 OK, 201 Created, 400, 500, 502, 503 |
| Response writer wrapper | a type that wraps http.ResponseWriter to remember the status |
recorder, statusRecorder |
atomic.Int64 / atomic.Bool |
values safe to change from many goroutines | nextID.Add(1), draining.Store(true) |
httptest.NewServer |
a real HTTP server on a local port, for tests and demos | srv.URL |
| Metric | a named number measured over time | http_requests_total |
| Time series (series) | one metric with one exact set of label values, over time | http_requests_total{code="500"} |
| Counter | a metric that only goes up | orders_placed_total 3 |
| Gauge | a metric that goes up and down | customers_waiting 4 |
GaugeFunc |
a gauge whose value comes from a function at scrape time | db_pool_in_use 9 |
| Histogram | counts observations into buckets, plus sum and count | payment_seconds_bucket{le="0.1"} 98 |
| Bucket | a range "at or below X" in a histogram | le="0.5" |
le |
"less than or equal": the bucket's upper limit | le="+Inf" = everything |
| Cumulative | each bucket includes all smaller ones | 18, 18, 18, 20, 20 |
_sum / _count |
total of all observed values / number of observations | sum ÷ count = average |
| Summary | computes percentiles inside the app; can't be aggregated | quantile="0.99"} 0.9 |
| Objectives | a summary's percentiles and allowed errors | {0.5: 0.05, 0.99: 0.001} |
| Observe | record one value in a histogram or summary | duration.Observe(0.12) |
| Inc / Dec / Add / Set | change a counter or gauge | inFlight.Inc() |
| Label | a key=value tag that splits a metric into series | code="500" |
CounterVec / HistogramVec |
a counter / histogram with labels | WithLabelValues("/orders", "200") |
| Cardinality | number of different series (label-value combinations) | 2 for status, 5000 for user_id |
| High cardinality | a label with too many (or unbounded) values | user ids, order ids, raw URLs |
| Prometheus | a popular system that scrapes and stores metrics | scrapes /metrics |
| client_golang | Prometheus's Go library | github.com/prometheus/client_golang |
| Registry | the set of metrics a handler exposes | prometheus.NewRegistry() |
MustRegister |
add metrics to a registry; panics on problems | reg.MustRegister(requests) |
promhttp.HandlerFor |
the HTTP handler that writes a registry's metrics | mux.Handle("/metrics", ...) |
/metrics endpoint |
the URL where a service shows its metrics as text | # TYPE ... counter |
# HELP / # TYPE |
description and type lines in the metrics text | # TYPE payment_seconds histogram |
| Scrape | Prometheus fetching /metrics every few seconds |
GET /metrics |
| PromQL | Prometheus's query language | rate(...), histogram_quantile |
| Rate | change per second of a counter | errors per second |
| Alert | a notification when a query crosses a limit | p99 > 500 ms |
| Latency / duration | how long a request takes | 120 ms |
| Percentile | the value below which X% of observations fall | p99 |
| p50 (median) | half of the requests were faster | p50: 10ms |
| p99 | 99% of requests were faster than this | p99: 2s |
| Tail latency | the slow few requests that averages hide | the 15 two-second requests |
| Average (mean) | sum ÷ count | average: 39.85ms |
| RED method | Rate, Errors, Duration: what to measure for services | http_requests_total + histogram |
| USE method | Utilization, Saturation, Errors: what to measure for resources | pool in use / waiting |
| Utilization | how busy a resource is | 9 of 10 connections |
| Saturation | how much work is waiting for a resource | 4 requests waiting |
| Connection pool | a set of reusable database connections | *sql.DB (episode 67) |
| Trace | the whole journey of one request across services | trace 8c5d6237… |
| Distributed tracing | tracing a request across many services | order → kitchen |
| Span | one named, timed step of work, with a parent | kitchen.cook 120ms |
| Root span | the first span of a trace; it has no parent | POST /orders (parent all zeros) |
| Parent / child span | a span started inside another span | db.insert_order is a child of POST /orders |
| Trace id | 32-hex id shared by every span of a trace | 8c5d6237cdf6... |
| Span id | 16-hex id of one span | 991a961f6cc4091d |
| OpenTelemetry (OTel) | open standard and libraries for traces, metrics, logs | go.opentelemetry.io/otel |
| API (OTel) | what your code calls to make spans | tracer.Start |
| SDK (OTel) | processes, samples and exports spans | go.opentelemetry.io/otel/sdk |
| TracerProvider | creates tracers and holds span processing settings | sdktrace.NewTracerProvider(...) |
| Tracer | the object that starts spans | provider.Tracer("shop") |
| Span processor | receives spans as they start and end | WithSpanProcessor(recorder) |
tracetest.SpanRecorder |
a processor that keeps finished spans in memory (for tests) | recorder.Ended() |
| Exporter | sends finished spans/metrics/logs to a backend | OTLP, Zipkin, stdout |
| OTLP | OpenTelemetry's own protocol for sending signals | the OTLP exporter |
| Tracing backend | stores spans and draws timelines | Jaeger, Grafana Tempo (named in the video) |
| Instrumentation library | wraps a library (HTTP, gRPC, SQL) to create spans for you | an HTTP wrapper |
| Propagation | passing the trace context to the next service | prop.Inject / prop.Extract |
| Propagator | the code that writes and reads the trace context headers | propagation.TraceContext{} |
HeaderCarrier |
adapts http.Header for a propagator |
propagation.HeaderCarrier(req.Header) |
| Inject | write the trace context into outgoing headers | before http.DefaultClient.Do(req) |
| Extract | read the trace context from incoming headers | first line of kitchen |
| W3C Trace Context | the web standard for trace headers | traceparent, tracestate |
traceparent |
header version-traceid-parentid-flags |
00-8c5d...-991a...-01 |
| Hex (hexadecimal) | digits 0-9 and a-f; 2 hex characters = 1 byte | 8d |
| Trace flags | 01 sampled, 00 not sampled |
-01 |
| Sampling | recording only some traces to save cost | TraceIDRatioBased(0.1) |
| Remote span context | a trace context that came from another service | remote=true |
| Health check | an endpoint that reports whether an instance is OK | /healthz |
| Liveness | "is the process alive?"; failing means restart | /healthz → 200 ok |
| Readiness | "should I get traffic now?"; failing means no traffic | /readyz → 503 not ready |
| Instance / pod | one running copy of a service | one of 30 copies |
| Load balancer | spreads requests over instances | skips not-ready instances |
| Orchestrator | runs and restarts instances | Kubernetes (named in the video) |
| SIGTERM | the "please stop" signal sent during a deploy | draining.Store(true) |
| Draining | finishing current requests while refusing new ones | readyz 503 after SIGTERM |
| Graceful shutdown | drain, finish, then exit (episode 74) | fail readiness first |
| 503 Service Unavailable | "can't serve right now" | http.StatusServiceUnavailable |
| expvar | standard package publishing variables as JSON at /debug/vars |
expvar.NewInt("orders_placed") |
/debug/vars |
expvar's endpoint; also shows cmdline and memstats |
"orders_placed": 2 |
| Panic | a crash that stops the program unless recovered | counter cannot decrease in value |
recover |
catches a panic in a deferred function | the cardinality demo |
Summary
- Observability = seeing inside a running system from its signals: metrics raise the alarm, traces find where, logs explain why.
- Logs:
log/slog(Go 1.21+) withJSONHandler, levels, stable keys, no secrets; a middleware putsWith("request_id", id)into the context, so every line of a request shares its id. - Metrics with client_golang: counter (only up), gauge (up and down), histogram (cumulative buckets → p99, aggregatable), summary (in-app percentiles, not aggregatable); serve them at
/metricsfor Prometheus to scrape; keep label cardinality low. - Measure RED (Rate, Errors, Duration) for services and USE (Utilization, Saturation, Errors) for resources.
- Traces with OpenTelemetry: one trace id, a tree of spans, carried across services by the W3C
traceparentheader (Inject/Extract); sample to save cost. - Health checks: cheap liveness (
/healthz, restart me) vs readiness (/readyz, send me traffic; fails while starting and draining); expvar gives quick JSON at/debug/vars.
Next episode: Configuration and dependency injection: twelve-factor config, flags, environment and files, config validation, functional options, constructor injection, wiring in main.