SyntaxPilot
Courses/Go Programming/Lesson 58
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Go Programming · Lesson 58 of 89

Execution tracing and performance tuning

execution tracerruntime/tracego tool traceflight recorderlatency vs throughputreducing allocationspreallocating slices and mapsstrings.Builder and bytes.Buffersync.Pool use casesinlining and the inlining budgetbounds-check elimination-gcflags=-d=ssa/check_bcemeasure first and avoid premature optimisation

These are the notes for Go episode 58, the last of the performance block. Eight orders take seven times longer than they should, yet the CPU profile is empty: "Total samples = 0". Waiting uses no CPU, so a profiler can't see it. The execution tracer records every event instead; with runtime/trace tasks and regions and go tool trace we find the queue for one stove. Then the flight recorder (Go 1.25), latency vs throughput, and four tuning techniques, each measured: preallocating slices and maps, sync.Pool, inlining and its budget, and bounds-check elimination. The fifth technique: measure first.

All outputs here are from real runs (Go 1.24.7, linux/amd64, a machine with 2 CPUs). Timings change between runs; allocation counts and compiler messages don't.

Facts and sources

  • Go 1.25 release notes: "The new runtime/trace.FlightRecorder API provides a lightweight way to capture a runtime execution trace by continuously recording the trace into an in-memory ring buffer. When a significant event occurs, a program can call FlightRecorder.WriteTo to snapshot the last few seconds of the trace to a file." The time and amount of data "may be configured within the FlightRecorderConfig".
  • go tool trace -h (Go 1.24.7): views in a web browser; -pprof=TYPE for net, sync, syscall, sched profiles.
  • The inlining budget of 80 and the costs are the compiler's own -gcflags=-m=2 messages, shown below.

What profiles can't see

// $ go test -run Rush -v -cpuprofile cpu.prof
// $ go tool pprof -top -nodecount=3 cpu.prof
// ---- kitchen.go ----
package shop

import (
    "sync"
    "time"
)

var stove sync.Mutex // one stove for the whole kitchen

func Cook(order int) {
    time.Sleep(time.Millisecond) // chop
    stove.Lock()
    time.Sleep(3 * time.Millisecond) // fry
    stove.Unlock()
}

// ---- kitchen_test.go ----
package shop

import (
    "sync"
    "testing"
    "time"
)

func TestRush(t *testing.T) {
    start := time.Now()
    var wg sync.WaitGroup
    for order := 1; order <= 8; order++ {
        wg.Add(1)
        go func() { defer wg.Done(); Cook(order) }()
    }
    wg.Wait()
    t.Log("8 orders in", time.Since(start).Round(time.Millisecond))
}

Real output:

=== RUN   TestRush
    kitchen_test.go:17: 8 orders in 28ms
--- PASS: TestRush (0.03s)
PASS
ok      shop    0.204s
$ go tool pprof -top -nodecount=3 cpu.prof
File: shop.test
Type: cpu
Duration: 201.16ms, Total samples = 0
Showing nodes accounting for 0, 0% of 0 total

Expected about 4 ms (1 chop + 3 fry, all in parallel); got 28 ms. A CPU profile samples only running code; sleeping and waiting for a lock use no CPU.

The execution tracer

Watching the kitchen Watching a Go program
photos every few seconds show only busy cooks CPU profile: samples running code
a CCTV recording: every moment, every cook, even standing still execution trace: every runtime event with a timestamp
labels on the recording: "order 4: waiting for the stove" tasks and regions: your own names on the timeline
  • The execution tracer records events: a goroutine is created, starts running, stops, blocks (on a lock, channel, I/O, sleep), is unblocked; GC phases; system calls; processors starting and stopping.
  • runtime/trace: trace.Start(w) / trace.Stop() record into any io.Writer; go test -trace trace.out does it without code changes.
  • Task: one logical piece of work (trace.NewTask(ctx, "order"), ended with task.End()); region: a named step inside a goroutine (trace.WithRegion(ctx, "fry", f), or trace.StartRegion).
// $ go run .
// ---- main.go ----
package main

import (
    "context"
    "fmt"
    "os"
    "runtime/trace"
    "sync"
    "time"
)

var stove sync.Mutex // one stove for the whole kitchen

func cook(ctx context.Context, order int) {
    ctx, task := trace.NewTask(ctx, "order") // one task per order
    defer task.End()
    trace.WithRegion(ctx, "chop", func() { time.Sleep(time.Millisecond) })
    trace.WithRegion(ctx, "wait for stove", func() { stove.Lock() })
    trace.WithRegion(ctx, "fry", func() { time.Sleep(3 * time.Millisecond) })
    stove.Unlock()
}

func main() {
    f, _ := os.Create("trace.out")
    defer f.Close()
    trace.Start(f) // record everything the runtime does from here
    defer trace.Stop()

    start := time.Now()
    var wg sync.WaitGroup
    for order := 1; order <= 8; order++ {
        wg.Add(1)
        go func() { defer wg.Done(); cook(context.Background(), order) }()
    }
    wg.Wait()
    fmt.Println("8 orders in", time.Since(start).Round(time.Millisecond))
    fmt.Println("cooking time per order: 1 ms chop + 3 ms fry")
}

Real output:

8 orders in 27ms
cooking time per order: 1 ms chop + 3 ms fry

Reading it with go tool trace

go tool trace trace.out starts a local web server. Its pages include View trace by proc (a timeline per processor), Goroutine analysis, User-defined tasks, User-defined regions, and blocking profiles (network, synchronization, syscall, scheduler latency).

  • View trace by proc (real screenshot in the video): Proc 0 and Proc 1 show only thin slivers of running code over ~27 ms. The processors are idle: the program is waiting, not computing.
  • User-defined regions (real data from the same trace):
Region Count Durations
"chop" 8 all in the 1 ms bucket
"fry" 8 all in the 2.5–4 ms bucket
"wait for stove" 8 1 at 251 ns, 1 at ~2.5 ms, 2 at ~6.3 ms, 1 at ~10 ms, 3 at ~15.8 ms+

One stove = one mutex, held while frying: 8 × 3 ms = 24 ms, plus the chop. The fix is more stoves or a shorter critical section (the code that runs while holding the lock), not faster code.

The flight recorder (Go 1.25)

  • A full trace of a busy server is big and costs time to write; the rare slow request comes at an unknown moment.
  • runtime/trace.FlightRecorder keeps the trace in an in-memory ring buffer (a fixed-size buffer where new data overwrites the oldest), so it always holds the last few seconds.
  • When something bad happens, WriteTo saves that window to a file; FlightRecorderConfig sets how much time and data to keep.

The pattern, in words (the channel's compiler is Go 1.24.7, so this isn't run here): create a FlightRecorder with a FlightRecorderConfig, start it when the server starts, and in a request handler, if the request took longer than your limit, open a file and call the recorder's WriteTo with it. Then open that file with go tool trace. See the runtime/trace package documentation for Go 1.25+ for the exact constructor and config fields.

Latency vs throughput

A chai stall A server
how long one customer waits for the first sip latency: time for one request
cups sold per hour throughput: requests per second
a huge pot: many cups, but a long first wait the trade-off: batching raises throughput, hurts latency

Decide which number your users feel (often a latency percentile, like "99% of requests under 200 ms"), and measure that.

Technique 1: fewer allocations — preallocate

// $ go test -bench . -benchmem
// ---- orders.go ----
package shop

func Grow(n int) []int {
    var ids []int // starts empty: append must grow it again and again
    for i := 0; i < n; i++ {
        ids = append(ids, i)
    }
    return ids
}

func Prealloc(n int) []int {
    ids := make([]int, 0, n) // room for n from the start
    for i := 0; i < n; i++ {
        ids = append(ids, i)
    }
    return ids
}

func MapGrow(n int) map[int]bool {
    m := map[int]bool{}
    for i := 0; i < n; i++ {
        m[i] = true
    }
    return m
}

func MapPrealloc(n int) map[int]bool {
    m := make(map[int]bool, n) // size hint
    for i := 0; i < n; i++ {
        m[i] = true
    }
    return m
}

// ---- orders_test.go ----
package shop

import "testing"

func BenchmarkGrow(b *testing.B)        { for i := 0; i < b.N; i++ { Grow(10_000) } }
func BenchmarkPrealloc(b *testing.B)    { for i := 0; i < b.N; i++ { Prealloc(10_000) } }
func BenchmarkMapGrow(b *testing.B)     { for i := 0; i < b.N; i++ { MapGrow(10_000) } }
func BenchmarkMapPrealloc(b *testing.B) { for i := 0; i < b.N; i++ { MapPrealloc(10_000) } }

Real output:

BenchmarkGrow-2                13965        185700 ns/op      357624 B/op         19 allocs/op
BenchmarkPrealloc-2            37429         40037 ns/op       81920 B/op          1 allocs/op
BenchmarkMapGrow-2              1676        724763 ns/op      591480 B/op         79 allocs/op
BenchmarkMapPrealloc-2          4108        246690 ns/op      295552 B/op         33 allocs/op
  • make([]T, 0, n): length 0, capacity n: no regrowing. make(map[K]V, n): a size hint.
  • Text: strings.Builder and bytes.Buffer (a growable byte buffer that also implements io.Reader/io.Writer) both have Grow(n).

Technique 2: reuse — sync.Pool

// $ go test -bench . -benchmem
// ---- receipt.go ----
package shop

import (
    "bytes"
    "strconv"
    "sync"
)

func Render(items []int) int { // a new 128 KB buffer for every receipt
    buf := bytes.NewBuffer(make([]byte, 0, 128<<10))
    for _, p := range items {
        buf.WriteString("item ₹" + strconv.Itoa(p) + "\n")
    }
    return buf.Len()
}

var bufPool = sync.Pool{ // a shelf of used buffers
    New: func() any { return bytes.NewBuffer(make([]byte, 0, 128<<10)) },
}

func RenderPooled(items []int) int {
    buf := bufPool.Get().(*bytes.Buffer) // take one from the shelf
    buf.Reset()                          // wipe the old receipt!
    for _, p := range items {
        buf.WriteString("item ₹" + strconv.Itoa(p) + "\n")
    }
    n := buf.Len()
    bufPool.Put(buf) // put it back for the next receipt
    return n
}

// ---- receipt_test.go ----
package shop

import "testing"

var items = []int{120, 80, 45, 300, 60}

func BenchmarkRender(b *testing.B)       { for i := 0; i < b.N; i++ { Render(items) } }
func BenchmarkRenderPooled(b *testing.B) { for i := 0; i < b.N; i++ { RenderPooled(items) } }

Real output (two runs):

BenchmarkRender-2              13885        107620 ns/op      131078 B/op          3 allocs/op
BenchmarkRenderPooled-2      5087568           262.5 ns/op         6 B/op          2 allocs/op

BenchmarkRender-2              20929         59808 ns/op      131078 B/op          3 allocs/op
BenchmarkRenderPooled-2      6078786           448.4 ns/op         6 B/op          2 allocs/op
  • Get returns a pooled object, or calls New if the pool is empty; Put returns it.
  • Rules: always reset what you Get; never use an object after Put; the GC may empty the pool at any time, so it's a cache, never storage. Use it for big, short-lived, frequently made objects (buffers, encoders).
  • (A 64 KB buffer made with a constant size stayed on the stack in a first try, so there was nothing to pool: measure before pooling.)

Technique 3: inlining and the budget

  • Inlining: the compiler pastes a small function's body into its caller: no call overhead, and the pasted code can be optimised with the caller.
  • Every function gets a cost (roughly its number of operations); functions over the inlining budget (80) are not inlined. PGO raises the budget for hot functions.
// $ go build -gcflags=-m=2 .
// ---- bill.go ----
package shop

func GST(price int) int { return price * 18 / 100 } // small

func Receipt(items []int) (total, bad int) { // bigger
    for _, p := range items {
        switch {
        case p < 0:
            bad++
        case p == 0:
            continue // a free item
        case p > 10_000:
            total += p + GST(p) - p/10 // discount on big items
        default:
            total += p + GST(p)
        }
        if total > 1_000_000 {
            total = 1_000_000
            break
        }
    }
    for i := 0; i < bad; i++ {
        total -= 5 // fine per bad item
    }
    if bad > 3 {
        total = 0 // too many bad items: reject the bill
    }
    return total, bad
}

Real output:

# shop
./bill.go:3:6: can inline GST with cost 6 as: func(int) int { return price * 18 / 100 }
./bill.go:5:6: cannot inline Receipt: function too complex: cost 87 exceeds budget 80
./bill.go:13:29: inlining call to GST
./bill.go:15:29: inlining call to GST
./bill.go:5:14: items does not escape

Without the case p == 0 and the final if, the same function had can inline Receipt with cost 75 (real run). Small functions are free: write them for clarity.

Technique 4: bounds-check elimination

  • Every index d[i] is bounds-checked: if i is outside 0..len(d)-1, Go panics. This makes Go memory-safe, but each check is a compare and a branch.
  • Bounds-check elimination (BCE): the compiler removes checks it can prove can't fail. -gcflags=-d=ssa/check_bce/debug=1 prints each check that remains (Found IsInBounds).
// $ go build -gcflags=-d=ssa/check_bce/debug=1 .
// ---- code.go ----
package shop

// SumCode adds the 4 digits of an order code, checked one by one.
func SumCode(d []int) int {
    return d[0] + d[1] + d[2] + d[3] // 4 bounds checks
}

// SumCodeFast tells the compiler the length once.
func SumCodeFast(d []int) int {
    _ = d[3] // one check: if this works, 0..3 are all fine
    return d[0] + d[1] + d[2] + d[3]
}

// Total ranges over the slice: the compiler knows i is in range.
func Total(prices []int) int {
    t := 0
    for i := range prices {
        t += prices[i]
    }
    return t
}

Real output:

# shop
./code.go:5:13: Found IsInBounds
./code.go:5:20: Found IsInBounds
./code.go:5:27: Found IsInBounds
./code.go:5:34: Found IsInBounds
./code.go:10:10: Found IsInBounds

Four checks in SumCode, one in SumCodeFast (the _ = d[3] hint), none in Total. Only worth it in the hottest loops, after a profile points there.

Technique 5: measure first

  1. Decide the goal: latency or throughput, and a number.
  2. Benchmark it (go test -bench -benchmem).
  3. Find the cause: CPU profile (busy), heap profile (allocations), trace (waiting).
  4. Fix the biggest cause only; measure again. Compare runs with benchstat (episode 42).

Tuning code nobody measured is premature optimisation: harder to read, usually no gain. Keep code clear; tune the hot path (the code that runs most).

Mini project: the fast kitchen

  1. Give the kitchen N stoves with a buffered channel as a semaphore (episode 52) and compare 1, 2, 4 and 8 stoves.
  2. Trace it and look at the "wait for stove" regions.
  3. Pool each order's ticket buffer; benchmark with -benchmem.
  4. Find a function just over budget 80 and split it.
Solution (step 1)
package main

import (
    "fmt"
    "sync"
    "time"
)

func rush(stoves int) time.Duration {
    stove := make(chan struct{}, stoves) // a semaphore: N free stoves
    start := time.Now()
    var wg sync.WaitGroup
    for order := 1; order <= 8; order++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            time.Sleep(time.Millisecond) // chop
            stove <- struct{}{}           // take a stove
            time.Sleep(3 * time.Millisecond) // fry
            <-stove                       // give it back
        }()
    }
    wg.Wait()
    return time.Since(start).Round(time.Millisecond)
}

func main() {
    for _, n := range []int{1, 2, 4, 8} {
        fmt.Printf("%d stove(s): 8 orders in %v\n", n, rush(n))
    }
}

Real output:

1 stove(s): 8 orders in 27ms
2 stove(s): 8 orders in 14ms
4 stove(s): 8 orders in 8ms
8 stove(s): 8 orders in 4ms

Steps 2–4 use exactly the patterns shown above: wrap the stove wait in trace.WithRegion, use the bufPool pattern, and check -gcflags=-m=2 before and after splitting.

Common mistakes (and the real messages)

Mistake What you see Why Fix
CPU profile for a waiting problem Total samples = 0 waiting uses no CPU go tool trace, block profile
slow work while holding a lock wait for stove up to ~16 ms+ everyone queues shorter critical section, more capacity
pooled object not reset the previous receipt's lines appear Get returns used objects buf.Reset()
sync.Pool as storage objects disappear the GC may empty it use it only as a cache
huge function on the hot path cost 87 exceeds budget 80 not inlined split out small helpers
tuning without a benchmark no measurable gain not the hot path measure first

Practice

  1. A request takes 2 s; CPU usage is 3%. Which tool first?
  2. What's the difference between a task and a region in runtime/trace?
  3. Why does make([]int, 0, n) give 1 allocation where var ids []int gives 19?
  4. When would you not use sync.Pool?
Solutions

1. An execution trace (or block profile): the time is spent waiting.

2. A task is a logical operation that can span goroutines (one order); a region is a named time interval within one goroutine (one step).

3. With capacity n, append never has to grow; without it, append repeatedly allocates a bigger array and copies.

4. For small, cheap objects (the pool's overhead is larger than the allocation), for anything that must be kept, and when no benchmark shows allocation cost.

Quick quiz

1. 200 requests/s, each takes 900 ms, CPU 5%, pprof has almost no samples. First tool? (From the video.)

A. a CPU profile again · B. an execution trace · C. raise GOGC · D. inline everything

Answer

B. Low CPU and slow requests means waiting; a trace shows what each goroutine waits for.

2. What is Go's inlining budget?

Answer

80 (cost 87 exceeds budget 80).

3. What does _ = d[3] do before using d[0]..d[3]?

Answer

One bounds check up front, which lets the compiler remove the checks for indexes 0–3.

Interview questions

  1. Profile vs trace? A profile samples where CPU time (or memory) goes; a trace records every event over time, including blocking and scheduling.
  2. How do you take a trace? go test -trace, runtime/trace.Start/Stop, or /debug/pprof/trace via net/http/pprof; view with go tool trace.
  3. What are tasks and regions? User annotations in traces: a task = a logical operation, a region = a named interval in a goroutine.
  4. What is the flight recorder? Go 1.25's runtime/trace.FlightRecorder: keeps the last seconds of trace in a ring buffer and writes them on demand.
  5. Latency vs throughput? Time per request vs requests per time; batching trades one for the other.
  6. How do you reduce allocations? Preallocate (make with capacity, size hints), reuse buffers (sync.Pool), avoid += on strings, keep values on the stack.
  7. When does Go inline a function? When its cost is within the budget (80), or higher for hot functions with PGO; see -gcflags=-m.
  8. What is bounds-check elimination? The compiler removing index checks it can prove are safe; see -d=ssa/check_bce/debug=1.

Glossary

Term Meaning Example
Execution tracer records runtime events with timestamps runtime/trace
Execution trace the recorded events trace.out
Event one thing the runtime did goroutine blocks on a lock
runtime/trace the package that writes traces trace.Start(f)
go test -trace records a trace of a test run -trace trace.out
go tool trace web viewer for traces View trace by proc
Task a logical operation in a trace trace.NewTask(ctx, "order")
Region a named interval inside a goroutine trace.WithRegion(ctx, "fry", f)
Proc (P) a logical processor running goroutines Proc 0, Proc 1
Critical section code that runs while holding a lock frying with the stove locked
Flight recorder keeps the last seconds of trace in memory (Go 1.25) FlightRecorder.WriteTo
Ring buffer fixed-size buffer where new data overwrites the oldest the flight recorder's memory
Latency time for one request 900 ms
Throughput work per unit of time 200 requests/s
Percentile the value under which a share of measurements fall p99 latency
Preallocate reserve room up front make([]int, 0, n)
Capacity how many elements fit before a slice must grow cap(ids)
Size hint expected number of map entries make(map[int]bool, n)
bytes.Buffer a growable byte buffer buf.WriteString
sync.Pool a cache of reusable objects bufPool.Get()
Inlining pasting a function's body into its caller inlining call to GST
Inlining cost the compiler's size estimate of a function cost 6
Inlining budget the maximum cost that is inlined: 80 exceeds budget 80
-gcflags=-m=2 explain inlining and escape decisions in detail go build -gcflags=-m=2 .
Bounds check the run-time check that an index is in range d[3]
Bounds-check elimination (BCE) removing checks proven unnecessary _ = d[3]
check_bce debug flag listing remaining checks Found IsInBounds
Hot path the code that runs most often the inner loop
Premature optimisation tuning before measuring rewriting cold code
benchstat compares benchmark runs statistically episode 42
Semaphore a counter limiting how many may proceed make(chan struct{}, 2)

Summary

  • A CPU profile can't see waiting; the execution tracer records every event. Use runtime/trace (tasks, regions) or go test -trace, and read it with go tool trace.
  • The flight recorder (Go 1.25) keeps the last seconds for rare slow moments.
  • Decide latency or throughput first.
  • Tune, measured: preallocate (make with capacity, size hints, Grow), sync.Pool for big short-lived buffers, small functions get inlined (budget 80), BCE for hot loops.
  • Measure first, fix the biggest cause, measure again.

Next episode: building command-line tools: os.Args, the flag package, subcommands, stdin/stdout/stderr, exit codes, os/exec, signals.