These are the notes for Go episode 57. Raju's closing report is slow. The obvious guess (the sort) is wrong, and timing every line by hand doesn't scale. So we invent a sampling profiler, then use Go's real one, pprof: a CPU profile read with go tool pprof -top and -list, flat vs cum, the flame graph, then a fix that makes the report about 50 times faster. Then the other profiles: heap (alloc_space vs inuse_space), goroutine, block and mutex, profiling a live server with net/http/pprof, and profile-guided optimisation with default.pgo.
All outputs here are from real runs (Go 1.24.7, linux/amd64, a machine with 2 CPUs). Timings and sample counts change between runs; the shape stays the same.
Facts and sources
- Go 1.24.7 source,
runtime/pprof/pprof.go:StartCPUProfilesamples atconst hz = 100(100 samples per second). go help build(Go 1.24.7):-pgo file— "When the special name "auto" is specified, for each main package in the build, the go command selects a file named "default.pgo" in the package's directory if that file exists ... The default is "auto"."- Profile-guided optimization (Go documentation): PGO supported "Starting in Go 1.20"; "Before Go 1.21, the default is -pgo=off"; "As of Go 1.22, benchmarks for a representative set of Go programs show that building with PGO improves performance by around 2-14%"; the compiler "may decide to more aggressively inline functions which the profile indicates are called frequently"; collecting profiles "directly from the production environment is recommended".
- Go 1.26 release notes: "The pprof tool web UI, enabled with the -http flag, now defaults to the flame graph view"; an experimental
goroutineleakprofile (GOEXPERIMENT=goroutineleakprofile).
Why guessing fails
package main
import (
"fmt"
"regexp"
"sort"
"strconv"
"time"
)
func Report(prices []int) string {
t := time.Now()
sort.Ints(prices) // slow part?
fmt.Println("sort took: ", time.Since(t))
out := ""
for _, p := range prices {
line := "item ₹" + strconv.Itoa(p) + "\n"
if regexp.MustCompile(`^item ₹\d+\n$`).MatchString(line) {
out += line
}
}
return out
}
func main() {
prices := make([]int, 2000)
for i := range prices {
prices[i] = (i * 7919) % 1000
}
t := time.Now()
Report(prices)
fmt.Println("Report took:", time.Since(t))
}
Real output:
sort took: 103.966µs
Report took: 27.348371ms
The sort is about 0.4% of the time. Timing every line by hand doesn't scale, and the timers change the timing.
The profiler idea: sampling
| Finding the slow step in a kitchen | A CPU profiler |
|---|---|
| take a photo every few seconds of what each cook is doing | sampling: interrupt the program regularly and record where it is (Go: 100 times a second) |
| count the photos per task: 60 of 100 show chopping | a profile: samples counted per function |
| note who asked for the onions | the call stack: which function called which |
A profile is a statistical picture, not a stopwatch, so it is cheap enough to run on real programs.
CPU profile: go test -cpuprofile and go tool pprof -top
The program under test (package shop):
// ---- bill.go ----
package shop
import (
"regexp"
"sort"
"strconv"
)
func Report(prices []int) string {
sort.Ints(prices) // slow part?
out := ""
for _, p := range prices {
line := "item ₹" + strconv.Itoa(p) + "\n"
if regexp.MustCompile(`^item ₹\d+\n$`).MatchString(line) {
out += line
}
}
return out
}
// ---- bill_test.go ----
package shop
import "testing"
func BenchmarkReport(b *testing.B) {
prices := make([]int, 2000)
for i := range prices {
prices[i] = (i * 7919) % 1000
}
for i := 0; i < b.N; i++ {
Report(prices)
}
}
$ go test -bench . -cpuprofile cpu.prof
$ go tool pprof -top -cum -nodecount=9 cpu.prof
Real output of the second command (one run):
File: shop.test
Type: cpu
Duration: 2.13s, Total samples = 1.84s (86.24%)
Showing nodes accounting for 0.01s, 0.54% of 1.84s total
Showing top 9 nodes out of 170
flat flat% sum% cum cum%
0 0% 0% 1.08s 58.70% shop.BenchmarkReport
0 0% 0% 1.08s 58.70% shop.Report
0 0% 0% 1.08s 58.70% testing.(*B).launch
0 0% 0% 1.08s 58.70% testing.(*B).runN
0 0% 0% 0.69s 37.50% regexp.Compile (inline)
0 0% 0% 0.69s 37.50% regexp.MustCompile
0 0% 0% 0.69s 37.50% regexp.compile
0 0% 0% 0.59s 32.07% runtime.bgsweep
0.01s 0.54% 0.54% 0.41s 22.28% runtime.mallocgc
| Column | Meaning |
|---|---|
| flat | time spent in the function itself |
| cum (cumulative) | time in the function plus everything it called |
| flat% / cum% | the same as a share of all samples |
| sum% | running total of flat% down the table |
-top prints the table; -cum sorts by cum; -nodecount=9 keeps nine rows. runtime.mallocgc hands out heap memory and runtime.bgsweep is the GC's background sweep (episode 56): lots of garbage.
CPU profile: -list
$ go test -bench . -cpuprofile cpu.prof
$ go tool pprof -list=shop.Report cpu.prof
Real output (the -list part):
ROUTINE ======================== shop.Report in bill.go
0 980ms (flat, cum) 52.97% of Total
. . 9:func Report(prices []int) string {
. . 10: sort.Ints(prices) // slow part?
. . 11: out := ""
. . 12: for _, p := range prices {
. 20ms 13: line := "item ₹" + strconv.Itoa(p) + "\n"
. 660ms 14: if regexp.MustCompile(`^item ₹\d+\n$`).MatchString(line) {
. 300ms 15: out += line
. . 16: }
. . 17: }
. . 18: return out
. . 19:}
Flame graphs
- Each box is a function; a box sits under (pprof draws root on top) the function that called it.
- Width = share of samples. Left-to-right order means nothing.
- Look for the widest boxes deep in the chain: that's where time burns. Thin boxes aren't worth optimising.
go tool pprof -http=127.0.0.1:8090 cpu.profopens the web UI; since Go 1.26 it opens on the flame graph.
The video shows a real screenshot of the flame graph view for this benchmark: shop.Report → regexp.MustCompile (wide) and runtime.concatstrings (the +=), plus a wide runtime.bgsweep (the GC).
The fix, measured
// $ go test -bench . -benchmem
// ---- bill.go ----
package shop
import (
"regexp"
"sort"
"strconv"
"strings"
)
func Report(prices []int) string { // the original
sort.Ints(prices)
out := ""
for _, p := range prices {
line := "item ₹" + strconv.Itoa(p) + "\n"
if regexp.MustCompile(`^item ₹\d+\n$`).MatchString(line) {
out += line
}
}
return out
}
var lineFormat = regexp.MustCompile(`^item ₹\d+\n$`) // fix 1: compile once
func ReportFast(prices []int) string {
sort.Ints(prices)
var sb strings.Builder // fix 2: one growing buffer, no copies
for _, p := range prices {
line := "item ₹" + strconv.Itoa(p) + "\n"
if lineFormat.MatchString(line) {
sb.WriteString(line)
}
}
return sb.String()
}
// ---- bill_test.go ----
package shop
import "testing"
func prices() []int {
p := make([]int, 2000)
for i := range p {
p[i] = (i * 7919) % 1000
}
return p
}
func BenchmarkReport(b *testing.B) {
p := prices()
for i := 0; i < b.N; i++ {
Report(p)
}
}
func BenchmarkReportFast(b *testing.B) {
p := prices()
for i := 0; i < b.N; i++ {
ReportFast(p)
}
}
Real output:
BenchmarkReport-2 67 23896532 ns/op 33848040 B/op 119826 allocs/op
BenchmarkReportFast-2 2738 442877 ns/op 145406 B/op 3818 allocs/op
PASS
About 54× faster. The loop: measure → read the profile → fix the widest box → measure again.
Heap profiles: alloc_space vs inuse_space
-memprofile mem.profwrites a heap profile: where memory was allocated (sampled, about one sample per 512 KB allocated by default).-sample_index=alloc_space: every byte allocated since the start, garbage included → finds GC pressure.-sample_index=inuse_space(the default): bytes still in use when the profile was written → finds leaks.- Also
alloc_objects/inuse_objectscount objects instead of bytes.
$ go test -bench . -memprofile mem.prof
$ go tool pprof -sample_index=alloc_space -top -nodecount=5 mem.prof
$ go tool pprof -sample_index=inuse_space -top -nodecount=3 mem.prof
Real output (for the original Report benchmark):
Type: alloc_space
Showing nodes accounting for 1893.41MB, 93.67% of 2021.42MB total
Dropped 16 nodes (cum <= 10.11MB)
Showing top 5 nodes out of 26
flat flat% sum% cum cum%
1500.23MB 74.22% 74.22% 2021.42MB 100% shop.Report
162.57MB 8.04% 82.26% 162.57MB 8.04% regexp/syntax.(*compiler).inst (inline)
104.59MB 5.17% 87.43% 104.59MB 5.17% regexp.onePassCopy
93.51MB 4.63% 92.06% 93.51MB 4.63% regexp/syntax.(*parser).newRegexp (inline)
32.51MB 1.61% 93.67% 85.01MB 4.21% regexp.makeOnePass
Type: inuse_space
Showing nodes accounting for 0, 0% of 0 total
flat flat% sum% cum cum%
About 2 GB allocated, nothing still in use at the end.
Goroutine profiles
// $ go run .
// ---- main.go ----
package main
import (
"fmt"
"os"
"runtime/pprof"
"time"
)
func askKitchen(order int, reply chan string) {
reply <- fmt.Sprint("order ", order, " ready") // nobody reads this
}
func main() {
for i := 0; i < 100; i++ {
go askKitchen(i, make(chan string)) // forgot to receive
}
time.Sleep(100 * time.Millisecond)
pprof.Lookup("goroutine").WriteTo(os.Stdout, 1) // 1 = short text
}
Real output (shortened; the program's own stack lines are kept):
goroutine profile: total 101
100 @ 0x46bf0e 0x40b3e5 0x40b037 0x4cf72f 0x472a01
# 0x4cf72e main.askKitchen+0xae main.go:11
1 @ 0x432bb1 0x46b33d 0x4bbd51 0x4bbb85 0x4b89ab 0x4cf65d 0x43cc0b 0x472a01
# 0x4bbd50 runtime/pprof.writeRuntimeProfile+0xb0 .../src/runtime/pprof/pprof.go:796
...
# 0x4cf65c main.main+0xbc main.go:19
pprof.Lookup(name)returns a built-in profile:goroutine,heap,allocs,threadcreate,block,mutex.- Identical stacks are grouped and counted, so a goroutine leak (episode 54) shows as one big number.
Block and mutex profiles
// $ go test -run Rush -blockprofile block.prof -mutexprofile mutex.prof
// $ go tool pprof -top -nodecount=4 block.prof
// $ go tool pprof -top -nodecount=4 mutex.prof
// ---- kitchen.go ----
package shop
import (
"sync"
"time"
)
type Stove struct{ mu sync.Mutex } // one stove, many cooks
func (s *Stove) Cook() {
s.mu.Lock()
time.Sleep(2 * time.Millisecond) // cooking while holding the lock
s.mu.Unlock()
}
// ---- kitchen_test.go ----
package shop
import (
"sync"
"testing"
)
func TestRush(t *testing.T) {
var s Stove
var wg sync.WaitGroup
for cook := 0; cook < 8; cook++ {
wg.Add(1)
go func() { defer wg.Done(); for i := 0; i < 10; i++ { s.Cook() } }()
}
wg.Wait()
}
Real output:
Type: delay
Showing nodes accounting for 1521.81ms, 100% of 1521.81ms total
Showing top 4 nodes out of 14
flat flat% sum% cum cum%
1166.05ms 76.62% 76.62% 1166.05ms 76.62% sync.(*Mutex).Lock
177.95ms 11.69% 88.32% 177.95ms 11.69% runtime.chanrecv1
177.81ms 11.68% 100% 177.81ms 11.68% sync.(*WaitGroup).Wait
0 0% 100% 177.95ms 11.69% main.main
Type: delay
Showing nodes accounting for 1.17s, 100% of 1.17s total
flat flat% sum% cum cum%
1.17s 100% 100% 1.17s 100% sync.(*Mutex).Unlock (inline)
0 0% 100% 1.17s 100% shop.(*Stove).Cook
0 0% 100% 1.17s 100% shop.TestRush.func1
- Block profile: where goroutines waited (locks, channels,
WaitGroup,select). - Mutex profile: which lock holders made others wait (charged where the lock is released).
- Outside tests:
runtime.SetBlockProfileRate(rate)andruntime.SetMutexProfileFraction(rate); rate 1 records every event.
Profiling a live server: net/http/pprof
// $ go run .
// ---- main.go ----
package main
import (
"bufio"
"fmt"
"net/http"
_ "net/http/pprof" // blank import: adds the /debug/pprof/ pages
"time"
)
func main() {
go http.ListenAndServe("localhost:6060", nil) // your real server
time.Sleep(200 * time.Millisecond)
// what you'd open in a browser, or fetch with go tool pprof:
for _, page := range []string{"goroutine?debug=1", "heap?debug=1"} {
resp, err := http.Get("http://localhost:6060/debug/pprof/" + page)
if err != nil {
fmt.Println(err)
return
}
first, _ := bufio.NewReader(resp.Body).ReadString('\n')
resp.Body.Close()
fmt.Printf("%-18s -> %s", page, first)
}
}
Real output:
goroutine?debug=1 -> goroutine profile: total 6
heap?debug=1 -> heap profile: 2: 65632 [2: 65633] @ heap/1048576
- A blank import (
_ "net/http/pprof") runs the package'sinit, which registers handlers onhttp.DefaultServeMux. - From your machine:
go tool pprof http://host:6060/debug/pprof/profile?seconds=30(30-second CPU profile),.../heap,.../goroutine. - Never expose these pages publicly: serve them on a private port or behind authentication.
Profile-guided optimisation (PGO)
// $ go test -bench . -cpuprofile default.pgo
// $ go build -o app .
// $ go version -m app
// ---- main.go ----
package main
import (
"fmt"
"strconv"
"strings"
)
func Bill(n int) string {
var sb strings.Builder
for i := 1; i <= n; i++ {
sb.WriteString("item ₹" + strconv.Itoa(i*10) + "\n")
}
return sb.String()
}
func main() { fmt.Print(Bill(2)) }
// ---- main_test.go ----
package main
import "testing"
func BenchmarkBill(b *testing.B) {
for i := 0; i < b.N; i++ {
Bill(1000)
}
}
Real output (shortened):
BenchmarkBill-2 14086 78190 ns/op
PASS
$ go build -o app .
$ go version -m app
app: go1.24.7
path shop
build -buildmode=exe
build -compiler=gc
build -pgo=default.pgo
...
- PGO: the compiler uses a CPU profile to optimise hot code, mainly by inlining hot functions more aggressively. Since Go 1.20;
-pgo=auto(usedefault.pgoin the main package directory) is the default since Go 1.21; around 2–14% faster as of Go 1.22. - Use a profile from production, commit it as
default.pgo, refresh it now and then.
Which profile answers which question
| Question | Profile | How |
|---|---|---|
| Slow, CPU busy? | CPU | go test -cpuprofile, pprof.StartCPUProfile, /debug/pprof/profile |
| GC busy, too much garbage? | heap, alloc_space |
-memprofile, /debug/pprof/allocs |
| Memory keeps growing? | heap, inuse_space |
-memprofile, /debug/pprof/heap |
| Stuck, too many goroutines? | goroutine | pprof.Lookup("goroutine"), /debug/pprof/goroutine |
| Waiting on locks/channels? | block | -blockprofile, SetBlockProfileRate |
| Lock held too long? | mutex | -mutexprofile, SetMutexProfileFraction |
Mini project: the slow menu
Write Menu(dishes []string) string using +=, fmt.Sprintf and regexp.MustCompile in the loop; benchmark with 5000 dishes; profile; fix one thing at a time.
Solution
// $ go test -bench . -benchmem
// ---- menu.go ----
package shop
import (
"fmt"
"regexp"
"strings"
)
func Menu(dishes []string) string {
out := ""
for i, d := range dishes {
if regexp.MustCompile(`^[a-z ]+$`).MatchString(d) {
out += fmt.Sprintf("%d. %s\n", i+1, d)
}
}
return out
}
var dishName = regexp.MustCompile(`^[a-z ]+$`)
func MenuFast(dishes []string) string {
var sb strings.Builder
sb.Grow(len(dishes) * 24)
for i, d := range dishes {
if dishName.MatchString(d) {
fmt.Fprintf(&sb, "%d. %s\n", i+1, d)
}
}
return sb.String()
}
// ---- menu_test.go ----
package shop
import "testing"
func dishes() []string {
d := make([]string, 5000)
for i := range d {
d[i] = "masala dosa"
}
return d
}
func BenchmarkMenu(b *testing.B) {
d := dishes()
for i := 0; i < b.N; i++ {
Menu(d)
}
}
func BenchmarkMenuFast(b *testing.B) {
d := dishes()
for i := 0; i < b.N; i++ {
MenuFast(d)
}
}
Real output:
BenchmarkMenu-2 9 115730039 ns/op 247025728 B/op 195272 allocs/op
BenchmarkMenuFast-2 542 2231287 ns/op 241074 B/op 9747 allocs/op
About 50× faster and 1000× less memory. The CPU profile's -list=shop.Menu points at the MustCompile line first; the heap profile (alloc_space) at the += line.
Common mistakes (and the real messages)
| Mistake | What you see | Why | Fix |
|---|---|---|---|
| guessing the slow part | sort took: 103.966µs of 27.348371ms |
intuition is often wrong | profile first |
| reading only flat | shop.Report flat 0 |
the time is in callees | look at cum and -list |
inuse_space to find GC pressure |
0, 0% of 0 total |
the garbage is gone | -sample_index=alloc_space |
regexp.MustCompile in a loop |
regexp.MustCompile ~37% cum |
compiles every call | a package-level variable |
/debug/pprof on the internet |
internals exposed | anyone can read and trigger profiles | private port only |
| optimising thin boxes | no measurable change | not where time goes | fix the widest box |
Practice
- A function has flat 2% and cum 70%. Where should you look?
- Your service's memory grows all day. Which profile and sample index?
- Why can't a CPU profile show time spent waiting for a mutex?
- Where must
default.pgobe forgo buildto use it?
Solutions
1. In its callees: run -list on it, or look at the boxes above/below it in the flame graph.
2. The heap profile with inuse_space (the default), compared between two moments.
3. It only samples goroutines running on a CPU; a goroutine blocked on a mutex isn't running. Use the block or mutex profile, or a trace (episode 58).
4. In the main package's directory.
Quick quiz
1. shop.Load flat 1% cum 75%; json.Unmarshal flat 45% cum 45%; strings.Split flat 15%. Which first? (From the video.)
A. shop.Load · B. json.Unmarshal · C. strings.Split · D. none
Answer
B. Load's cum is high only because it calls Unmarshal, which does 45% of the work itself.
2. How many samples per second does Go's CPU profiler take?
Answer
100.
3. What does a box's width mean in a flame graph?
Answer
Its share of the samples (time), including its callees.
Interview questions
- What is a sampling profiler? It interrupts the program at a fixed rate and records the current call stack; counts per function give the profile.
- flat vs cum? flat: time in the function itself; cum: including callees.
- How do you take a CPU profile in Go?
go test -cpuprofile,pprof.StartCPUProfile/StopCPUProfile, or/debug/pprof/profilevianet/http/pprof. - inuse_space vs alloc_space? Live memory at profile time vs everything allocated since start.
- How do you find a goroutine leak? Goroutine profile: many goroutines with the same stack.
- Block vs mutex profile? Block: where goroutines waited; mutex: which lock holders caused waiting.
- How do you profile production safely?
net/http/pprofon a private port, short CPU profiles; profiles are low overhead. - What is PGO in Go? Building with a CPU profile (
default.pgo) so the compiler inlines hot code more; around 2–14% faster.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Profiler | a tool that measures where a program spends time or memory | pprof |
| Profiling | running a program under a profiler | go test -cpuprofile |
| Sampling | recording the program's state at regular moments | 100 times a second |
| Profile | the collected samples, counted per function | cpu.prof |
| Call stack | the chain of functions that led to the current point | main → Report → MustCompile |
| pprof | Go's profiling package and tool | go tool pprof |
| CPU profile | samples of code running on the CPU | -cpuprofile cpu.prof |
go tool pprof -top |
the busiest functions as a table | -top -cum -nodecount=9 |
-list |
source lines with samples | -list=shop.Report |
| flat | time in the function itself | 0 for Report |
| cum (cumulative) | time in the function and its callees | 58.70% |
| Flame graph | boxes stacked by call chain, width = time | pprof -http view |
-http |
opens pprof's web UI | -http=127.0.0.1:8090 |
| Immutable | can't be changed after creation; += makes a new string |
Go strings |
strings.Builder |
a growable buffer for building strings | sb.WriteString(line) |
| Heap profile | where memory was allocated | -memprofile mem.prof |
alloc_space |
all bytes allocated since start | 2021.42MB total |
inuse_space |
bytes still live at profile time | 0 total |
| GC pressure | lots of allocation that keeps the GC busy | out += line |
| Goroutine profile | stacks of all live goroutines | goroutine profile: total 101 |
pprof.Lookup |
get a named profile | pprof.Lookup("goroutine") |
| Goroutine leak | goroutines blocked forever | 100 stuck senders |
| Block profile | where goroutines waited | sync.(*Mutex).Lock |
| Mutex profile | which lock holders made others wait | sync.(*Mutex).Unlock |
SetBlockProfileRate, SetMutexProfileFraction |
turn those profiles on in a program | rate 1 = every event |
net/http/pprof |
HTTP handlers serving profiles | /debug/pprof/ |
| Blank import | importing only for side effects | _ "net/http/pprof" |
| PGO | profile-guided optimisation | default.pgo |
default.pgo |
the CPU profile go build uses automatically |
in the main package dir |
go version -m |
prints a binary's build information | build -pgo=default.pgo |
| Inlining | pasting a small function's body into its caller | more aggressive with PGO |
Summary
- Don't guess: measure. A sampling profiler records the call stack 100 times a second.
- CPU profile:
go test -cpuprofile,go tool pprof -top(flat vs cum) and-listfor lines; flame graph: width = time. - Heap profile:
alloc_spacefor GC pressure,inuse_spacefor leaks. - Goroutine profile finds stuck goroutines; block and mutex profiles find waiting.
net/http/pprofprofiles live servers (private port only).- PGO: commit a production CPU profile as
default.pgofor around 2–14% speed.
Next episode: execution tracing and performance tuning: runtime/trace, go tool trace, the flight recorder, latency vs throughput, fewer allocations, sync.Pool, inlining, bounds-check elimination.