SyntaxPilot
Courses/Go Programming/Lesson 42
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Go Programming · Lesson 42 of 89

Benchmarks and fuzzing

benchmark functions*testing.Bb.Nb.Loop (Go 1.24)b.ResetTimerb.ReportAllocs-bench-benchmemns/opB/opallocs/opbenchstatsub-benchmarksnative fuzzing

These are the notes for Go episode 42. Raju's receipt joins items with s += item, and a friend says strings.Builder is faster. Timing one call with time.Now gives numbers that jump around, so you invent the benchmark: run the code b.N times and report the average, ns/op. Then you meet the lies benchmarks can tell (setup counted, work removed by the compiler) and their fixes (b.ResetTimer, b.Loop), memory columns (-benchmem, B/op, allocs/op), sub-benchmarks, noise and benchstat. Finally a unit test passes but fuzzing finds a name that Short cuts in the middle of a letter, saves it in testdata/fuzz, and every later go test re-runs it.

All outputs and errors here are from real runs (Go 1.24.7, linux/amd64, 2 CPUs at 2.10 GHz). Timings change on every run and on every machine: your ns/op numbers will differ; allocation counts, the b.N sequence and error messages are stable. In the SyntaxPilot compiler, a program whose first lines are // $ go test ... is run as a package: each // ---- name ---- line starts a new file (a go.mod for module shop is added), and the commands run in order (episode 40 explains this). On your computer: go mod init shop, create the files, run the commands.

Facts and sources

  • go doc testing.B.Loop (Go 1.24.7): "Loop returns true as long as the benchmark should continue running." "Loop resets the benchmark timer the first time it is called in a benchmark, so any setup performed prior to starting the benchmark loop does not count toward the benchmark measurement. Likewise, when it returns false, it stops the timer so cleanup code is not measured." "The compiler never optimizes away calls to functions within the body of a "for b.Loop() { ... }" loop." "Benchmarks should either use Loop or contain a loop to b.N, but not both."
  • Go 1.24 release notes (tip.golang.org/doc/go1.24): Go 1.24 arrived in February 2025. "Benchmarks may now use the faster and less error-prone testing.B.Loop method to perform benchmark iterations ... in place of the typical loop structures involving b.N": "The benchmark function will execute exactly once per -count, so expensive setup and cleanup steps execute only once." "Function call parameters and results are kept alive, preventing the compiler from fully optimizing away the loop body."
  • go help testflag: -benchtime t "Run enough iterations of each benchmark to take t ... The default is 1 second (1s). The special syntax Nx means to run the benchmark N times (for example, -benchtime 100x)." -count n "Run each test, benchmark, and fuzz seed n times (default 1)." -fuzz regexp "the command line argument must match exactly one package within the main module, and regexp must match exactly one fuzz test within that package." -fuzztime t "The default is to run forever."
  • Go Fuzzing docs (golang.google.cn/doc/fuzz): fuzzing is in the standard toolchain "beginning in Go 1.18". Allowed argument types: string, []byte, int, int8, int16, int32/rune, int64, uint, uint8/byte, uint16, uint32, uint64, float32, float64, bool. The seed corpus comes from f.Add calls and files in testdata/fuzz/{FuzzTestName}; these run by default with go test. A failing input is written to testdata/fuzz/{FuzzTestName} and the next go test uses it as a regression test. A fuzz test is the FuzzXxx(f *testing.F) function; the fuzz target is the function passed to f.Fuzz.
  • benchstat doc comment (go.googlesource.com/perf cmd/benchstat): "Benchstat computes statistical summaries and A/B comparisons of Go benchmarks." "Each benchmark should be run at least 10 times to gather a statistically significant sample of results." "The p-value measures how likely it is that any differences were due to random chance." "the '~' means benchstat did not detect a statistically significant difference between the two inputs." The example table below is from that page.

The problem: timing one call

Raju's shop prints a line of items on every receipt. Two ways to join them: += and strings.Builder. The obvious way to compare them: time each one with time.Now() and time.Since().

package main

import (
    "fmt"
    "strings"
    "time"
)

func joinPlus(items []string) string {
    s := ""
    for _, it := range items {
        s += it + ","
    }
    return s
}

func joinBuilder(items []string) string {
    var b strings.Builder
    for _, it := range items {
        b.WriteString(it)
        b.WriteString(",")
    }
    return b.String()
}

func main() {
    items := []string{"chai", "samosa", "vada pav", "lassi"}
    for i := 0; i < 3; i++ {
        t := time.Now()
        joinPlus(items)
        fmt.Println("+=      took", time.Since(t))
        t = time.Now()
        joinBuilder(items)
        fmt.Println("Builder took", time.Since(t))
    }
}

Output of two runs (go run main.go):

+=      took 4.653µs
Builder took 456ns
+=      took 430ns
Builder took 180ns
+=      took 321ns
Builder took 159ns

+=      took 4.236µs
Builder took 561ns
+=      took 436ns
Builder took 227ns
+=      took 260ns
Builder took 268ns

The numbers jump around. The first call is about ten times slower (the CPU caches are cold), and in the second run the last pair even flips: Builder looks slower. µs is a microsecond (a millionth of a second); ns is a nanosecond (a billionth of a second, a thousand times smaller).

Why it fails: one call takes a few hundred nanoseconds. In that time the noise (cold caches, another program taking the CPU for a moment, the cost of reading the clock itself) is bigger than the thing being measured. One reading tells you nothing.

Benchmarks: run it many times (timing one ball vs a whole spell)

Timing one cricket ball with a phone is mostly your thumb. Time 1,000 balls and divide by 1,000 and the thumb error almost vanishes. That is a benchmark: run the code many times and report the average time per run. How many times? Go decides, and the count is b.N.

The rules: - it lives in a _test.go file; - its name starts with Benchmark (BenchmarkXxx); - it takes one parameter b *testing.B (a pointer to testing.B, the benchmark's handle); - it runs the code b.N times.

// $ go test
// $ go test -bench=.
// ---- bill.go ----
package shop

import "strings"

func JoinPlus(items []string) string {
    s := ""
    for _, it := range items {
        s += it + ","
    }
    return s
}

func JoinBuilder(items []string) string {
    var b strings.Builder
    for _, it := range items {
        b.WriteString(it)
        b.WriteString(",")
    }
    return b.String()
}

// ---- bill_test.go ----
package shop

import "testing"

var items = []string{"chai", "samosa", "vada pav", "lassi"}

func BenchmarkJoinPlus(b *testing.B) {
    for i := 0; i < b.N; i++ {
        JoinPlus(items)
    }
}

func BenchmarkJoinBuilder(b *testing.B) {
    for i := 0; i < b.N; i++ {
        JoinBuilder(items)
    }
}

Output (real run; your timings will differ):

$ go test
testing: warning: no tests to run
PASS
ok      shop    0.002s
$ go test -bench=.
goos: linux
goarch: amd64
pkg: shop
cpu: Intel(R) Xeon(R) Processor @ 2.10GHz
BenchmarkJoinPlus-2          6605364           177.6 ns/op
BenchmarkJoinBuilder-2      10385799           113.1 ns/op
PASS
ok      shop    2.662s

(The first part, plain go test printing "no tests to run", is from a run of the same kind of package; benchmarks never run without -bench.)

How to read a line:

Part Meaning
BenchmarkJoinPlus the benchmark's name
-2 GOMAXPROCS: how many CPUs Go used (2 here)
6605364 the final b.N: how many iterations ran
177.6 ns/op average nanoseconds per operation (one trip round the loop)
  • -bench=regexp runs benchmarks whose name matches the pattern; . matches all. It works like -run for tests.
  • Builder wins, but with only four items, not by much.

How b.N is chosen

// $ go test -bench=Builder
// ---- bill_test.go ----
package shop

import (
    "fmt"
    "strings"
    "testing"
)

func JoinBuilder(items []string) string {
    var b strings.Builder
    for _, it := range items {
        b.WriteString(it)
        b.WriteString(",")
    }
    return b.String()
}

var items = []string{"chai", "samosa", "vada pav", "lassi"}

func BenchmarkJoinBuilder(b *testing.B) {
    fmt.Println("called with b.N =", b.N)
    for i := 0; i < b.N; i++ {
        JoinBuilder(items)
    }
}

Output (real run; your timings will differ):

$ go test -bench=Builder
called with b.N = 1
goos: linux
goarch: amd64
pkg: shop
cpu: Intel(R) Xeon(R) Processor @ 2.10GHz
BenchmarkJoinBuilder-2      called with b.N = 100
called with b.N = 10000
called with b.N = 1000000
called with b.N = 11315289
11315289           104.2 ns/op
PASS
ok      shop    1.293s

Go calls the benchmark function several times with a growing b.N: 1, 100, 10,000, 1,000,000, then a final count it predicts will fill the benchtime (default 1 second, -benchtime=2s or -benchtime=100x to change it). The final b.N is the number printed in the result line.

Setup cost and b.ResetTimer

The menu must be loaded (100 ms, like reading a file) before it can be sorted. That load is setup, not what we want to measure. -benchtime=100x runs exactly 100 iterations.

// $ go test -bench=. -benchtime=100x
// ---- menu_test.go ----
package shop

import (
    "slices"
    "testing"
    "time"
)

func loadMenu() []string {
    time.Sleep(100 * time.Millisecond) // like reading a big file
    return []string{"vada pav", "chai", "samosa", "lassi", "poha"}
}

func BenchmarkSortNoReset(b *testing.B) {
    menu := loadMenu() // setup
    for i := 0; i < b.N; i++ {
        slices.Sorted(slices.Values(menu))
    }
}

func BenchmarkSortReset(b *testing.B) {
    menu := loadMenu() // setup
    b.ResetTimer()     // start the stopwatch here
    for i := 0; i < b.N; i++ {
        slices.Sorted(slices.Values(menu))
    }
}

Output (real run; your timings will differ):

$ go test -bench=. -benchtime=100x
goos: linux
goarch: amd64
pkg: shop
cpu: Intel(R) Xeon(R) Processor @ 2.10GHz
BenchmarkSortNoReset-2           100       1005746 ns/op
BenchmarkSortReset-2             100           783.1 ns/op
PASS
ok      shop    0.414s

Without the reset, the 100 ms setup is spread over 100 sorts: about 1,000,000 ns (1 ms) per "sort", a lie. b.ResetTimer() sets the elapsed time (and allocation counters) back to zero, so only the loop is measured: well under a microsecond per sort. Related: b.StopTimer() and b.StartTimer() pause and resume the stopwatch inside the loop.

b.Loop and the vanishing loop

withGST adds 18% GST (Goods and Services Tax) to a price in paise (1 rupee = 100 paise), rounding to the nearest paisa.

// $ go test -bench=.
// ---- gst_test.go ----
package shop

import "testing"

// GST: price in paise plus 18%, rounded
func withGST(paise int) int {
    return (paise*118 + 50) / 100
}

func BenchmarkGSTOld(b *testing.B) {
    for i := 0; i < b.N; i++ {
        withGST(4000) // result thrown away
    }
}

func BenchmarkGSTLoop(b *testing.B) {
    for b.Loop() { // Go 1.24
        withGST(4000)
    }
}

Output (real run; your timings will differ):

$ go test -bench=.
goos: linux
goarch: amd64
pkg: shop
cpu: Intel(R) Xeon(R) Processor @ 2.10GHz
BenchmarkGSTOld-2       1000000000           0.4033 ns/op
BenchmarkGSTLoop-2      603246115            1.903 ns/op
PASS
ok      shop    1.603s

0.4 ns per call is impossible. This CPU runs at 2.1 GHz (2.1 billion clock ticks a second), so one tick is about 0.48 ns. A function call, a multiply, an add and a divide can't happen in less than one tick. What happened: the compiler inlined withGST (copied its body into the loop) and then saw that the result is never used, so it deleted the work. That is dead code elimination. The benchmark measured an empty loop.

for b.Loop() { ... } (Go 1.24, February 2025) fixes this: calls written directly inside the loop body are never optimized away, so we get a real ~1.9 ns. b.Loop also: - resets the timer the first time it is called, so setup before the loop isn't counted (no b.ResetTimer needed); - stops the timer when it returns false, so cleanup after the loop isn't counted; - runs the benchmark function once per -count, instead of once per growing b.N, so expensive setup runs once.

Use b.Loop for new code. Never mix it with a b.N loop in the same benchmark (see Common mistakes). With the old b.N style, the usual workaround was storing the result in a package-level variable (a "sink") so the compiler couldn't remove it.

Memory: -benchmem and allocs/op (new tiffin box for every item)

Imagine buying a new, bigger tiffin box for every item you pack, moving everything over and throwing the old box away. Go strings can't be changed (they are immutable), so s += item makes a brand-new string each time and copies all the old text into it. Each new block of memory taken from the heap (the memory area for values that outlive a function call) is an allocation. Allocations cost time, and the garbage collector (the part of the Go runtime that finds and frees memory nobody uses any more) must clean each one up later.

strings.Builder keeps one buffer with spare room and only moves to a bigger one now and then; b.Grow(n) reserves the room up front.

// $ go test -bench=. -benchmem
// ---- bill.go ----
package shop

import "strings"

func JoinPlus(items []string) string {
    s := ""
    for _, it := range items {
        s += it + ","
    }
    return s
}

func JoinBuilder(items []string) string {
    var b strings.Builder
    for _, it := range items {
        b.WriteString(it)
        b.WriteString(",")
    }
    return b.String()
}

// ---- bill_test.go ----
package shop

import (
    "slices"
    "testing"
)

var items = slices.Repeat([]string{"chai"}, 1000) // 1000 lines

func BenchmarkJoinPlus(b *testing.B) {
    for b.Loop() {
        JoinPlus(items)
    }
}

func BenchmarkJoinBuilder(b *testing.B) {
    for b.Loop() {
        JoinBuilder(items)
    }
}

Output (real run; your timings will differ):

$ go test -bench=. -benchmem
goos: linux
goarch: amd64
pkg: shop
cpu: Intel(R) Xeon(R) Processor @ 2.10GHz
BenchmarkJoinPlus-2             1222        965959 ns/op     2684239 B/op       1000 allocs/op
BenchmarkJoinBuilder-2         94279         11772 ns/op       17912 B/op         13 allocs/op
PASS
ok      shop    2.295s
  • slices.Repeat(s, n) (Go 1.23) returns a new slice with s repeated n times: here 1,000 "chai"s.
  • With 1,000 lines, += is about 80 times slower in this run.
  • B/op: bytes allocated per operation: 2.6 MB for += (every step copies the whole receipt so far), 17.9 KB for Builder.
  • allocs/op: allocations per operation: exactly 1,000 (one per line) vs 13.

Sub-benchmarks and b.ReportAllocs

b.Run(name, func(b *testing.B)) creates a sub-benchmark with its own name, like t.Run creates subtests. b.ReportAllocs() turns on the memory columns for that benchmark even without -benchmem.

// $ go test -bench=.
// ---- bill_test.go ----
package shop

import (
    "fmt"
    "slices"
    "strings"
    "testing"
)

func JoinPlus(items []string) string {
    s := ""
    for _, it := range items {
        s += it + ","
    }
    return s
}

func JoinGrow(items []string) string {
    var b strings.Builder
    b.Grow(len(items) * 5) // reserve room once
    for _, it := range items {
        b.WriteString(it)
        b.WriteString(",")
    }
    return b.String()
}

func BenchmarkJoin(b *testing.B) {
    for _, n := range []int{10, 100, 1000} {
        items := slices.Repeat([]string{"chai"}, n)
        b.Run(fmt.Sprintf("plus/n=%d", n), func(b *testing.B) {
            b.ReportAllocs()
            for b.Loop() {
                JoinPlus(items)
            }
        })
        b.Run(fmt.Sprintf("grow/n=%d", n), func(b *testing.B) {
            b.ReportAllocs()
            for b.Loop() {
                JoinGrow(items)
            }
        })
    }
}

Output (real run; your timings will differ):

$ go test -bench=.
goos: linux
goarch: amd64
pkg: shop
cpu: Intel(R) Xeon(R) Processor @ 2.10GHz
BenchmarkJoin/plus/n=10-2            2304675           487.6 ns/op       328 B/op         10 allocs/op
BenchmarkJoin/grow/n=10-2           11403918           102.7 ns/op        64 B/op          1 allocs/op
BenchmarkJoin/plus/n=100-2             99112         12766 ns/op       26376 B/op        100 allocs/op
BenchmarkJoin/grow/n=100-2           1732596           669.6 ns/op       512 B/op          1 allocs/op
BenchmarkJoin/plus/n=1000-2             1446        813052 ns/op     2684233 B/op       1000 allocs/op
BenchmarkJoin/grow/n=1000-2           185520          6013 ns/op        5376 B/op          1 allocs/op
PASS
ok      shop    7.025s
n += ns/op growth Grow ns/op growth
10 487.6 102.7
100 12,766 ×26 669.6 ×6.5
1000 813,052 ×64 6,013 ×9

Ten times more items costs += far more than ten times the time: each step copies everything before it, so the total work is about 1 + 2 + … + n, which grows like n². That is quadratic growth. JoinGrow grows roughly in step with n (linear), with exactly 1 allocation at every size.

  • -bench=Join/grow runs only the grow sub-benchmarks (each /-separated part of the pattern matches one level of names).
  • -run='^$' is a test pattern that matches no test name, so only benchmarks run: use it when benchmarking.

Noise, -count and benchstat

// $ go test -bench=JoinBuilder -count=5
// ---- bill_test.go ----
package shop

import (
    "strings"
    "testing"
)

func JoinBuilder(items []string) string {
    var b strings.Builder
    for _, it := range items {
        b.WriteString(it)
        b.WriteString(",")
    }
    return b.String()
}

var items = []string{"chai", "samosa", "vada pav", "lassi"}

func BenchmarkJoinBuilder(b *testing.B) {
    for b.Loop() {
        JoinBuilder(items)
    }
}

Output (real run; your timings will differ):

$ go test -bench=JoinBuilder -count=5
goos: linux
goarch: amd64
pkg: shop
cpu: Intel(R) Xeon(R) Processor @ 2.10GHz
BenchmarkJoinBuilder-2       9751136           117.5 ns/op
BenchmarkJoinBuilder-2      12359983           102.8 ns/op
BenchmarkJoinBuilder-2      11107392           122.2 ns/op
BenchmarkJoinBuilder-2      12428426           104.5 ns/op
BenchmarkJoinBuilder-2      11530644           117.8 ns/op
PASS
ok      shop    6.440s

Same code, five runs: 102.8 to 122.2 ns/op, a spread of almost 20%. So "5% faster" from one run before and one after proves nothing. You need many runs of each version and statistics that say whether the difference is bigger than the wobble. That is benchstat, from the Go team's golang.org/x/perf repository:

$ go install golang.org/x/perf/cmd/benchstat@latest
$ go test -run='^$' -bench=. -count=10 > old.txt
  ... change the code ...
$ go test -run='^$' -bench=. -count=10 > new.txt
$ benchstat old.txt new.txt

Example output from the benchstat documentation (not from our machine):

                   │  old.txt  │            new.txt             │
                   │  sec/op   │  sec/op    vs base             │
Encode/format=json   1.718µ ± 1%   1.423µ ± 1%  -17.20% (p=0.000 n=10)
Encode/format=gob    3.066µ ± 0%   3.070µ ± 2%        ~ (p=0.446 n=10)
geomean              2.295µ        2.090µ        -8.94%
Part Meaning
1.718µ the median (middle value) of the 10 runs, in seconds per op
± 1% the spread: a 95% confidence interval around the median
-17.20% the change from old to new (vs base)
p=0.000 the p-value: how likely a difference this big is pure chance; small (below 0.05) means real
n=10 samples used from each file
~ no statistically significant difference: don't celebrate
geomean geometric mean over all benchmarks: the overall change

Rules: at least 10 runs (ideally 20) per version, the same number every time, on a quiet machine.

Fuzzing: when your examples aren't enough

Short fits a customer's name on the receipt (at most 6 letters). Raju's table test:

// $ go test -v
// ---- name.go ----
package shop

// Short fits a customer's name on the receipt: at most 6 letters.
func Short(name string) string {
    if len(name) <= 6 {
        return name
    }
    return name[:6]
}

// ---- name_test.go ----
package shop

import "testing"

func TestShort(t *testing.T) {
    cases := map[string]string{
        "Raju":        "Raju",
        "Priyanka":    "Priyan",
        "Lakshmipati": "Lakshm",
    }
    for in, want := range cases {
        if got := Short(in); got != want {
            t.Errorf("Short(%q) = %q, want %q", in, got, want)
        }
    }
}

Output (real run; your timings will differ):

$ go test -v
=== RUN   TestShort
--- PASS: TestShort (0.00s)
PASS
ok      shop    0.003s

It passes, but only for the names Raju thought of.

The phone drop test

In a phone factory In Go fuzzing
a machine drops phones at random angles, thousands of times fuzzing: the computer invents inputs and runs your code with each
start from known hard drops (corner, edge) seed corpus: starting inputs added with f.Add
rule: the screen must not crack property: a rule true for every input
save the angle that cracked it, re-test every new model the failing input is saved in testdata/fuzz/FuzzXxx and re-run by every go test

Go's fuzzer is coverage-guided: it watches which parts of your code each input reaches and keeps mutating ("changing a little") the inputs that reach new code. Built into the toolchain since Go 1.18.

Writing a fuzz test

  • name starts with Fuzz (FuzzXxx), takes f *testing.F, returns nothing, in a _test.go file;
  • f.Add(...) adds seeds; their types must match the target's parameters, in order;
  • f.Fuzz(func(t *testing.T, ...)) gives the fuzz target, exactly one per fuzz test;
  • with random input you can't write the expected answer, so check a property. Here: if the name was valid UTF-8, the result must be valid UTF-8 (utf8.ValidString).
// $ go test -fuzz=FuzzShort -fuzztime=10s
// $ go test -v
// ---- name.go ----
package shop

// Short fits a customer's name on the receipt: at most 6 letters.
func Short(name string) string {
    if len(name) <= 6 {
        return name
    }
    return name[:6]
}

// ---- name_test.go ----
package shop

import (
    "testing"
    "unicode/utf8"
)

func FuzzShort(f *testing.F) {
    f.Add("Raju") // seed corpus
    f.Add("Priyanka")
    f.Fuzz(func(t *testing.T, name string) {
        got := Short(name)
        if utf8.ValidString(name) && !utf8.ValidString(got) {
            t.Errorf("Short(%q) = %q: broken letters", name, got)
        }
    })
}

Output (real run; the fuzzer is random, so your failing input and its file name will differ):

$ go test -fuzz=FuzzShort -fuzztime=10s
fuzz: elapsed: 0s, gathering baseline coverage: 0/2 completed
fuzz: elapsed: 0s, gathering baseline coverage: 2/2 completed, now fuzzing with 2 workers
fuzz: elapsed: 0s, execs: 3574 (40978/sec), new interesting: 7 (total: 9)
--- FAIL: FuzzShort (0.09s)
    --- FAIL: FuzzShort (0.00s)
        name_test.go:14: Short("00000ݕ") = "00000\xdd": broken letters

    Failing input written to testdata/fuzz/FuzzShort/d2b01acf94f5cb32
    To re-run:
    go test -run=FuzzShort/d2b01acf94f5cb32
FAIL
exit status 1
FAIL    shop    0.090s
$ go test -v
=== RUN   FuzzShort
=== RUN   FuzzShort/seed#0
=== RUN   FuzzShort/seed#1
=== RUN   FuzzShort/d2b01acf94f5cb32
    name_test.go:14: Short("00000ݕ") = "00000\xdd": broken letters
--- FAIL: FuzzShort (0.00s)
    --- PASS: FuzzShort/seed#0 (0.00s)
    --- PASS: FuzzShort/seed#1 (0.00s)
    --- FAIL: FuzzShort/d2b01acf94f5cb32 (0.00s)
FAIL
exit status 1
FAIL    shop    0.003s

Reading the fuzz lines: - gathering baseline coverage: first it runs the seeds (and any saved inputs) to learn which code they reach; - workers: copies of the test binary running inputs in parallel (default: GOMAXPROCS, 2 here); - execs: inputs tried so far, and per second; - new interesting: inputs that reached new code and were added to the generated corpus.

In 0.09 s it found "00000ݕ": five zeros and a letter outside English. ݕ is 2 bytes in UTF-8, so name[:6] cut it in half and left the broken byte \xdd.

Why: len counts bytes, not letters (episode 17). In UTF-8, English letters are 1 byte, most others 2, 3 or 4 bytes (Devanagari letters are 3). name[:6] cuts at byte 6, which can be the middle of a letter, so a customer named in Hindi or Tamil gets a broken receipt.

Crash reproduction: the failing input was written to testdata/fuzz/FuzzShort/<hash>. The plain go test -v after it ran seed#0, seed#1 and the saved file as subtests, and failed on it: a regression test for free. Re-run just that one with go test -run=FuzzShort/<hash>. Commit testdata/fuzz with your tests. A crash file looks like:

go test fuzz v1
string("00000ݕ")

Fixing it, and fuzzing again

Convert to []rune (letters, not bytes) and cut that. A second property: the result has at most 6 letters (utf8.RuneCountInString).

// $ go test -fuzz=FuzzShort -fuzztime=5s
// ---- name.go ----
package shop

// Short fits a customer's name on the receipt: at most 6 letters.
func Short(name string) string {
    r := []rune(name) // letters, not bytes
    if len(r) <= 6 {
        return name
    }
    return string(r[:6])
}

// ---- name_test.go ----
package shop

import (
    "testing"
    "unicode/utf8"
)

func FuzzShort(f *testing.F) {
    f.Add("Raju") // seed corpus
    f.Add("Priyanka")
    f.Fuzz(func(t *testing.T, name string) {
        got := Short(name)
        if utf8.ValidString(name) && !utf8.ValidString(got) {
            t.Errorf("Short(%q) = %q: broken letters", name, got)
        }
        if utf8.RuneCountInString(got) > 6 {
            t.Errorf("Short(%q) = %q: too long", name, got)
        }
    })
}

Output (real run; your timings will differ):

$ go test -fuzz=FuzzShort -fuzztime=5s
fuzz: elapsed: 0s, gathering baseline coverage: 0/9 completed
fuzz: elapsed: 0s, gathering baseline coverage: 9/9 completed, now fuzzing with 2 workers
fuzz: elapsed: 3s, execs: 107799 (35927/sec), new interesting: 19 (total: 28)
fuzz: elapsed: 6s, execs: 120746 (4316/sec), new interesting: 21 (total: 30)
fuzz: elapsed: 6s, execs: 120746 (0/sec), new interesting: 21 (total: 30)
PASS
ok      shop    6.024s

(The baseline here is 9 inputs because the earlier run's interesting inputs were kept in the Go build cache, $GOCACHE/fuzz; yours may differ.)

  • Over 100,000 names in about 5 seconds, no failure: PASS.
  • Without -fuzztime, fuzzing runs until it finds a failure or you press Ctrl+C.
  • Plain go test never fuzzes: it only runs the seeds and the saved files in testdata/fuzz, in milliseconds.
  • Fuzz arguments can only be string, []byte, integers, floats and bool.
  • -fuzz must match exactly one fuzz test, in one package.
  • Fuzz anything that reads text from outside: parsers, decoders, validators.

Benchmarks and fuzzing, written down

func BenchmarkX(b *testing.B) {
    // setup (not timed)
    for b.Loop() {
        X()
    }
}

func FuzzX(f *testing.F) {
    f.Add("seed")
    f.Fuzz(func(t *testing.T, s string) {
        // check a property of X(s)
    })
}
Command What it does
go test -bench=. run all benchmarks (tests too)
go test -run='^$' -bench=Join -benchmem only benchmarks matching Join, with memory columns
go test -bench=. -benchtime=100x exactly 100 iterations each
go test -bench=. -count=10 > new.txt 10 samples for benchstat
go test -fuzz=FuzzX -fuzztime=30s fuzz FuzzX for 30 seconds
go test -run=FuzzX/<hash> re-run one saved failing input

Mini project: a faster, safer receipt

  1. Receipt(items []Item) string prints name xQty Rs total per line. Write it with += (ReceiptPlus), then with strings.Builder + Grow.
  2. Sub-benchmarks for 10, 100 and 1,000 items with b.ReportAllocs.
  3. FuzzReceipt: the receipt is valid UTF-8, has one line per item, never panics, and both versions agree.
  4. Compare the two versions with -count=10 and benchstat.
Solution

receipt.go:

package shop

import (
    "fmt"
    "strings"
)

// Item is one line on the receipt.
type Item struct {
    Name  string
    Qty   int
    Price int // rupees per piece
}

// ReceiptPlus builds the receipt with += (slow for long receipts).
func ReceiptPlus(items []Item) string {
    s := ""
    for _, it := range items {
        s += fmt.Sprintf("%s x%d Rs %d\n", it.Name, it.Qty, it.Qty*it.Price)
    }
    return s
}

// Receipt builds the same text with a strings.Builder sized up front.
func Receipt(items []Item) string {
    var b strings.Builder
    b.Grow(len(items) * 24) // a typical line is under 24 bytes
    for _, it := range items {
        fmt.Fprintf(&b, "%s x%d Rs %d\n", it.Name, it.Qty, it.Qty*it.Price)
    }
    return b.String()
}

receipt_test.go:

package shop

import (
    "fmt"
    "strings"
    "testing"
    "unicode/utf8"
)

func makeItems(n int) []Item {
    items := make([]Item, n)
    for i := range items {
        items[i] = Item{Name: "chai", Qty: 2, Price: 20}
    }
    return items
}

func BenchmarkReceipt(b *testing.B) {
    for _, n := range []int{10, 100, 1000} {
        items := makeItems(n)
        b.Run(fmt.Sprintf("plus/n=%d", n), func(b *testing.B) {
            b.ReportAllocs()
            for b.Loop() {
                ReceiptPlus(items)
            }
        })
        b.Run(fmt.Sprintf("builder/n=%d", n), func(b *testing.B) {
            b.ReportAllocs()
            for b.Loop() {
                Receipt(items)
            }
        })
    }
}

func FuzzReceipt(f *testing.F) {
    f.Add("chai", 2, 20)
    f.Add("समोसा", 1, 15)
    f.Fuzz(func(t *testing.T, name string, qty, price int) {
        if strings.Contains(name, "\n") {
            t.Skip("a name with a newline would add a line")
        }
        got := Receipt([]Item{{name, qty, price}})
        if utf8.ValidString(name) && !utf8.ValidString(got) {
            t.Errorf("receipt is not valid UTF-8: %q", got)
        }
        if n := strings.Count(got, "\n"); n != 1 {
            t.Errorf("got %d lines, want 1: %q", n, got)
        }
        if got != ReceiptPlus([]Item{{name, qty, price}}) {
            t.Errorf("Receipt and ReceiptPlus disagree")
        }
    })
}

Run:

go test -run='^$' -bench=Receipt -benchmem
go test -fuzz=FuzzReceipt -fuzztime=30s

To compare with benchstat, benchmark only plus into old.txt and only builder into new.txt (rename the sub-benchmarks to the same name, e.g. n=1000, in each run so benchstat pairs them), with -count=10 each, then benchstat old.txt new.txt. The seeds pass with go test; expect the Builder version to need far fewer allocations as n grows. The t.Skip handles a name containing a newline, which would legitimately add a line: a fuzz test must state its property precisely.

Common mistakes (and the real messages)

Mistake Real message / result Why Fix
plain go test, expecting benchmarks testing: warning: no tests to run benchmarks need -bench go test -bench=.
func BenchmarkA(b testing.B) wrong signature for BenchmarkA, must be: func BenchmarkA(b *testing.B) not a pointer (found by go vet, which go test runs) b *testing.B
result unused in a b.N loop 0.4033 ns/op the compiler removed the call for b.Loop()
setup inside the timed part 1005746 ns/op for a sub-microsecond sort setup counted b.ResetTimer() or b.Loop
b.Loop inside a b.N loop B.Loop called with timer stopped use one style, not both only for b.Loop()
f.Add(4) for a string target mismatched type in call to (*testing.F).Add: int, fuzz target expects string seed types must match f.Add("4")
f.Fuzz(func(t *testing.T, s []string)) fuzzing arguments can only have the following types: string, bool, float32, float64, int, int8, int16, int32, int64, uint, uint8, uint16, uint32, uint64, []byte unsupported type string or []byte
go test -fuzz=. with two fuzz tests testing: will not fuzz, -fuzz matches more than one fuzz test: [FuzzA FuzzB] one fuzz test at a time -fuzz=FuzzA
comparing one run before and after "5% faster" that is just noise runs wobble -count=10 + benchstat
not committing testdata/fuzz the bug can return unnoticed the regression input is lost commit it

Practice

  1. (Homework) Build the receipt mini project (solution above).
  2. Benchmark strconv.Itoa(n) against fmt.Sprint(n) with -benchmem. Which allocates less?
Solution
func BenchmarkItoa(b *testing.B) {
    for b.Loop() {
        strconv.Itoa(123456)
    }
}

func BenchmarkSprint(b *testing.B) {
    for b.Loop() {
        fmt.Sprint(123456)
    }
}

Run go test -run='^$' -bench=. -benchmem. strconv.Itoa is faster and makes fewer allocations: fmt.Sprint takes ...any, so the number is boxed into an interface and formatted with reflection-like machinery. Check your own numbers.

  1. A benchmark reports 0.25 ns/op on a 3 GHz CPU. What do you suspect, and how do you fix it?
Solution

One tick at 3 GHz is about 0.33 ns, so less than a tick per call means the work was removed (dead code elimination). Rewrite it with for b.Loop() { ... }.

  1. Write a fuzz test for func Reverse(s string) string with the property: reversing twice gives the original, for valid UTF-8 input.
Solution
func FuzzReverse(f *testing.F) {
    f.Add("chai")
    f.Add("नमस्ते")
    f.Fuzz(func(t *testing.T, s string) {
        if !utf8.ValidString(s) {
            t.Skip()
        }
        if got := Reverse(Reverse(s)); got != s {
            t.Errorf("Reverse(Reverse(%q)) = %q", s, got)
        }
    })
}

A byte-by-byte Reverse fails; reversing a []rune passes.

  1. Your benchmark loads a 50 MB file before measuring. Old b.N style: where does b.ResetTimer() go? With b.Loop?
Solution

Right after loading, before the loop. With b.Loop you need nothing: it resets the timer the first time it is called, and the function (so the load) runs once.

Quick quiz

1. What does this benchmark line tell you? (From the video.)

BenchmarkJoinPlus-2   1222   965959 ns/op   2684239 B/op   1000 allocs/op

A. It ran 1222 times; each call took ~0.97 ms and made 1000 allocations · B. It ran 2 times on 1222 CPUs · C. Each call took 965959 seconds · D. It allocated 1000 bytes in total

Answer

A. 1222 is the final b.N; 965959 ns/op ≈ 0.97 ms per call; 2684239 B/op (2.6 MB) and 1000 allocs/op are per call; -2 is GOMAXPROCS.

2. Which values of b.N does Go try first?

Answer

1, 100, 10000, 1000000, ... until a run fills the benchtime (1 s by default).

3. Does plain go test fuzz?

Answer

No. It runs the seed corpus and the saved inputs in testdata/fuzz as ordinary subtests. Fuzzing needs -fuzz.

4. benchstat shows ~ (p=0.446 n=10). Is the new version faster?

Answer

No detectable difference: ~ means not statistically significant.

Interview questions

  1. How do you write and run a benchmark in Go? func BenchmarkX(b *testing.B) in a _test.go file, loop with for b.Loop() (or b.N times), run with go test -bench=X -benchmem.
  2. What is b.N and who sets it? The iteration count; the testing package grows it (1, 100, 10000, …) until the run fills -benchtime.
  3. What does b.Loop (Go 1.24) improve? It times only the loop (setup and cleanup excluded), runs the function once per -count, and keeps calls in the loop body from being optimized away.
  4. Why can a benchmark report an impossibly small time? The compiler inlined the function and eliminated it because the result was unused (dead code elimination).
  5. What do B/op and allocs/op mean, and why care? Bytes and heap allocations per operation; allocations cost time now and garbage-collector work later.
  6. Why is building a string with += in a loop slow? Strings are immutable; each += allocates a new string and copies the old content: quadratic time.
  7. How do you decide if an optimization really helped? Run both versions at least 10 times (-count=10) and compare with benchstat; trust a small p-value, not one run.
  8. What is fuzzing, and how is Go's fuzzer guided? Automatic generation of inputs from a seed corpus; coverage-guided mutation keeps inputs that reach new code.
  9. Fuzz test vs fuzz target? The FuzzX(f *testing.F) function vs the function passed to f.Fuzz.
  10. What happens when the fuzzer finds a failure? It minimizes the input, writes it to testdata/fuzz/FuzzX/<hash>, and every later go test runs it as a regression test.
  11. What makes a good fuzz property? Something true for every input without knowing the answer: no panic, valid output, round-trip (decode(encode(x)) == x), two implementations agree.

Glossary

Term Meaning Example
Benchmark code that runs a function many times to measure its average speed BenchmarkJoinPlus
*testing.B the handle Go passes to a benchmark: b.N, b.Loop, b.Run, timers func BenchmarkX(b *testing.B)
BenchmarkXxx the naming rule: starts with Benchmark, in a _test.go file BenchmarkSortReset
b.N the number of iterations Go asks for; grows until the run fills the benchtime 1, 100, 10000, 1000000, 11315289
b.Loop() Go 1.24 loop condition: times only the loop, keeps the work from being removed for b.Loop() { withGST(4000) }
b.ResetTimer() sets the benchmark's elapsed time and counters back to zero after loadMenu()
b.ReportAllocs() shows B/op and allocs/op for this benchmark, even without -benchmem inside each b.Run
b.Run creates a sub-benchmark with its own name b.Run("grow/n=100", ...)
Sub-benchmark a named benchmark inside another, often one per input size BenchmarkJoin/grow/n=10
-bench=regexp flag: run benchmarks matching the pattern -bench=., -bench=Join/grow
-benchtime flag: how long (default 1s) or how many times (100x) each benchmark runs -benchtime=100x
-benchmem flag: add B/op and allocs/op columns go test -bench=. -benchmem
-count n flag: run each test/benchmark n times -count=10
-run='^$' a test pattern matching no name, so only benchmarks run go test -run='^$' -bench=.
ns/op average nanoseconds per operation (one loop iteration) 113.1 ns/op
B/op bytes allocated per operation 2684239 B/op
allocs/op heap allocations per operation 1000 allocs/op
GOMAXPROCS suffix the -2 after a benchmark name: CPUs Go used BenchmarkJoinPlus-2
Allocation taking a new block of memory from the heap each s += it
Heap memory for values that must outlive the function that made them a new string
Garbage collector the runtime part that frees heap memory nobody uses cleans up old receipt strings
Immutable cannot be changed after creation; Go strings are immutable += makes a new string
strings.Builder a growable byte buffer for building strings efficiently b.WriteString("chai")
Grow(n) reserves room for n more bytes up front b.Grow(len(items) * 5)
slices.Repeat returns a slice repeated n times (Go 1.23) slices.Repeat([]string{"chai"}, 1000)
Dead code elimination the compiler deleting work whose result is never used 0.4033 ns/op
Inlining the compiler copying a small function's body into its caller withGST in the loop
Clock tick / GHz one step of the CPU clock; 2.1 GHz = 2.1 billion ticks a second (~0.48 ns each)
Microsecond (µs) a millionth of a second 4.653µs
Nanosecond (ns) a billionth of a second; 1 µs = 1000 ns 456ns
Setup preparation a benchmark needs that should not be timed loadMenu()
Noise random variation in measurements unrelated to your code 102.8–122.2 ns/op
Quadratic growth time grows with the square of the input: 10× input ≈ 100× time += joining
Linear growth time grows in step with the input JoinGrow
Median the middle value of sorted measurements benchstat's 1.718µ
p-value how likely a difference this large is pure chance p=0.000
Statistically significant unlikely to be chance (benchstat: small p; ~ means not) -17.20%
benchstat Go team tool that summarizes and compares benchmark results benchstat old.txt new.txt
Fuzzing automatically generating many inputs to find ones that break code go test -fuzz=FuzzShort
Coverage-guided the fuzzer keeps inputs that reach new code and mutates them "new interesting"
Fuzz test FuzzXxx(f *testing.F) function in a test file FuzzShort
Fuzz target the function passed to f.Fuzz; runs on each input func(t *testing.T, name string)
*testing.F the handle passed to a fuzz test: f.Add, f.Fuzz func FuzzShort(f *testing.F)
f.Add adds a seed input; types must match the target f.Add("Raju")
f.Fuzz registers the fuzz target (exactly one per fuzz test) f.Fuzz(func(t *testing.T, s string) {...})
Corpus a collection of inputs seeds plus saved inputs
Seed corpus the starting inputs: f.Add calls and files in testdata/fuzz/FuzzX "Raju", "Priyanka"
Property a rule that must hold for every input output is valid UTF-8
Worker a process running fuzz inputs in parallel "fuzzing with 2 workers"
execs number of inputs tried execs: 3574 (40978/sec)
New interesting inputs that reached new code, added to the corpus new interesting: 7
-fuzz=regexp flag: fuzz the one fuzz test matching the pattern -fuzz=FuzzShort
-fuzztime flag: how long to fuzz (default: forever) -fuzztime=10s
testdata/fuzz folder where failing inputs are saved, one folder per fuzz test testdata/fuzz/FuzzShort/d2b01acf94f5cb32
Crash file a saved failing input go test fuzz v1 + string("00000ݕ")
Regression test a test that makes sure a fixed bug never returns the saved input re-run by go test
UTF-8 the text encoding Go uses: 1 to 4 bytes per letter ݕ = 2 bytes
Byte 8 bits; len and s[:6] count bytes "Raju" = 4 bytes
Rune one Unicode letter (code point), type int32 []rune(name)
utf8.ValidString reports whether a string is valid UTF-8 false for "00000\xdd"
utf8.RuneCountInString counts letters (runes), not bytes "राजू" → 4
GST Goods and Services Tax, here 18% withGST(4000) → 4720
Paise 1/100 of a rupee; prices kept in paise avoid fractions 4000 paise = Rs 40

Summary

  • One time.Now reading is noise; a benchmark runs code b.N times and reports ns/op. Go grows b.N (1, 100, 10000, …) until a run fills -benchtime (1 s).
  • Plain go test runs no benchmarks: use -bench. The -2 suffix is GOMAXPROCS.
  • Setup counted makes a benchmark lie: b.ResetTimer() in b.N style. Impossibly fast means the compiler removed the work.
  • In Go 1.24+, write for b.Loop(): setup and cleanup excluded, work never removed, function run once.
  • -benchmem / b.ReportAllocs show B/op and allocs/op; += made 1000 allocations vs 1 with Grow, and grows quadratically.
  • b.Run makes sub-benchmarks per size. Results wobble: -count=10 and benchstat (median, ±, p-value, ~).
  • A fuzz test (FuzzXxx, *testing.F) adds seeds with f.Add and checks a property in f.Fuzz. go test -fuzz=FuzzX -fuzztime=… fuzzes; failures go to testdata/fuzz and every go test re-runs them. Commit them.
  • Next episode: concurrency vs parallelism and goroutines.