These are the notes for Go episode 44. Raju's cloud kitchen has 2 stoves (CPU cores) and 10,000 orders (goroutines). Someone must decide which order cooks on which stove, for how long, and what to do with orders waiting for a delivery. That someone is the scheduler, and you invent it: first the obvious design, one OS thread per goroutine, which crashes with fatal error: thread exhaustion; then one shared queue with one lock, which doesn't scale; then the real design, G, M and P, with local run queues, a global run queue and work stealing. Then you handle a goroutine that never stops (cooperative and asynchronous preemption), goroutines that wait for the kernel (blocking system calls and handoff) or the network (the network poller), how goroutine stacks start small and grow by copying, and container-aware GOMAXPROCS (Go 1.25).
All outputs and errors here are from real runs (Go 1.24.7, linux/amd64, a machine with 2 CPUs). Thread counts, queue lengths and timings change a little on every run and every machine; the shape stays the same.
Facts and sources
- Dmitry Vyukov, "Scalable Go Scheduler Design Doc", 2 May 2012 (Google Doc). Problems of the scheduler of that time: a single global mutex (
Sched.Lock) and centralised state for all goroutine operations; threads handing goroutines to each other; a memory cache per thread; frequent blocking and unblocking of threads around system calls. The proposal: "P represents a resource that is required to execute Go code"; there are exactly GOMAXPROCS Ps; each P has its own list of runnable goroutines; a P with an empty list picks a random other P and steals about half of its runnable goroutines; when a thread enters a system call, its P can be handed to another thread. - Go runtime source,
src/runtime/proc.go(Go 1.24.7), top comment: "G - goroutine. M - worker thread, or machine. P - processor, a resource that is required to execute Go code. M must have an associated P to execute Go code, however it can be blocked or in a syscall w/o an associated P. Design doc at https://golang.org/s/go11sched." In the same file: the global run queue is checked "once in a while to ensure fairness" (if pp.schedtick%61 == 0);forcePreemptNS = 10 * 1000 * 1000 // 10ms; a P is retaken from a syscall "if it's there for more than 1 sysmon tick (at least 20us)";maxstacksize = 1000000000on 64-bit.src/runtime/runtime2.go: each P hasrunq [256]guintptr(a local run queue of 256).src/runtime/stack.go: a growing stack getsnewsize := oldsize * 2. - Go 1.14 release notes: "Goroutines are now asynchronously preemptible. As a result, loops without function calls no longer potentially deadlock the scheduler or significantly delay garbage collection." Supported on all platforms except
windows/arm,darwin/arm,js/wasmandplan9/*. A consequence: programs "will see more slow system calls fail withEINTRerrors". - Go 1.4 release notes: stacks are contiguous: "When a stack limit is reached, a new, larger stack is allocated, all active frames for the goroutine are copied there, and any pointers into the stack are updated." And "the default starting size for a goroutine's stack in 1.4 has been reduced from 8192 bytes to 2048 bytes."
- Go 1.1 release notes: "Due to tighter coupling of the run-time and network libraries, fewer context switches are required on network operations."
- Go 1.25 release notes (Go 1.25 "arrives in August 2025"), Container-aware GOMAXPROCS: "In prior versions of Go, GOMAXPROCS defaults to the number of logical CPUs available at startup (runtime.NumCPU)." "On Linux, the runtime considers the CPU bandwidth limit of the cgroup containing the process, if any. If the CPU bandwidth limit is lower than the number of logical CPUs available, GOMAXPROCS will default to the lower limit." "In container runtime systems like Kubernetes, cgroup CPU bandwidth limits generally correspond to the "CPU limit" option. The Go runtime does not consider the "CPU requests" option." "On all OSes, the runtime periodically updates GOMAXPROCS if the number of logical CPUs available or the cgroup CPU bandwidth limit change." "Both of these behaviors are automatically disabled if GOMAXPROCS is set manually via the GOMAXPROCS environment variable or a call to runtime.GOMAXPROCS. They can also be disabled explicitly with the GODEBUG settings containermaxprocs=0 and updatemaxprocs=0, respectively."
go doc runtime/debug.SetMaxThreads: "If it attempts to use more than this many, the program crashes. … The initial setting is 10,000 threads."go doc runtime/debug.SetMaxStack: "The initial setting is 1 GB on 64-bit systems, 250 MB on 32-bit systems."- Network poller back ends in the runtime source:
netpoll_epoll.go(Linux),netpoll_kqueue.go(macOS and BSDs),netpoll_windows.go(I/O completion ports).
The problem: 10,000 orders, 2 stoves
Raju runs a cloud kitchen:
| In the kitchen | In the computer |
|---|---|
| 2 stoves: only two dishes cook at any moment | 2 CPU cores: each runs one piece of code at a time |
| 10,000 orders at lunch | 10,000 goroutines (small jobs started with go, episode 43) |
| an order waiting for the milkman or a phone call | a goroutine waiting for I/O (input/output: files, disks, the network) |
| the person who decides which order goes on which stove | the scheduler |
A scheduler is the part of a system that decides which job runs on which worker, when, and for how long. Go has its own scheduler inside every Go program, as part of the runtime (the code Go adds to every program to manage goroutines, memory and garbage collection).
Design 1: one cook per order (one OS thread per goroutine)
An OS thread is a worker the operating system (Linux, Windows, macOS) can run on a CPU. The obvious design: give every order its own cook, every goroutine its own thread. runtime.LockOSThread() glues the calling goroutine to its own thread, so we can simulate that design. debug.SetMaxThreads(100) sets the most threads the program may use (normally 10,000).
package main
import (
"fmt"
"runtime"
"runtime/debug"
"time"
)
// one cook (OS thread) for every order
func cook(order int) {
runtime.LockOSThread() // this goroutine owns its thread
time.Sleep(time.Second) // the order takes a second
}
func main() {
debug.SetMaxThreads(100) // our kitchen has room for 100 cooks
for order := 1; order <= 1000; order++ {
go cook(order)
}
time.Sleep(2 * time.Second)
fmt.Println("all 1000 orders done")
}
Real output (the start of it):
runtime: program exceeds 100-thread limit
fatal error: thread exhaustion
runtime stack:
runtime.throw({0x4b6ca6?, 0x0?})
/usr/local/go1.24.7/src/runtime/panic.go:1101 +0x48 fp=0xc0007e7e08 sp=0xc0007e7dd8 pc=0x467ae8
...
Why it fails. Threads are heavy:
- The kernel (the core of the operating system, which controls the hardware) reserves a stack for each thread, usually megabytes (often 8 MB of address space on Linux).
- Switching a CPU from one thread to another is a context switch: the kernel saves one thread's registers and loads another's. It costs far more than switching goroutines inside Go.
- So a server with 100,000 connections and one thread each would hit the default 10,000-thread limit and crash the same way.
What Go really does
Same limit of 100 threads, ten times more orders, no LockOSThread. pprof.Lookup("threadcreate").Count() returns how many OS threads the program has created so far (runtime/pprof is the profiling package; profiling gets its own episode).
package main
import (
"fmt"
"runtime"
"runtime/debug"
"runtime/pprof"
"time"
)
func cook(order int) {
time.Sleep(time.Second) // the order takes a second
}
func main() {
debug.SetMaxThreads(100) // the same small kitchen
for order := 1; order <= 10000; order++ {
go cook(order)
}
fmt.Println("goroutines:", runtime.NumGoroutine())
time.Sleep(2 * time.Second)
fmt.Println("all 10000 orders done")
threads := pprof.Lookup("threadcreate").Count()
fmt.Println("OS threads created:", threads)
}
Real output:
goroutines: 10001
all 10000 orders done
OS threads created: 4
10,001 goroutines (10,000 cooks plus main) ran on 4 threads. Running many goroutines on a few threads is called M:N scheduling (M goroutines on N threads).
Design 2: one queue, one lock
A few cooks and one order rail at the door. A list of jobs waiting to run is a run queue. Two cooks must not take the same ticket, so the rail gets a lock (a mutex: only one thread may use the protected thing at a time; sync.Mutex is next episode).
With 2 cooks this is fine. With 32 cores, every thread waits in line for the same lock all day: lock contention. Go's scheduler in 2012 worked like this, and Dmitry Vyukov's design document listed the single global mutex as the first problem. His fix: give each stove its own rail and touch the shared one rarely. That is the G-M-P model.
G, M and P (the kitchen)
| Kitchen | Runtime | What it is |
|---|---|---|
| an order ticket | G (goroutine) | the code to run, its own stack, and how far it got |
| a cook | M (machine) | an OS thread; only an M actually runs code on a CPU |
| a stove station with its own rail | P (processor) | the right to run Go code, plus a local run queue of up to 256 Gs |
| the rail at the door | global run queue | shared queue for overflow, protected by a lock |
Rules:
- An M must hold a P to run Go code. An M can exist without a P (blocked in a system call, or idle).
- There are exactly GOMAXPROCS Ps (by default, the number of CPUs; 2 on this machine). So at most GOMAXPROCS goroutines run Go code at the same moment.
- A new goroutine (
go f()) goes on the local run queue of the current P. No lock is needed, because only that P uses it. - If the local queue is full (256), half of it moves to the global run queue.
- For fairness, a P looks at the global queue every 61st time it schedules, so goroutines there are never forgotten.
Run queues and work stealing (seen for real)
GODEBUG is an environment variable with runtime debugging settings. GODEBUG=schedtrace=100 makes the runtime print a line about the scheduler every 100 ms. It must be set before the program starts, so the program runs itself again as a child process (exec.Command(os.Args[0])) with that setting. os.Args[0] is the path of the running program.
package main
import (
"fmt"
"os"
"os/exec"
"time"
)
func work() { // some chopping: pure CPU work
for i, x := 0, 0; i < 30_000_000; i++ {
x += i
}
}
func main() {
if os.Getenv("CHILD") == "" { // re-run ourselves with tracing on
cmd := exec.Command(os.Args[0])
cmd.Env = append(os.Environ(), "CHILD=1", "GODEBUG=schedtrace=100")
cmd.Stdout, cmd.Stderr = os.Stdout, os.Stdout
cmd.Run()
return
}
for i := 0; i < 1000; i++ {
go work() // 1000 orders, all created by main
}
time.Sleep(450 * time.Millisecond)
fmt.Println("done")
}
Real output:
SCHED 0ms: gomaxprocs=2 idleprocs=0 threads=4 spinningthreads=1 needspinning=0 idlethreads=0 runqueue=0 [2 0]
SCHED 105ms: gomaxprocs=2 idleprocs=0 threads=4 spinningthreads=0 needspinning=1 idlethreads=1 runqueue=648 [118 218]
SCHED 210ms: gomaxprocs=2 idleprocs=0 threads=4 spinningthreads=0 needspinning=1 idlethreads=1 runqueue=652 [109 210]
SCHED 311ms: gomaxprocs=2 idleprocs=0 threads=4 spinningthreads=0 needspinning=1 idlethreads=1 runqueue=655 [100 202]
SCHED 412ms: gomaxprocs=2 idleprocs=0 threads=4 spinningthreads=0 needspinning=1 idlethreads=1 runqueue=656 [90 193]
done
How to read a line:
| Field | Meaning |
|---|---|
gomaxprocs=2 |
2 Ps |
idleprocs=0 |
no P is idle: both stoves are busy |
threads=4 |
4 Ms exist |
spinningthreads |
Ms actively looking for work |
idlethreads |
Ms parked with nothing to do |
runqueue=648 |
goroutines in the global run queue |
[118 218] |
the local run queues of P0 and P1 |
main created all 1000 goroutines on one P. That P's local queue filled at 256, and half of it moved to the global queue again and again (hundreds there). Yet both local queues have work: the other P got some by work stealing.
Work stealing
When a P has nothing in its local queue, it looks for work in this order: its local queue, the global queue, the network poller (below), and then it picks another P at random and steals about half of that P's local queue. Like a free cook taking half the tickets from a busy station's rail, so no stove stands idle while there is work.
Preemption: the cook who never looks up
Preemption means taking the CPU away from a running goroutine so another can run.
- Cooperative preemption: the goroutine steps aside by itself at safe moments: function calls (the function's start checks a flag), blocking on channels, locks, sleeps, I/O, or
runtime.Gosched()(episode 43). It only works if every goroutine reaches such a moment. - A loop with no function calls inside never reaches one. With one P, it would keep the CPU forever.
- sysmon (system monitor) is a runtime thread that runs without a P, wakes up regularly (starting every 20 microseconds, sleeping longer when idle) and watches the Ps. If a goroutine has run for more than 10 ms, sysmon asks it to step aside.
- Asynchronous preemption (Go 1.14): the runtime sends the thread a signal (an interrupt the operating system delivers to a thread, like
SIGURGon Unix), and the goroutine is stopped right where it is, even in the middle of a loop.
GODEBUG=asyncpreemptoff=1 switches asynchronous preemption off, which lets us see the old behaviour. context.WithTimeout (episode 51) kills the child process after 2 seconds.
package main
import (
"context"
"fmt"
"os"
"os/exec"
"runtime"
"time"
)
func hog() { // a cook who never looks up from the pan
for i := 0; ; i++ {
}
}
func main() {
if os.Getenv("CHILD") == "" {
for _, debug := range []string{"asyncpreemptoff=1", ""} {
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
cmd := exec.CommandContext(ctx, os.Args[0])
cmd.Env = append(os.Environ(), "CHILD=1", "GODEBUG="+debug)
cmd.Stdout = os.Stdout
fmt.Printf("GODEBUG=%q: ", debug)
if cmd.Run() != nil {
fmt.Println("killed after 2s, main never got the CPU back")
}
cancel()
}
return
}
runtime.GOMAXPROCS(1) // one stove
start := time.Now()
go hog()
time.Sleep(10 * time.Millisecond) // main steps away briefly
fmt.Println("main is back after", time.Since(start).Round(10*time.Millisecond))
}
Real output (five more runs gave 10ms, 20ms, 20ms, 20ms, 10ms):
GODEBUG="asyncpreemptoff=1": killed after 2s, main never got the CPU back
GODEBUG="": main is back after 20ms
Without asynchronous preemption, main never gets the only P back: the program is stuck, exactly how Go behaved before 1.14. With it, sysmon notices hog running too long and the signal pushes it aside in about 10-20 ms.
Blocking system calls and handoff
A system call (syscall) is a request from a program to the kernel: read a file, write to a pipe, open a socket. A blocking system call puts the thread to sleep inside the kernel until it can finish, like a cook waiting at the door for the milkman.
Handoff: when an M blocks in a syscall, its P would sit idle. So sysmon (after at least 20 microseconds) or the scheduler takes the P away and gives it to another M (a spare idle thread, or a new one). When the syscall returns, the goroutine must get a P again before it continues. That is why blocking syscalls can create extra threads, but never stop other goroutines.
A pipe is a one-way channel between two file descriptors (small numbers the kernel uses for open files). syscall.Pipe makes a plain pipe whose reads really block (pipes from os.Pipe are non-blocking and use the network poller instead).
package main
import (
"fmt"
"runtime"
"syscall"
"time"
)
func main() {
runtime.GOMAXPROCS(1) // one stove
var fd [2]int
syscall.Pipe(fd[:]) // a plain, blocking pipe
go func() { // waits for the milk delivery
buf := make([]byte, 16)
n, _ := syscall.Read(fd[0], buf) // blocks in the kernel
fmt.Println("delivery arrived:", string(buf[:n]))
}()
go func() { // the milkman comes just before 1 second
time.Sleep(900 * time.Millisecond)
syscall.Write(fd[1], []byte("milk"))
}()
for i := 1; i <= 6; i++ { // meanwhile, keep cooking
fmt.Println("tick", i, "- still cooking other orders")
time.Sleep(250 * time.Millisecond)
}
}
Real output:
tick 1 - still cooking other orders
tick 2 - still cooking other orders
tick 3 - still cooking other orders
tick 4 - still cooking other orders
delivery arrived: milk
tick 5 - still cooking other orders
tick 6 - still cooking other orders
With only one P, main kept ticking while the reader slept in the kernel: the P was handed off.
The network poller
Blocking one thread per network connection is design 1 again. Instead, Go asks the kernel to watch all sockets at once (epoll on Linux, kqueue on macOS/BSD, I/O completion ports on Windows) and to report which ones are ready. This watcher is the network poller (netpoller). A goroutine that reads from a connection with no data is parked (set aside, not running, holding no thread); when the poller reports the socket ready, the goroutine goes back on a run queue. Like a receptionist who watches every phone and calls the kitchen only when someone answers.
A TCP connection is a reliable two-way network connection between two programs; net.Listen waits for connections, net.Dial makes one.
package main
import (
"fmt"
"net"
"runtime/pprof"
"syscall"
"time"
)
func threads() int { return pprof.Lookup("threadcreate").Count() }
func main() {
// 1000 customers on the network, each waiting for a reply
ln, _ := net.Listen("tcp", "127.0.0.1:0")
for i := 0; i < 1000; i++ {
go func() {
conn, _ := net.Dial("tcp", ln.Addr().String())
conn.Read(make([]byte, 1)) // waits; nobody replies
}()
}
time.Sleep(time.Second)
fmt.Println("1000 network waits -> OS threads:", threads())
// 50 waits on plain blocking pipes (raw system calls)
for i := 0; i < 50; i++ {
go func() {
var fd [2]int
syscall.Pipe(fd[:])
syscall.Read(fd[0], make([]byte, 1)) // blocks a thread
}()
}
time.Sleep(time.Second)
fmt.Println("+50 blocking syscalls -> OS threads:", threads())
}
Real output (three runs: 6/55, 6/55, 7/55):
1000 network waits -> OS threads: 6
+50 blocking syscalls -> OS threads: 55
1000 network waits used a handful of threads; 50 raw blocking syscalls needed about one thread each. Network code in Go scales to huge numbers of connections; raw blocking syscalls each cost a thread. (127.0.0.1:0 means "this machine, any free port".)
Goroutine stacks grow by copying
A stack is the memory where a function keeps its local variables; each call gets a block called a frame. Threads get a big, fixed stack. A goroutine starts with 2 KB (since Go 1.4; 8 KB before). At the start of each function call, a short check sees whether the stack has room; if not, the runtime allocates a stack twice the size, copies every frame there, and updates every pointer into the old stack. This is stack growth by copying (contiguous stacks). Like moving all the plates to a shelf twice as big, in the same order.
Recursion is a function calling itself (episode 10). unsafe.Pointer(&x) turned into a uintptr gives the address of x as a plain number; the unsafe package does low-level things that bypass Go's safety, and we only use it to look.
package main
import (
"fmt"
"unsafe"
)
// depth counts plates down a stack of n plates
func depth(n int) int {
if n == 0 {
return 0
}
return 1 + depth(n-1)
}
func main() {
x := 42 // lives on main's stack
before := uintptr(unsafe.Pointer(&x))
fmt.Println("deep recursion:", depth(1_000_000))
after := uintptr(unsafe.Pointer(&x))
fmt.Println("x is still", x)
fmt.Println("x moved in memory:", before != after)
}
Real output:
deep recursion: 1000000
x is still 42
x moved in memory: true
A million frames do not fit in 2 KB, so the stack grew many times; x moved with it and your code never noticed. This is one reason Go has no pointer arithmetic (episode 11): the runtime must be free to move stacks.
The limit: stack overflow
debug.SetMaxStack sets the largest stack a single goroutine may have (default 1 GB on 64-bit).
package main
import (
"fmt"
"runtime/debug"
)
func depth(n int) int {
if n == 0 {
return 0
}
return 1 + depth(n-1)
}
func main() {
fmt.Println("100 plates:", depth(100))
debug.SetMaxStack(1 << 20) // allow only 1 MB of stack
fmt.Println("1,000,000 plates:", depth(1_000_000))
}
Real output (start):
100 plates: 100
runtime: goroutine stack exceeds 1048576-byte limit
runtime: sp=0xc0001e0388 stack=[0xc0001e0000, 0xc0002e0000]
fatal error: stack overflow
runtime stack:
...
Recursion with no base case hits the default limit (real output, after about 4 seconds of growing to 1 GB):
func forever(n int) int { return forever(n+1) + 1 }
runtime: goroutine stack exceeds 1000000000-byte limit
runtime: sp=0xc020160398 stack=[0xc020160000, 0xc040160000]
fatal error: stack overflow
A stack overflow is a fatal error, not a panic: recover (episode 26) cannot catch it. Fix the recursion (add a base case, or use a loop).
GOMAXPROCS and container-aware GOMAXPROCS (Go 1.25)
GOMAXPROCS is the number of Ps: how many goroutines can run Go code at the same moment. runtime.GOMAXPROCS(0) only reads it; runtime.GOMAXPROCS(n) sets it and returns the old value; the environment variable GOMAXPROCS=n sets it at start.
package main
import (
"fmt"
"os"
"os/exec"
"runtime"
)
func main() {
fmt.Println("NumCPU:", runtime.NumCPU(), " GOMAXPROCS:", runtime.GOMAXPROCS(0))
if os.Getenv("GOMAXPROCS") == "" { // run again, set by hand
cmd := exec.Command(os.Args[0])
cmd.Env = append(os.Environ(), "GOMAXPROCS=1")
cmd.Stdout = os.Stdout
fmt.Print("with GOMAXPROCS=1 -> ")
cmd.Run()
}
}
Real output:
NumCPU: 2 GOMAXPROCS: 2
with GOMAXPROCS=1 -> NumCPU: 2 GOMAXPROCS: 1
Containers. A container packages one app with what it needs and runs it on a shared machine with a share of the resources. On Linux the share is enforced by cgroups (control groups). Kubernetes (a widely used system for running containers on many machines) calls the CPU share a CPU limit. The cgroup CPU bandwidth limit says how much CPU time the container may use in each short period.
- Before Go 1.25: GOMAXPROCS defaulted to
runtime.NumCPU(), the CPUs the machine has. On a 64-CPU machine with a 2-CPU limit, Go used 64 Ps; together they used up the 2-CPU allowance early in each period and Linux paused the app until the next one. That pause is CPU throttling, and it shows up as slow, jumpy response times. The fix used to be settingGOMAXPROCSby hand (or a library that reads the cgroup). - Go 1.25 (August 2025): on Linux, if the cgroup CPU limit is lower than the CPU count, GOMAXPROCS defaults to the limit; and on all OSes the runtime periodically updates GOMAXPROCS if the CPU count or the limit changes. Kubernetes "CPU requests" are not considered.
- Both behaviours are switched off if you set GOMAXPROCS yourself (env var or
runtime.GOMAXPROCS), or withGODEBUG=containermaxprocs=0andGODEBUG=updatemaxprocs=0.
(The runs above used Go 1.24.7 on a machine without a CPU limit, so the container rule is described from the release notes, not shown.)
Mini project: kitchen inspector
Write a program that runs itself twice, with GOMAXPROCS=1 and GOMAXPROCS=2. Each run starts 4 CPU-heavy goroutines and 20 goroutines blocked on a raw pipe read, waits for the CPU work, and prints the time taken and the number of OS threads created. Bonus: add GODEBUG=schedtrace=100 and read the queues.
Solution
sync.WaitGroup waits until all the CPU goroutines have called Done; it is explained properly in episode 45.
package main
import (
"fmt"
"os"
"os/exec"
"runtime"
"runtime/pprof"
"sync"
"syscall"
"time"
)
// chop is pure CPU work.
func chop() {
x := 0
for i := 0; i < 300_000_000; i++ {
x += i
}
_ = x
}
func main() {
if os.Getenv("CHILD") == "" {
for _, procs := range []string{"1", "2"} {
cmd := exec.Command(os.Args[0])
cmd.Env = append(os.Environ(), "CHILD=1", "GOMAXPROCS="+procs)
cmd.Stdout = os.Stdout
cmd.Run()
}
return
}
start := time.Now()
var wg sync.WaitGroup // waits for all goroutines (episode 45)
for i := 0; i < 4; i++ { // 4 CPU-heavy orders
wg.Add(1)
go func() { defer wg.Done(); chop() }()
}
for i := 0; i < 20; i++ { // 20 orders stuck waiting on a pipe
go func() {
var fd [2]int
syscall.Pipe(fd[:])
syscall.Read(fd[0], make([]byte, 1))
}()
}
wg.Wait()
fmt.Printf("GOMAXPROCS=%d: 4 CPU jobs took %v, OS threads: %d\n",
runtime.GOMAXPROCS(0), time.Since(start).Round(100*time.Millisecond),
pprof.Lookup("threadcreate").Count())
}
Real output (two runs):
GOMAXPROCS=1: 4 CPU jobs took 600ms, OS threads: 22
GOMAXPROCS=2: 4 CPU jobs took 200ms, OS threads: 23
GOMAXPROCS=1: 4 CPU jobs took 500ms, OS threads: 22
GOMAXPROCS=2: 4 CPU jobs took 200ms, OS threads: 23
With 2 Ps the CPU work runs on both cores at once (about 2-3 times as fast in these runs). The 20 blocked readers need about 20 threads whatever GOMAXPROCS is: GOMAXPROCS limits threads running Go code, not threads blocked in syscalls.
Common mistakes (and the real messages)
| Mistake | Real message / result | Why | Fix |
|---|---|---|---|
one locked thread per job (LockOSThread everywhere) |
runtime: program exceeds 100-thread limit / fatal error: thread exhaustion |
threads are heavy and limited | plain goroutines |
| thousands of raw blocking syscalls | +50 blocking syscalls -> OS threads: 55 |
each blocked syscall holds a thread | use net/os (poller), limit concurrency |
| recursion with no base case | runtime: goroutine stack exceeds 1000000000-byte limit / fatal error: stack overflow |
stack limit (1 GB) | add a base case, or a loop |
recover to survive a stack overflow |
fatal error: stack overflow (program ends anyway) |
fatal errors are not panics | fix the recursion |
| GOMAXPROCS = all CPUs inside a container with a lower CPU limit (before Go 1.25) | slow, jumpy responses (CPU throttling) | more Ps than the CPU limit | Go 1.25+, or set GOMAXPROCS |
| relying on a busy loop never being interrupted | (Go 1.14+) the loop is preempted after ~10 ms | asynchronous preemption | use real synchronisation (episodes 45-50) |
| expecting goroutines to run in creation order | order varies run to run | local queues, stealing, preemption | never depend on scheduling order |
Practice
- Predict the output, then run it (Go 1.14 or newer):
package main
import (
"fmt"
"runtime"
"time"
)
func main() {
runtime.GOMAXPROCS(1)
go func() {
for {
}
}()
time.Sleep(time.Millisecond)
fmt.Println("done")
}
- Why does a goroutine blocked in
syscall.Readnot freeze a program withGOMAXPROCS=1? - Print the old value returned by
runtime.GOMAXPROCS(1)and the new value. - (Homework) Run any program of yours with
GODEBUG=schedtrace=1000 go run main.goand explain each field. - (Homework) Read the top comment of
$(go env GOROOT)/src/runtime/proc.go.
Solution 1
Real output: done. The empty loop has no function calls, but since Go 1.14 sysmon preempts it with a signal, so main gets the only P back.
Solution 2
The M (thread) blocks in the kernel, but its P is handed off to another M, so other goroutines keep running. When the syscall returns, the goroutine waits for a P again.
Solution 3
package main
import (
"fmt"
"runtime"
)
func main() {
old := runtime.GOMAXPROCS(1) // set to 1, returns the previous value
fmt.Println("previous:", old, "now:", runtime.GOMAXPROCS(0))
}
Real output on a 2-CPU machine: previous: 2 now: 1
Quick quiz
1. One P, a goroutine spinning in for {}, main sleeps 1 ms then prints done. Go 1.24: what happens? (From the video.)
A. Hangs forever · B. Prints done · C. fatal error: all goroutines are asleep · D. panic: deadlock
Answer
B. Asynchronous preemption (Go 1.14) takes the P from the loop. Before Go 1.14 it could hang (A).
2. What do the numbers in runqueue=648 [118 218] mean?
Answer
648 goroutines in the global run queue; 118 and 218 in the local run queues of P0 and P1.
3. How big is a new goroutine's stack, and what happens when it is full?
Answer
2 KB. The runtime allocates a stack twice as big, copies all frames and fixes pointers into it.
4. A P's local run queue is empty and the global queue is empty. What does it do?
Answer
It checks the network poller, then steals about half of the goroutines from a random other P.
Interview questions
- Explain the G-M-P model. G = goroutine, M = OS thread, P = processor (permission to run Go code plus a local run queue). An M needs a P to run Go code; there are GOMAXPROCS Ps.
- Why not one OS thread per goroutine? Threads have big stacks, kernel-level context switches and a limit (10,000 by default in Go); goroutines start at 2 KB and switch in user space.
- What is work stealing? An idle P takes about half of a random busy P's local run queue, balancing load without a global lock.
- Why does the global run queue exist, and how is starvation avoided? It holds overflow (local queues hold 256) and goroutines without a P; each P checks it every 61st schedule.
- Cooperative vs asynchronous preemption? Cooperative: goroutines yield at safe points (function calls, blocking). Asynchronous (Go 1.14): sysmon signals a goroutine that ran over ~10 ms and stops it anywhere.
- What happens when a goroutine makes a blocking system call? Its M blocks in the kernel; the P is handed off to another M; on return the goroutine needs a P again.
- What is the network poller? A runtime component using epoll/kqueue/IOCP to wait on many sockets with few threads; waiting goroutines are parked, not blocking threads.
- How do goroutine stacks grow? Start at 2 KB; when full, a stack twice the size is allocated and frames copied (contiguous stacks, Go 1.4). Max 1 GB on 64-bit.
- What is GOMAXPROCS and what changed in Go 1.25? The number of Ps. Go 1.25 defaults it to the cgroup CPU limit on Linux when that is lower than the CPU count, and updates it periodically.
- Is a stack overflow recoverable? No: it is a fatal error, not a panic.
Glossary
| Term | Meaning | Example |
|---|---|---|
| CPU core | a part of the processor that runs one piece of code at a time | this machine has 2 |
| Goroutine | a lightweight function running concurrently, started with go |
go cook(order) |
| Runtime | code Go adds to every program to manage goroutines, memory and GC | runtime.NumGoroutine() |
| Scheduler | the part of the runtime deciding which goroutine runs where and when | Go's G-M-P scheduler |
| OS thread | a worker the operating system can run on a CPU | each M is one |
| Kernel | the core of the operating system that controls the hardware | Linux |
| Context switch | the CPU switching from one thread to another, saving and loading state | done by the kernel for threads |
runtime.LockOSThread |
glue the calling goroutine to its current thread | used in design 1 |
debug.SetMaxThreads |
set the most OS threads the program may use (default 10,000) | debug.SetMaxThreads(100) |
| Thread exhaustion | fatal error when the thread limit is exceeded | fatal error: thread exhaustion |
| M:N scheduling | running M goroutines on N OS threads | 10,000 goroutines on 4 threads |
pprof.Lookup("threadcreate") |
profile counting created OS threads | .Count() → 4 |
| Run queue | a list of jobs waiting to run | a ticket rail |
| Lock (mutex) | lets only one thread use something at a time | the rail's key |
| Lock contention | many threads waiting for the same lock | 32 cooks, one key |
| G-M-P model | Go's scheduler design: goroutines (G) run on threads (M) holding processors (P) | gomaxprocs=2 |
| G | a goroutine: code, stack and state | the order ticket |
| M (machine) | an OS thread that runs code | the cook |
| P (processor) | the right to run Go code plus a local run queue; GOMAXPROCS of them | the stove station |
| Local run queue | a P's own queue of runnable goroutines, up to 256 | [118 218] |
| Global run queue | a shared queue for overflow, behind a lock | runqueue=648 |
| Work stealing | an idle P taking about half of another P's queue | P1 got work from P0 |
| Fairness | every waiting goroutine eventually runs | global queue checked every 61st schedule |
| GODEBUG | environment variable with runtime debug settings | GODEBUG=schedtrace=100 |
| schedtrace | GODEBUG setting printing scheduler state periodically | SCHED 105ms: ... |
| Child process | a program started by another program | exec.Command(os.Args[0]) |
| Preemption | taking the CPU away from a running goroutine | after ~10 ms |
| Cooperative preemption | goroutines stepping aside at safe points | function calls, runtime.Gosched() |
| Asynchronous preemption | stopping a goroutine anywhere using a signal (Go 1.14) | the hog loop |
| Signal | an interrupt the OS delivers to a thread | SIGURG |
| sysmon | runtime monitor thread running without a P | preempts hogs, retakes Ps |
asyncpreemptoff=1 |
GODEBUG setting that disables async preemption | the stuck run |
| Deadlock | no goroutine can make progress | main never gets the P back |
| I/O | input and output: files, disks, network | reading a pipe |
| System call (syscall) | a request from a program to the kernel | syscall.Read |
| Blocking system call | a syscall that sleeps in the kernel until it can finish | reading an empty pipe |
| Handoff | moving a P from a blocked M to another M | ticks continue |
| Pipe | a one-way channel between two file descriptors | syscall.Pipe(fd[:]) |
| File descriptor | a small number the kernel uses for an open file or socket | fd[0] |
| TCP connection | a reliable two-way network connection | net.Dial("tcp", addr) |
| Socket | the program's end of a network connection | conn |
| Network poller (netpoller) | runtime part that waits on many sockets with epoll/kqueue/IOCP | 1000 waits, 6 threads |
| epoll / kqueue / IOCP | kernel ways to wait on many sockets (Linux / macOS, BSD / Windows) | netpoll_epoll.go |
| Parked goroutine | a goroutine set aside, not running and holding no thread | waiting on conn.Read |
| Stack | memory holding a function's local variables | where x lives |
| Frame | one function call's block on the stack | one per depth call |
| Recursion | a function calling itself | depth(n-1) |
| Stack growth by copying | allocate a 2x stack, copy frames, fix pointers | x moved in memory: true |
| Contiguous stack | one block of stack memory, moved when it grows (Go 1.4) | replaced segmented stacks |
unsafe / uintptr |
low-level package / an integer that can hold an address | uintptr(unsafe.Pointer(&x)) |
debug.SetMaxStack |
set the largest stack a goroutine may have (default 1 GB) | debug.SetMaxStack(1 << 20) |
| Stack overflow | a stack growing past its limit: fatal error | fatal error: stack overflow |
| Fatal error | an error that ends the program and cannot be recovered | thread exhaustion, stack overflow |
| GOMAXPROCS | the number of Ps | runtime.GOMAXPROCS(0) |
runtime.NumCPU |
logical CPUs the machine has | 2 |
| Container | a package holding one app, run with a share of a machine | a Docker container |
| cgroup | Linux control group: enforces a container's resource share | cpu.max |
| Kubernetes CPU limit | the CPU share a container may use | limits: cpu: 2 |
| CPU throttling | Linux pausing a container that used up its CPU share | slow, jumpy responses |
| Container-aware GOMAXPROCS | Go 1.25: default GOMAXPROCS follows the cgroup CPU limit and updates periodically | 2 on a 64-CPU machine with limit 2 |
Summary
- One OS thread per goroutine runs out of threads (
fatal error: thread exhaustion); one global queue with one lock doesn't scale. - G-M-P: G = goroutine, M = OS thread, P = processor. An M needs a P to run Go code; there are GOMAXPROCS Ps.
- Each P has a local run queue (256); overflow goes to the global run queue, checked every 61st schedule.
- An idle P checks the global queue and the network, then steals about half of a random P's queue.
- Cooperative preemption at safe points; asynchronous preemption (Go 1.14) by signal after ~10 ms, driven by sysmon.
- A blocking system call keeps its thread but hands off the P; the network poller parks network waits without holding threads.
- Goroutine stacks start at 2 KB and grow by copying (2x); the limit is 1 GB, and a stack overflow is fatal.
- Go 1.25: GOMAXPROCS follows the container's CPU limit and is updated periodically, unless you set it yourself.
Next episode: WaitGroup and Mutex: sync.WaitGroup, WaitGroup.Go (Go 1.25), race conditions, critical sections, sync.Mutex, sync.RWMutex, lock granularity, copying a lock bug and go vet copylocks.