SyntaxPilot
Courses/AI: Zero to Agents/Lesson 9

AI: Zero to Agents · Lesson 9 of 40 · 12 min

Neural networks

queued

These are the notes for AI episode 9. In episode 8 one neuron failed at the staircase light (XOR). Here you connect neurons into a neural network that solves it, then make the network learn by itself. Along the way: input, hidden and output layers, fully connected (dense) layers, the forward pass, why activations matter, random initialisation and the symmetry problem, local minima, counting parameters, and backpropagation. Every Python example below runs as-is in the SyntaxPilot compiler, and the outputs shown are real (Python 3.13).

A team, not a hero: the restaurant kitchen

In a kitchen In a neural network
The customer's order comes in Input layer: the raw numbers
Helper cooks, each doing one small job, unseen by the customer Hidden layer: neurons that find simple patterns
The head chef combines their work into the dish Output layer: the final answer

Split XOR into two easy questions

  • Question 1: is at least one switch up? That's OR, and one neuron can learn it (one straight line splits it).
  • Question 2: are both switches up? That's AND, also easy for one neuron.
  • Combine: the light is ON when OR is yes but AND is no.
switches (a, b)  →  [OR neuron, AND neuron]  →  [OR but not AND]  →  light
   input layer          hidden layer               output layer

Demo 1: a network built by hand

# staircase light: on when the two switches differ
switches = [(0, 0, 0), (0, 1, 1), (1, 0, 1), (1, 1, 0)]

def neuron(inputs, weights, bias):
    total = sum(x * w for x, w in zip(inputs, weights)) + bias
    return 1 if total >= 0 else 0

for a, b, on in switches:
    h1 = neuron([a, b], [1, 1], -0.5)     # hidden neuron 1: OR
    h2 = neuron([a, b], [1, 1], -1.5)     # hidden neuron 2: AND
    out = neuron([h1, h2], [1, -1], -0.5) # output: OR but not AND
    print(f"switches {a} {b} -> OR {h1}, AND {h2} -> light {out}  (should be {on})")

Output:

switches 0 0 -> OR 0, AND 0 -> light 0  (should be 0)
switches 0 1 -> OR 1, AND 0 -> light 1  (should be 1)
switches 1 0 -> OR 1, AND 0 -> light 1  (should be 1)
switches 1 1 -> OR 1, AND 1 -> light 0  (should be 0)
  • OR: weights 1 and 1, bias −0.5: any switch up makes the total ≥ 0.
  • AND: bias −1.5: needs both switches.
  • Output: +1 × OR − 1 × AND − 0.5: fires only for OR without AND.

Three neurons in two layers solve what one neuron couldn't.

The words for what you built

Term Meaning
Input layer the raw numbers coming in (here: two switches)
Hidden layer neurons in between; their answers are stepping stones, not the final output
Output layer the neuron(s) that give the answer
Fully connected (dense) layer every neuron receives every output of the layer before
Forward pass running the inputs through the layers, one after another, to get the output
Deep network a network with many hidden layers; learning with them is deep learning

Why every layer needs an activation

Without activations, each layer is just a weighted sum (a straight line), and lines stacked on lines are still a line:

layer 1: y = 3x + 1
layer 2: z = 2y + 5
together: z = 2(3x + 1) + 5 = 6x + 7     ← still one straight line

A hundred layers without activations collapse into one line, and XOR stays impossible. The bend each activation adds (the step's jump, the sigmoid's curve, ReLU's corner) is what gives a network its power.

Demo 2: learning from all zeros gets stuck

Nine knobs in a list p: two hidden sigmoid neurons (3 knobs each: 2 weights + bias) and one output neuron (2 weights + bias). (The video squeezes the program into 20 lines so it fits on screen, so there are no blank lines.)

import math, random
switches = [(0, 0, 0), (0, 1, 1), (1, 0, 1), (1, 1, 0)]
def sig(t): return 1 / (1 + math.exp(-t))
def network(p, a, b):                  # 2 inputs -> 2 hidden -> 1 output
    h1 = sig(p[0] * a + p[1] * b + p[2])
    h2 = sig(p[3] * a + p[4] * b + p[5])
    return sig(p[6] * h1 + p[7] * h2 + p[8])
def loss(p): return sum((network(p, a, b) - on) ** 2 for a, b, on in switches) / 4
random.seed(1)
p, h, lr = [0.0] * 9, 0.0001, 2.0      # every weight starts at 0
for epoch in range(1, 4001):
    slopes = []
    for i in range(9):
        nudged = p.copy(); nudged[i] += h
        slopes.append((loss(nudged) - loss(p)) / h)
    p = [p[i] - lr * slopes[i] for i in range(9)]
    if epoch % 1000 == 0: print(f"epoch {epoch}: loss {loss(p):.4f}")
print("lights:", [round(network(p, a, b), 2) for a, b, on in switches], "should be [0, 1, 1, 0]")
print("hidden 1:", [round(x, 2) for x in p[0:3]], " hidden 2:", [round(x, 2) for x in p[3:6]])

Output:

epoch 1000: loss 0.2500
epoch 2000: loss 0.2500
epoch 3000: loss 0.2500
epoch 4000: loss 0.2500
lights: [0.5, 0.5, 0.5, 0.5] should be [0, 1, 1, 0]
hidden 1: [0.0, 0.0, 0.0]  hidden 2: [0.0, 0.0, 0.0]

Why (the symmetry problem): two neurons that start exactly the same compute the same output and get the same slopes, so every step changes them identically. They stay "identical twins" forever, and two copies of one neuron are no better than one.

The fix: random initialisation

Kitchen version: if both helper cooks get the same instructions, both make dal and nobody makes rotis. Give them different first tasks, even at random, and each specialises. For a network: start every weight at a small random number.

Demo 3: a random start learns XOR

The only change is one line: p, h, lr = [random.uniform(-1, 1) for _ in range(9)], 0.0001, 2.0 # random start. random.seed(1) makes the random numbers repeatable.

Output (seed 1):

epoch 1000: loss 0.0056
epoch 2000: loss 0.0011
epoch 3000: loss 0.0006
epoch 4000: loss 0.0004
lights: [0.02, 0.98, 0.98, 0.02] should be [0, 1, 1, 0]
hidden 1: [-5.98, 6.12, 3.03]  hidden 2: [-5.7, 5.52, -2.99]

The network learned XOR by itself, and the two hidden neurons ended up different: each found its own job.

Output with random.seed(2) (a different random start):

epoch 1000: loss 0.1277
epoch 2000: loss 0.1261
epoch 3000: loss 0.1257
epoch 4000: loss 0.1255
lights: [0.02, 0.5, 0.98, 0.5] should be [0, 1, 1, 0]
hidden 1: [4.25, 8.67, -0.91]  hidden 2: [-3.6, 8.61, 2.67]

This start walked into a local minimum (episode 7): a dip that isn't the lowest valley. In practice, try another random start, or use more neurons (see Practice 2).

Demo 4: counting knobs, and the cost of nudging

A tiny handwritten-digit reader: a 28 × 28 pixel image (784 numbers) → 128 hidden neurons → 10 outputs (one per digit).

layers = [784, 128, 10]    # 28x28 digit image -> 128 hidden -> 10 digits
params = 0
for n_in, n_out in zip(layers, layers[1:]):
    weights, biases = n_in * n_out, n_out
    print(f"{n_in:3} -> {n_out:3}: {weights:,} weights + {biases} biases")
    params += weights + biases
print(f"total parameters: {params:,}")

ms_per_run = 1             # say one run of the network takes 1 ms
steps = 10_000
nudge = (params + 1) * ms_per_run * steps / 1000 / 3600
backprop = 3 * ms_per_run * steps / 1000 / 60
print(f"nudging every knob: {nudge:,.0f} hours")
print(f"backpropagation:    {backprop:.1f} minutes")

Output:

784 -> 128: 100,352 weights + 128 biases
128 ->  10: 1,280 weights + 10 biases
total parameters: 101,770
nudging every knob: 283 hours
backpropagation:    0.5 minutes
  • Parameters of a dense layer = inputs × neurons (weights) + neurons (biases).
  • Nudging runs the whole network once more for every knob, every step: 283 hours (almost 12 days) for a toy, and modern models have billions of parameters.
  • The backpropagation line assumes one backward pass costs about the same as a few forward passes (here, 3 runs per step).

Backpropagation: blame flows backwards

"The dal is too salty!" In a network
The customer complains: the error at the end the loss, measured at the output
The head chef thinks backwards: the dal cook added most salt the blame flows from the output back through each layer
Each cook gets a share in proportion to their contribution, and passes some back each weight gets its slope: its share of the blame
Everyone adjusts at once every weight takes a step downhill
  • Forward pass: inputs → output, then the loss says how wrong it is.
  • Backward pass: the blame is sent from the output back towards the inputs, layer by layer, using the chain rule from calculus.
  • One backward pass gives the slope of every weight, in about the cost of a few forward passes. This is backpropagation.
  • Made famous by "Learning representations by back-propagating errors", David Rumelhart, Geoffrey Hinton and Ronald Williams, Nature 323, 1986 (abstract).

One training step (every neural network)

  1. Forward pass on a batch of examples.
  2. Loss: how wrong were the answers?
  3. Backward pass (backpropagation): every weight gets its slope.
  4. Update: every weight steps downhill × the learning rate. Repeat for many epochs.

In PyTorch, step 3 is the single line loss.backward().

Glossary

Term Meaning Example
Neural network neurons connected in layers 2 hidden + 1 output neuron for XOR
Layer a group of neurons that all work on the same inputs the hidden layer
Input layer the raw numbers fed in two switches; 784 pixels
Hidden layer neurons between input and output the OR and AND neurons
Output layer the neurons that give the answer the light; 10 digit scores
Fully connected (dense) every neuron gets every output of the layer before 784 × 128 weights
Forward pass running inputs through the layers to the output network(p, a, b)
Deep network / deep learning many hidden layers / learning with them image and language models
Parameter a number the model learns (a weight or a bias) 101,770 in the digit reader
Initialisation the starting values of the weights [0.0] * 9 vs random
Symmetry problem identical neurons get identical slopes and stay identical the all-zero start
Random initialisation starting weights at small random numbers to break symmetry random.uniform(-1, 1)
Seed a number that makes "random" numbers repeatable random.seed(1)
Local minimum a dip in the loss that isn't the lowest seed 2: loss 0.1255
Backpropagation sending the blame backwards to get every weight's slope at once loss.backward()
Chain rule the calculus rule for passing slopes back through layers used inside backprop
OR / AND / XOR at least one / both / exactly one input is 1 the staircase light is XOR

Common mistakes

Mistake Code Error
Starting every weight at the same value p = [0.0] * 9 no error; loss stuck at 0.2500
Too few knobs p = [0.1] * 8 then p[8] IndexError: list index out of range
Forgetting an input network(p, a) TypeError: network() missing 1 required positional argument: 'b'
Stacking layers with no activation z = 2 * (3 * x + 1) + 5 no error; still one straight line, XOR impossible

Fixes: start with small random weights; count knobs per layer (inputs × neurons + biases); pass every input; put an activation after every hidden layer.

Mini project: a network for a new light

Make the light turn on when the two switches are the same (XNOR). First hand-build it (reuse OR and AND), then train the random-start network from Demo 3 on the new data and check that it learns it too.

Practice

  1. Train Demo 3 with seeds 1 to 5. Which ones get stuck?
  2. Give the network 3 hidden neurons. Does seed 2 still get stuck?
  3. Count the parameters of a 784 → 256 → 128 → 10 network.
  4. Hand-build a network for "light on when both switches are the same".
Solutions

1 and 2. A general version of the network with any number of hidden neurons (with 2 hidden neurons it uses the same knob layout and random numbers as Demo 3):

import math, random
switches = [(0, 0, 0), (0, 1, 1), (1, 0, 1), (1, 1, 0)]

def sig(t):
    return 1 / (1 + math.exp(-t))

def network(p, a, b, hidden):
    hs = [sig(p[3*k] * a + p[3*k + 1] * b + p[3*k + 2]) for k in range(hidden)]
    out = p[3*hidden:3*hidden + hidden]
    return sig(sum(w * x for w, x in zip(out, hs)) + p[-1])

def train(seed, hidden=2, epochs=4000, lr=2.0, h=0.0001):
    random.seed(seed)
    n = 3 * hidden + hidden + 1
    p = [random.uniform(-1, 1) for _ in range(n)]
    loss = lambda p: sum((network(p, a, b, hidden) - on) ** 2 for a, b, on in switches) / 4
    for _ in range(epochs):
        slopes = []
        for i in range(n):
            nudged = p.copy(); nudged[i] += h
            slopes.append((loss(nudged) - loss(p)) / h)
        p = [p[i] - lr * slopes[i] for i in range(n)]
    return loss(p)

for seed in [1, 2, 3, 4, 5]:
    print(f"2 hidden, seed {seed}: loss {train(seed):.4f}")
print(f"3 hidden, seed 2: loss {train(2, hidden=3):.4f}")
2 hidden, seed 1: loss 0.0004
2 hidden, seed 2: loss 0.1255
2 hidden, seed 3: loss 0.0004
2 hidden, seed 4: loss 0.0004
2 hidden, seed 5: loss 0.1255
3 hidden, seed 2: loss 0.0005

Seeds 2 and 5 get stuck in the local minimum. With 3 hidden neurons, seed 2 learns XOR: more neurons give more ways downhill.

3.

layers = [784, 256, 128, 10]
params = 0
for n_in, n_out in zip(layers, layers[1:]):
    params += n_in * n_out + n_out
    print(f"{n_in:3} -> {n_out:3}: {n_in * n_out + n_out:,}")
print(f"total parameters: {params:,}")
784 -> 256: 200,960
256 -> 128: 32,896
128 ->  10: 1,290
total parameters: 235,146

4.

switches = [(0, 0, 1), (0, 1, 0), (1, 0, 0), (1, 1, 1)]   # light on when the SAME

def neuron(inputs, weights, bias):
    total = sum(x * w for x, w in zip(inputs, weights)) + bias
    return 1 if total >= 0 else 0

for a, b, on in switches:
    h1 = neuron([a, b], [1, 1], -0.5)     # OR
    h2 = neuron([a, b], [1, 1], -1.5)     # AND
    out = neuron([h1, h2], [-1, 1], 0.5)  # NOT OR, or AND
    print(f"switches {a} {b} -> light {out}  (should be {on})")
switches 0 0 -> light 1  (should be 1)
switches 0 1 -> light 0  (should be 0)
switches 1 0 -> light 0  (should be 0)
switches 1 1 -> light 1  (should be 1)

Quick quiz 1

A dense network: 3 inputs → 4 hidden neurons → 2 outputs. How many weights + biases in total? A: 9 · B: 20 · C: 26 · D: 24

Answer

C: 26. Layer 1: 3 × 4 = 12 weights + 4 biases = 16. Layer 2: 4 × 2 = 8 weights + 2 biases = 10. Total 26.

Quick quiz 2

Every weight starts at 0. After 4,000 epochs both hidden neurons still have exactly the same weights. Why? A: learning rate too small · B: same start, same slopes · C: too few epochs · D: sigmoid can't learn

Answer

B. Identical neurons compute the same output and get the same slope, so every update keeps them identical. Random initialisation breaks the symmetry.

Interview questions

  1. Why can a network with a hidden layer learn XOR when one neuron can't? Each hidden neuron draws its own line (e.g. OR and AND); the output neuron combines them, so the network can carve out regions no single line can.
  2. What happens if you remove all activation functions? The whole network collapses into a single linear function, however many layers it has.
  3. Why not initialise all weights to zero? Neurons in a layer stay identical (symmetry), so the layer behaves like one neuron.
  4. What is backpropagation? An algorithm that sends the error backwards through the network with the chain rule, giving the gradient (slope) for every weight in about the cost of a few forward passes.
  5. How do you count parameters in a dense layer? inputs × outputs weights + outputs biases.
  6. What can you do if training gets stuck in a local minimum? Restart from another random initialisation, add neurons, or change the learning rate; with large networks and random batches this is less of a problem in practice.

Summary

  • A hidden layer lets a network split XOR into easy questions (OR, AND) and combine them.
  • Networks have input, hidden and output layers; dense layers connect everything; the forward pass runs inputs to output; deep = many hidden layers.
  • Activations stop layers collapsing into one straight line.
  • Starting from all zeros fails (symmetry); random initialisation lets the network learn XOR; some starts land in a local minimum.
  • A tiny digit reader already has 101,770 parameters; nudging each one would take 283 hours.
  • Backpropagation passes the blame backwards and gives every weight its slope at once.
  • Next episode: deep learning for images and sequences: CNNs and RNNs.