SyntaxPilot
Courses/AI: Zero to Agents/Lesson 8

AI: Zero to Agents · Lesson 8 of 40 · 13 min

The artificial neuron

queued

These are the notes for AI episode 8. So far every model we built was a straight line that predicts a number. Here you build the tiny building block behind every neural network: the artificial neuron. Along the way: inputs, weights, bias, the weighted sum, activation functions (step, sigmoid, ReLU), the threshold, the perceptron, the decision boundary and XOR, the one problem a single neuron cannot solve. Every Python example below runs as-is in the SyntaxPilot compiler, and the outputs shown are real (Python 3.13).

How you decide: should I go out for chai?

Your decision In a neuron
Each thing you check (is it raining? are friends going?) an input
How much each one matters (rain: a lot, friends: a bit) a weight (can be negative: rain pushes towards "no")
Your mood before checking anything (lazy today) the bias: a starting push that doesn't depend on any input
Add it all up, go if the total is above 0 the weighted sum, then the activation

Where the name comes from

A brain neuron An artificial neuron
Dendrites receive signals from other cells Inputs: numbers coming in
Strong or weak connections Weights: multiply each input
Fires if the total signal is strong enough Activation: turns the total into an output

In 1943, Warren McCulloch and Walter Pitts described the first mathematical (artificial) neuron. It is a simplified copy, not a real brain cell, but the name stuck.

The problem: a straight line gives impossible chances

A teacher has records for ten students: hours studied, hours slept the night before, and whether they passed (1) or failed (0). This is classification (episode 4). The only model we can train so far is a straight line.

# (hours studied, hours slept) -> passed? 1 = yes, 0 = no
students = [(1, 5, 0), (2, 8, 0), (3, 4, 0), (4, 6, 0), (4, 9, 1),
            (5, 3, 0), (5, 7, 1), (6, 5, 1), (7, 8, 1), (8, 4, 1)]

# the best straight line through this data (episode 7's method)
def line(study, sleep):
    return 0.19 * study + 0.12 * sleep - 1.05

for study, sleep in [(5, 7), (1, 3), (10, 9)]:
    chance = line(study, sleep)
    print(f"study {study:2}h, sleep {sleep}h -> chance of passing {chance:.0%}")

Output:

study  5h, sleep 7h -> chance of passing 74%
study  1h, sleep 3h -> chance of passing -50%
study 10h, sleep 9h -> chance of passing 193%

The weights 0.19, 0.12 and −1.05 are the least-squares line through the ten students, rounded. Why it fails: a straight line never stops; it goes up and down forever. A chance must stay between 0 and 1. We need to squash the total into a yes/no.

The fix: a decision after the sum

inputs  →  weighted sum (w·x + bias)  →  activation (fire?)  →  output
  • Weighted sum: multiply each input by its weight, add them up, add the bias.
  • Step function: if the total is ≥ 0, output 1 ("fire"), else 0. The output jumps in one step, so it's called a step function.
  • Threshold: the point where it jumps (here, a total of 0).
  • Inputs + weights + bias + weighted sum + activation = an artificial neuron.

Demo 2: a neuron in Python (weights set by hand)

students = [(1, 5, 0), (2, 8, 0), (3, 4, 0), (4, 6, 0), (4, 9, 1),
            (5, 3, 0), (5, 7, 1), (6, 5, 1), (7, 8, 1), (8, 4, 1)]

def neuron(study, sleep, w_study, w_sleep, bias):
    total = w_study * study + w_sleep * sleep + bias   # weighted sum
    return 1 if total >= 0 else 0                      # step: fire or not

w_study, w_sleep, bias = 1.0, 0.5, -8.0               # set by hand
correct = 0
for study, sleep, passed in students:
    guess = neuron(study, sleep, w_study, w_sleep, bias)
    correct += guess == passed
    print(f"study {study}h sleep {sleep}h -> guess {guess}  real {passed}")
print("correct:", correct, "out of", len(students))

Output:

study 1h sleep 5h -> guess 0  real 0
study 2h sleep 8h -> guess 0  real 0
study 3h sleep 4h -> guess 0  real 0
study 4h sleep 6h -> guess 0  real 0
study 4h sleep 9h -> guess 1  real 1
study 5h sleep 3h -> guess 0  real 0
study 5h sleep 7h -> guess 1  real 1
study 6h sleep 5h -> guess 1  real 1
study 7h sleep 8h -> guess 1  real 1
study 8h sleep 4h -> guess 1  real 1
correct: 10 out of 10

The rule "study + half your sleep must reach 8" gets all ten right. But the weights were chosen by a human staring at the data. With thousands of inputs, the neuron must learn them.

Demo 3: gradient descent on a step function gets stuck

students = [(1, 5, 0), (2, 8, 0), (3, 4, 0), (4, 6, 0), (4, 9, 1),
            (5, 3, 0), (5, 7, 1), (6, 5, 1), (7, 8, 1), (8, 4, 1)]

def step(total):
    return 1 if total >= 0 else 0

def loss(w):                     # w = [w_study, w_sleep, bias]
    return sum((step(w[0] * a + w[1] * b + w[2]) - p) ** 2
               for a, b, p in students) / len(students)

w = [0.0, 0.0, -1.0]             # a bad start: always says "fail"
h = 0.001
for epoch in range(1, 4):
    slopes = []
    for i in range(3):
        nudged = w.copy(); nudged[i] += h
        slopes.append((loss(nudged) - loss(w)) / h)
    w = [w[i] - 0.1 * slopes[i] for i in range(3)]
    print(f"epoch {epoch}: slopes {slopes}  loss {loss(w)}")

Output:

epoch 1: slopes [0.0, 0.0, 0.0]  loss 0.5
epoch 2: slopes [0.0, 0.0, 0.0]  loss 0.5
epoch 3: slopes [0.0, 0.0, 0.0]  loss 0.5

Why: a tiny nudge to a weight almost never flips a 0 into a 1, so the loss doesn't change at all. The step function is flat everywhere, with one sudden cliff. On flat ground there is no slope to follow (episode 7's gradient descent needs one).

The sigmoid: a smooth switch

At home Activation function
A light switch: only off or on Step: 0 or 1, jumps suddenly
A fan regulator / dimmer: turns smoothly Sigmoid: a smooth S-curve from 0 to 1
sigmoid(total) = 1 / (1 + e^(-total))      e ≈ 2.718
total sigmoid(total)
−10 0.0000454
−2 0.119
0 0.5
2 0.881
10 0.99995

Any total, however big, is squashed into 0..1, which we read as a chance. And it is smooth everywhere, so a small nudge always changes the output a little: gradient descent can feel the slope.

Demo 4: the neuron learns its own weights

import math
students = [(1, 5, 0), (2, 8, 0), (3, 4, 0), (4, 6, 0), (4, 9, 1),
            (5, 3, 0), (5, 7, 1), (6, 5, 1), (7, 8, 1), (8, 4, 1)]
def sigmoid(total):              # squashes any number into 0..1
    return 1 / (1 + math.exp(-total))

def loss(w):                     # the neuron: sigmoid of the weighted sum
    return sum((sigmoid(w[0] * a + w[1] * b + w[2]) - p) ** 2 for a, b, p in students) / len(students)
w, h, lr = [0.0, 0.0, -1.0], 0.001, 0.5      # same bad start
for epoch in range(1, 3001):
    slopes = []
    for i in range(3):
        nudged = w.copy(); nudged[i] += h
        slopes.append((loss(nudged) - loss(w)) / h)
    w = [w[i] - lr * slopes[i] for i in range(3)]
    if epoch % 500 == 0: print(f"epoch {epoch}: loss {loss(w):.4f}")
print("learned weights:", [round(x, 2) for x in w])
for a, b in [(5, 7), (1, 3), (10, 9)]:
    print(f"study {a:2}h, sleep {b}h -> chance {sigmoid(w[0] * a + w[1] * b + w[2]):.0%}")

Output:

epoch 500: loss 0.1212
epoch 1000: loss 0.0394
epoch 1500: loss 0.0296
epoch 2000: loss 0.0244
epoch 2500: loss 0.0208
epoch 3000: loss 0.0182
learned weights: [1.7, 0.88, -13.0]
study  5h, sleep 7h -> chance 84%
study  1h, sleep 3h -> chance 0%
study 10h, sleep 9h -> chance 100%

The learned weights 1.7 (study) and 0.88 (sleep) keep the same ratio as the hand-made rule (sleep counts about half as much as study), but the neuron found them on its own. The chances now stay between 0% and 100%.

Three activation functions

Activation What it does Where it's used
Step total ≥ 0 → 1, else 0 the first neurons (perceptron); can't be trained by gradient descent
Sigmoid smooth squash into 0..1 the final yes/no chance of a model
ReLU (rectified linear unit) max(0, total): negatives become 0, positives pass through the default inside most deep networks; very cheap to compute

A bit of history: the perceptron

  • 1943: Warren McCulloch and Walter Pitts describe the first mathematical neuron (weights fixed by hand).
  • 1957: Frank Rosenblatt, at the Cornell Aeronautical Laboratory, creates the perceptron: a neuron with a step activation and a rule to learn its weights from examples.
  • 1960: the Mark I Perceptron, a machine built to run it, is shown in public on 23 June 1960.
  • 1969: Marvin Minsky and Seymour Papert's book Perceptrons shows that a single-layer perceptron cannot learn XOR.

Source: Perceptron, Wikipedia.

The next problem: the staircase light (XOR)

A staircase light has a switch at the bottom and one at the top. It is on only when the two switches are different:

Switch 1 Switch 2 Light
0 0 off (0)
0 1 on (1)
1 0 on (1)
1 1 off (0)

This pattern is XOR (exclusive or).

import math
# staircase light: two switches, light is on when they differ
switches = [(0, 0, 0), (0, 1, 1), (1, 0, 1), (1, 1, 0)]
def neuron(w, a, b):
    return 1 / (1 + math.exp(-(w[0] * a + w[1] * b + w[2])))

def loss(w):
    return sum((neuron(w, a, b) - on) ** 2 for a, b, on in switches) / 4

w, h, lr = [0.5, -0.3, 0.1], 0.001, 0.5
for epoch in range(1, 5001):
    slopes = []
    for i in range(3):
        nudged = w.copy(); nudged[i] += h
        slopes.append((loss(nudged) - loss(w)) / h)
    w = [w[i] - lr * slopes[i] for i in range(3)]
    if epoch % 1000 == 0: print(f"epoch {epoch}: loss {loss(w):.4f}")
for a, b, on in switches:
    print(f"switches {a} {b} -> light {neuron(w, a, b):.2f}  (should be {on})")

Output:

epoch 1000: loss 0.2500
epoch 2000: loss 0.2500
epoch 3000: loss 0.2500
epoch 4000: loss 0.2500
epoch 5000: loss 0.2500
switches 0 0 -> light 0.50  (should be 0)
switches 0 1 -> light 0.50  (should be 1)
switches 1 0 -> light 0.50  (should be 1)
switches 1 1 -> light 0.50  (should be 0)

Why: one neuron draws one straight line

  • The neuron says "yes" on one side of the line where its weighted sum is 0, and "no" on the other. That line is the decision boundary.
  • For the students, one straight line splits pass from fail: the data is linearly separable.
  • For XOR, put the four switch positions on the corners of a square. The ON positions (0 1 and 1 0) are on opposite corners, and so are the OFF ones. No single straight line can put both ONs on one side and both OFFs on the other.
  • One neuron can only draw one line, so it can never learn XOR. The fix (next episode): use a team of neurons in layers.

Glossary

Term Meaning Example
Neuron (biological) a brain cell that receives signals and fires when they are strong enough the cells in your brain
Artificial neuron a tiny model: weighted sum of inputs + bias, through an activation sigmoid(1.7*study + 0.88*sleep - 13)
Input a number fed into the neuron hours studied
Weight how much an input counts; can be negative 1.0 for study, 0.5 for sleep
Bias a starting push that doesn't depend on any input −8
Weighted sum each input × its weight, added up, + bias 1.0*6 + 0.5*4 - 8 = 0
Activation function turns the weighted sum into the neuron's output step, sigmoid, ReLU
Step function outputs 1 if the total ≥ 0, else 0 1 if total >= 0 else 0
Threshold the point where a step function jumps a total of 0
Sigmoid smooth S-curve that squashes any number into 0..1 sigmoid(0) = 0.5
e a fixed number in maths, about 2.718 used in math.exp
ReLU rectified linear unit: max(0, total) relu(-3) = 0, relu(5) = 5
Perceptron Rosenblatt's 1957 neuron with a step activation that learns its weights Mark I Perceptron, 1960
Decision boundary the line where the weighted sum is 0; one side yes, the other no study + 0.5*sleep = 8
Linearly separable data that one straight line can split into its classes pass/fail students, AND, OR
XOR (exclusive or) true only when the two inputs differ the staircase light
Classification predicting a category (episode 4) pass or fail
Epoch one pass of training over all the data (episode 7) for epoch in range(1, 3001)

Common mistakes

Mistake Code Error
Huge negative total in the sigmoid 1 / (1 + math.exp(1000)) OverflowError: math range error
Forgetting the import math.exp(-2) without import math NameError: name 'math' is not defined. Did you forget to import 'math'?
Training a step function with gradient descent return 1 if total >= 0 else 0 no error; slopes [0.0, 0.0, 0.0], loss stuck at 0.5
Reading a line's output as a chance 0.19*study + 0.12*sleep - 1.05 no error; chances like -50% and 193%

Fixes: keep totals small (scale your inputs, or clip the total, for example max(-500, min(500, total))); add import math; train with a smooth activation (sigmoid, ReLU); squash the output with a sigmoid before reading it as a chance.

Mini project: your own decision neuron

Build a neuron that decides "go out for chai?" from three inputs: raining (0/1), friends going (0/1), and money in your pocket (₹). Choose weights and a bias by hand so it says yes on a dry day with friends, then write 6 days of data and train the sigmoid version with the gradient descent loop from Demo 4. Print the learned weights. Is rain's weight negative?

Practice

  1. In Demo 2, change the bias. For which biases are all ten students still correct?
  2. Train the sigmoid neuron on AND and OR (and XOR, for comparison).
  3. Write relu(total) and print it for −3, 0 and 5.
  4. Draw XOR's four corners on paper and try to split them with one line.
Solutions

1.

students = [(1, 5, 0), (2, 8, 0), (3, 4, 0), (4, 6, 0), (4, 9, 1),
            (5, 3, 0), (5, 7, 1), (6, 5, 1), (7, 8, 1), (8, 4, 1)]

def neuron(study, sleep, w_study, w_sleep, bias):
    total = w_study * study + w_sleep * sleep + bias
    return 1 if total >= 0 else 0

for bias in [-6.0, -7.0, -8.0, -8.5, -9.0, -10.0]:
    correct = sum(neuron(a, b, 1.0, 0.5, bias) == p for a, b, p in students)
    print(f"bias {bias:5}: {correct} out of {len(students)} correct")
bias  -6.0: 7 out of 10 correct
bias  -7.0: 9 out of 10 correct
bias  -8.0: 10 out of 10 correct
bias  -8.5: 10 out of 10 correct
bias  -9.0: 7 out of 10 correct
bias -10.0: 7 out of 10 correct

The bias moves the threshold: too high (−6) and failing students pass; too low (−9) and passing students fail.

2.

import math

def neuron(w, a, b):
    return 1 / (1 + math.exp(-(w[0] * a + w[1] * b + w[2])))

def train(data, epochs=5000, lr=0.5, h=0.001):
    def loss(w):
        return sum((neuron(w, a, b) - y) ** 2 for a, b, y in data) / len(data)
    w = [0.5, -0.3, 0.1]
    for _ in range(epochs):
        slopes = []
        for i in range(3):
            nudged = w.copy(); nudged[i] += h
            slopes.append((loss(nudged) - loss(w)) / h)
        w = [w[i] - lr * slopes[i] for i in range(3)]
    return w, loss(w)

gates = {"AND": [(0, 0, 0), (0, 1, 0), (1, 0, 0), (1, 1, 1)],
         "OR":  [(0, 0, 0), (0, 1, 1), (1, 0, 1), (1, 1, 1)],
         "XOR": [(0, 0, 0), (0, 1, 1), (1, 0, 1), (1, 1, 0)]}
for name, data in gates.items():
    w, final = train(data)
    outs = [round(neuron(w, a, b), 2) for a, b, _ in data]
    print(f"{name}: loss {final:.4f}  outputs {outs}")
AND: loss 0.0021  outputs [0.0, 0.05, 0.05, 0.94]
OR: loss 0.0011  outputs [0.05, 0.97, 0.97, 1.0]
XOR: loss 0.2500  outputs [0.5, 0.5, 0.5, 0.5]

AND and OR are linearly separable, so one neuron learns them. XOR is not.

3.

def relu(total):
    return max(0, total)

for t in [-3, 0, 5]:
    print(f"relu({t}) = {relu(t)}")
relu(-3) = 0
relu(0) = 0
relu(5) = 5

4. Whatever line you draw, one side always holds an ON corner and an OFF corner together. You need two lines (two neurons) and a third neuron to combine them: that is episode 9.

Quick quiz 1

Inputs: study = 6, sleep = 4. Weights 1.0 and 0.5, bias −8, step activation. What is the output? A: 0 · B: 1 · C: 0.5 · D: −8

Answer

B: 1. Weighted sum = 1.0 × 6 + 0.5 × 4 − 8 = 0, and 0 ≥ 0, so the step fires. It sits exactly on the threshold.

Quick quiz 2

Which of these can a single neuron NOT learn? A: OR · B: AND · C: XOR · D: "light on if switch 1 is up"

Answer

C: XOR. OR, AND and "switch 1 only" can each be split by one straight line. XOR's ON corners are opposite each other, so it needs two lines.

Interview questions

  1. What does an artificial neuron compute? A weighted sum of its inputs plus a bias, passed through an activation function.
  2. What is the bias for? It shifts the threshold, so the neuron can fire (or not) even when all inputs are 0.
  3. Why do we need activation functions? To turn the unbounded weighted sum into a useful output (a chance, a yes/no), and, in networks, to add the bends that let layers learn more than a straight line (episode 9).
  4. Why can't you train a step-function neuron with gradient descent? Its slope is 0 almost everywhere, so gradient descent gets no signal.
  5. Sigmoid vs ReLU? Sigmoid squashes into 0..1 (good for a final probability); ReLU is max(0, x), cheap, and the usual choice in hidden layers.
  6. What is linear separability, and why does XOR matter? Data one straight line can split. XOR is not linearly separable, so a single perceptron can't learn it (Minsky and Papert, 1969).

Summary

  • A neuron multiplies inputs by weights, adds a bias (the weighted sum) and passes it through an activation.
  • A plain line gave impossible chances (−50%, 193%); the step function gives yes/no but is flat, so gradient descent gets slopes of 0.
  • The smooth sigmoid lets gradient descent train the neuron: it learned weights 1.7 and 0.88 on its own.
  • ReLU (max(0, total)) is the default activation inside deep networks.
  • One neuron draws one straight line (its decision boundary), so it can't learn XOR.
  • Next episode: neural networks: layers of neurons, hidden neurons, and backpropagation.