0x60Lesson 7 of 17

Build a tiny autograd engine

Write a Value class that records every operation and backpropagates automatically - the core idea behind PyTorch.

32 min 7-question quiz 2 code exercises

By the end of this lesson you can

Represent a computation as a graph of Value objects that remember their inputs
Give each operation a local backward rule, and run them in reverse topological order
Use your engine to train a neuron with gradient descent

Working out gradients by hand gets old fast. Frameworks like PyTorch do it automatically: every operation you perform is recorded in a graph, and calling .backward() runs the chain rule over it. That’s automatic differentiation (autograd), and you can build a working version in about 50 lines. (This lesson follows the spirit of Andrej Karpathy’s micrograd.)

The plan:

A Value wraps a number, remembers which Values it was computed from (_children), and has a grad that starts at 0.
Every operation (+, *, tanh, ...) creates a new Value and attaches a tiny _backward function that knows the local rule - e.g. for out = a * b: a.grad += b.data * out.grad.
backward() sorts the graph so every node comes after its inputs (a topological sort), sets the output’s grad to 1, and calls each _backward in reverse.

Gradients use +=, not =, because a value used in several places must sum the gradients from each use.

value.py

1import math
2
3class Value:
4    """A number that remembers how it was computed, so it can backpropagate."""
5
6    def __init__(self, data, children=(), op=""):
7        self.data = data
8        self.grad = 0.0
9        self._backward = lambda: None
10        self._children = children
11        self.op = op
12
13    def __add__(self, other):
14        other = other if isinstance(other, Value) else Value(other)
15        out = Value(self.data + other.data, (self, other), "+")
16        def backward():
17            self.grad += out.grad
18            other.grad += out.grad
19        out._backward = backward
20        return out
21
22    def __mul__(self, other):
23        other = other if isinstance(other, Value) else Value(other)
24        out = Value(self.data * other.data, (self, other), "*")
25        def backward():
26            self.grad += other.data * out.grad
27            other.grad += self.data * out.grad
28        out._backward = backward
29        return out
30
31    def tanh(self):
32        t = math.tanh(self.data)
33        out = Value(t, (self,), "tanh")
34        def backward():
35            self.grad += (1 - t * t) * out.grad
36        out._backward = backward
37        return out
38
39    def __neg__(self):
40        return self * -1
41
42    def __sub__(self, other):
43        return self + (-other)
44
45    __radd__ = __add__
46    __rmul__ = __mul__
47
48    def backward(self):
49        order, seen = [], set()
50        def visit(node):
51            if node not in seen:
52                seen.add(node)
53                for child in node._children:
54                    visit(child)
55                order.append(node)
56        visit(self)
57        self.grad = 1.0
58        for node in reversed(order):
59            node._backward()
60
61# L = tanh(w*x + b) - y, then squared via multiplication
62w, x, b, y = Value(0.5), Value(2.0), Value(-0.3), Value(1.0)
63p = (w * x + b).tanh()
64d = p - y
65L = d * d
66L.backward()
67print(f"p={p.data:.4f} L={L.data:.4f}")
68print(f"dL/dw={w.grad:.4f} dL/db={b.grad:.4f} dL/dx={x.grad:.4f}")

Output

p=0.6044 L=0.1565
dL/dw=-1.0045 dL/db=-0.5022 dL/dx=-0.2511

Read backward() slowly - it’s the heart of every deep learning framework:

visit walks from the output down to the inputs and appends each node after its children, giving an order where inputs come first.
Reversed, that’s output first: each node’s gradient is complete (all its users have already added their contributions) by the time its own _backward passes gradient further down.

Training with your engine

With gradients on demand, training is a loop: forward pass, backward(), nudge every parameter against its gradient, and reset the grads to zero for the next step (they accumulate otherwise - PyTorch makes you call zero_grad() for exactly this reason).

train_neuron.py

1import math
2
3class Value:
4    """A number that remembers how it was computed, so it can backpropagate."""
5
6    def __init__(self, data, children=(), op=""):
7        self.data = data
8        self.grad = 0.0
9        self._backward = lambda: None
10        self._children = children
11        self.op = op
12
13    def __add__(self, other):
14        other = other if isinstance(other, Value) else Value(other)
15        out = Value(self.data + other.data, (self, other), "+")
16        def backward():
17            self.grad += out.grad
18            other.grad += out.grad
19        out._backward = backward
20        return out
21
22    def __mul__(self, other):
23        other = other if isinstance(other, Value) else Value(other)
24        out = Value(self.data * other.data, (self, other), "*")
25        def backward():
26            self.grad += other.data * out.grad
27            other.grad += self.data * out.grad
28        out._backward = backward
29        return out
30
31    def tanh(self):
32        t = math.tanh(self.data)
33        out = Value(t, (self,), "tanh")
34        def backward():
35            self.grad += (1 - t * t) * out.grad
36        out._backward = backward
37        return out
38
39    def __neg__(self):
40        return self * -1
41
42    def __sub__(self, other):
43        return self + (-other)
44
45    __radd__ = __add__
46    __rmul__ = __mul__
47
48    def backward(self):
49        order, seen = [], set()
50        def visit(node):
51            if node not in seen:
52                seen.add(node)
53                for child in node._children:
54                    visit(child)
55                order.append(node)
56        visit(self)
57        self.grad = 1.0
58        for node in reversed(order):
59            node._backward()
60
61# Learn y = tanh(2x - 1) from four examples
62data = [(-1.0, math.tanh(-3)), (0.0, math.tanh(-1)), (0.5, math.tanh(0)), (1.0, math.tanh(1))]
63w, b = Value(0.1), Value(0.0)
64for step in range(301):
65    loss = Value(0.0)
66    for x, y in data:
67        d = (w * x + b).tanh() - y
68        loss = loss + d * d
69    w.grad = b.grad = 0.0
70    loss.backward()
71    w.data -= 0.2 * w.grad
72    b.data -= 0.2 * b.grad
73    if step % 100 == 0:
74        print(f"step {step}: loss {loss.data:.4f}")
75print(f"w = {w.data:.2f}, b = {b.data:.2f}")

Output

step 0: loss 1.8224
step 100: loss 0.0001
step 200: loss 0.0000
step 300: loss 0.0000
w = 2.00, b = -1.00

Key takeaways

Autograd records each operation and its inputs, building a graph as you compute.
Every operation carries a local backward rule; gradients accumulate with +=.
backward() runs those rules in reverse topological order, starting from grad = 1.
Training: forward, zero the grads, backward, step - the same loop PyTorch uses.

Lesson quiz

7 questions · pass with 5 correct · up to 50 XP

Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.

Practice: write Python

Write Python in the editor and run it against sample inputs. Python runs locally in your browser using a WebAssembly runtime.

Exercise 1

Teach the engine new tricks

+25 XP

Add two operations to Value:

__pow__(self, exponent) for a constant number exponent: the result is data ** exponent, and the local derivative is exponent * data ** (exponent - 1).
relu(self): max(0, data), with local derivative 1 if data > 0 else 0.

The test code at the bottom computes two expressions and prints values and gradients.

Two expressions

main.py

Loading editor…

Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.

Exercise 2

Fit a line with your engine

+25 XP

Each input line is an x y point. Using the Value class, fit $\hat{y} = w x + b$ by minimizing the mean squared error with gradient descent: start from w = b = 0, learning rate 0.05, 500 steps (remember to reset the gradients each step). Print w= and b= to 2 decimals.

y = 3x + 2
Noisy points

main.py

Loading editor…

Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.

Questions about this lesson

Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.

Loading posts…