The perceptron

August 4, 2026

A neuron takes an input vector $\mathbf{p}$, weights it, adds a bias, and passes the result through a transfer function:

That is the whole computation. The tool below runs it, trains it, and draws what it decides.

Start on AND with no hidden neurons. Press train.

A worked example

Take two inputs, weights $\mathbf{W} = \begin{bmatrix} 1 & 1 \end{bmatrix}$, bias $b = -1.5$, and the hard limit transfer function

Run the four corners of the unit square through it:

That is AND, computed by one neuron with three numbers.

The line

The output changes where $n$ changes sign, so the boundary is

which for the numbers above is $p_1 + p_2 = 1.5$. A straight line, with $[1,1]$ on one side and the other three corners on the other.

Two facts about that line are worth holding onto, and both are visible in the tool:

$\mathbf{W}$ is perpendicular to it. The weight vector is drawn in the input space, and it points at the region that outputs 1. Changing $\mathbf{W}$ rotates the boundary.

$b$ shifts it without turning it. The distance from the origin is $-b / \lVert \mathbf{W} \rVert$. A neuron without a bias can only draw lines through the origin, which is most of what the bias is for.

The transfer function

The sum is linear. The transfer function is the only place a neuron can be anything else, and the choice decides what the neuron is for.

• hardlim returns 0 or 1. A decision, with no indication of confidence.
• logsig returns a value in $(0, 1)$, and $f'(n) = a(1 - a)$.
• tansig returns a value in $(-1, 1)$, and $f'(n) = 1 - a^2$.
• poslin passes positives through and clamps negatives to zero.
• purelin returns $n$ unchanged, which makes the whole network linear.

Click any neuron in the tool to cycle it. The layer keeps one function, written $f^m$ for layer $m$, which is the convention Hagan uses and the reason superscripts appear everywhere in this subject.

Training one neuron

The perceptron rule needs no calculus. Present a sample, compare, correct:

Take $\mathbf{W} = \begin{bmatrix} -1 & -1 \end{bmatrix}$, $b = 0$, and the sample $\mathbf{p} = [1, 1]$ with target $t = 1$:

Now $n = 1$ and $a = 1$. Fixed in one step.

The geometry explains why. Adding $e\mathbf{p}^T$ rotates $\mathbf{W}$ toward $\mathbf{p}$ when the target is 1 and away when it is 0, which turns the boundary until the point falls on the correct side. When every point is already correct, every $e$ is zero and the weights stop moving. Convergence is a fixed point, and it is reached in finite time whenever a separating line exists.

What one line cannot do

Switch the problem to XOR and train.

The boundary keeps moving and never settles, because no straight line puts $[0,1]$ and $[1,0]$ on one side with $[0,0]$ and $[1,1]$ on the other. This is not slow convergence. There is no assignment of two weights and a bias that classifies those four points, so the rule cannot find one.

Every problem a single neuron can solve is one a line can separate, and most problems are not like that.

Two layers, and the derivative that costs

Add hidden neurons. Each draws its own line, and the output neuron combines them, so the regions it can carve are intersections and unions of half-planes rather than a single half-plane. Watch the shading in the tool bend.

Training them is the harder part. The error depends on a hidden weight only through everything downstream of it, so the derivative has to be carried backwards. Define the sensitivity of layer $m$ as $\mathbf{s}^m = \partial E / \partial \mathbf{n}^m$. Then

and the gradient of any weight is $\mathbf{s}^m (\mathbf{a}^{m-1})^T$. One backward sweep reaches every weight in the network, which is the only reason training a network of any size is affordable.

Notice what the recursion requires: $\dot{\mathbf{F}}$, the derivative of the transfer function. Try setting a hidden layer to hardlim. The tool refuses to train, because $\mathbf{s}$ is zero everywhere and no weight ever moves. The perceptron rule exists precisely because that derivative does not.

Where it stops being tidy

Set two hidden neurons on XOR, press new weights, and train. Repeat.

Some starts solve it. Others settle at two of four correct and stay there. Two neurons are enough to represent XOR and not enough to make finding it reliable, because gradient descent goes downhill from where it happens to begin and stops at the first flat place it meets. Three neurons find it from nearly any start.

Nothing in the mathematics promises otherwise. Backpropagation computes a gradient exactly; a gradient is a local statement, and the surface it descends was never claimed to have one minimum.

The code

The rule from “Training one neuron”, for the one neuron it actually trains:

struct Perceptron {
    std::vector<double> w;
    double b = 0.0;
};

// n = w . p + b
double netInput(const Perceptron& neuron, const std::vector<double>& p) {
    double n = neuron.b;
    for (std::size_t j = 0; j < neuron.w.size(); j++) n += neuron.w[j] * p[j];
    return n;
}

// 0 below zero, 1 at and above, same as transfer.js's hardlim. No usable
// derivative, which is why the update below isn't gradient descent.
double hardlim(double n) { return n < 0.0 ? 0.0 : 1.0; }

double predict(const Perceptron& neuron, const std::vector<double>& p) {
    return hardlim(netInput(neuron, p));
}

struct Sample {
    std::vector<double> p;
    double t;
};

// e = t - a, then W += e*p, b += e. Skipped when e is already zero, which
// is also the convergence check: once every sample agrees, nothing here
// moves the weights again.
void perceptronUpdate(Perceptron& neuron, const Sample& sample) {
    const double a = predict(neuron, sample.p);
    const double e = sample.t - a;
    if (e == 0.0) return;
    for (std::size_t j = 0; j < neuron.w.size(); j++) neuron.w[j] += e * sample.p[j];
    neuron.b += e;
}

// One pass over the samples, updating after each.
void trainEpoch(Perceptron& neuron, const std::vector<Sample>& samples) {
    for (const auto& sample : samples) perceptronUpdate(neuron, sample);
}