Q-learning

August 3, 2026

Q-learning is often introduced through its update equation. The equation is important, but it only changes one number at a time. Those numbers live in a Q table, and watching that table evolve makes the algorithm much easier to understand.

In this example, a robot must walk to a box, pick it up, carry it to the bin, and drop it. Every entry in the table starts at zero. Press run and watch the values change.

Rows are states. Columns are the six actions the robot can take. Each cell stores the estimated value of taking one action from one state.

What one number means

$Q(s, a)$ is the expected discounted return from taking action $a$ in state $s$, then following the current policy afterwards.

The value includes future rewards, not just the immediate reward. A state far from the goal can still have a large value because it eventually leads to a successful delivery.

Click any row in the table to inspect its six action values.

Where the rows come from

A state should contain all of the information needed to choose the next action.

Here the state consists of:

• The robot’s position.
• Whether the robot is carrying the box.

There are twenty-one valid grid cells and two carrying states, giving forty-two rows.

The left half of the table represents the robot before picking up the box. The right half represents the robot while carrying it. Although the robot may occupy the same grid cell, those situations require different actions, so they must have different rows.

Without the carrying flag, both situations would share the same state and the agent could not learn a consistent policy.

The update

Each step updates exactly one table entry.

The update has three parts:

• The target is the immediate reward plus the discounted value of the best action available in the next state.
• The error is the difference between the target and the current estimate.
• Alpha ($\alpha$) controls how much of that error is written back into the table.

The status line beneath the visualization displays each of these quantities for every update.

The term

is the bootstrap estimate of future return. Early in training these estimates are poor, but each reward updates one state, and those updates gradually propagate through the table.

Watch the value propagate

Reset the simulation and step through the first successful delivery.

Initially, nearly every target is built from zeros, so most entries remain close to zero.

When the robot drops the box into the bin, the DROP action at that state receives the positive terminal reward.

On a later visit to the neighboring state, the target becomes

so that neighboring state acquires a positive value. States farther away are updated on later visits, producing a sequence of discounted values extending backward from the goal.

Over time this produces a value gradient. The greedy policy simply selects the largest value in each row, so the arrows emerge naturally from the learned values.

Alpha and gamma

Alpha ($\alpha$) is the learning rate. Large values cause each observation to have a stronger influence on the table. Smaller values average information over more experiences.

Gamma ($\gamma$) is the discount factor. It determines how strongly future rewards influence the current state. Larger values allow rewards to propagate farther through the state space, while smaller values emphasize immediate outcomes.

Exploration

The agent chooses a random action with probability $\epsilon$ and the current best action otherwise.

If $\epsilon = 0$ from the beginning, the agent never explores states whose values are still unknown. If $\epsilon = 1$ forever, it continues acting randomly after learning useful values.

Most Q-learning implementations begin with a high exploration rate and reduce it over time so that the agent first gathers experience, then exploits what it has learned.

When the table becomes too large

This example uses a fixed box and a fixed goal, keeping the state space small.

If the box and goal could each occupy any valid cell, the state would become

(robot position, box position, goal position, carrying flag),

giving

states, or

Q-values.

Adding more objects increases the table combinatorially.

This is the limitation of tabular Q-learning. As the state space grows, storing one value for every state-action pair becomes impractical. Methods such as Deep Q-Networks replace the table with a function approximator that estimates $Q(s,a)$ directly.

The code

The same agent in C++. A state is an int, an action is an int, and the table is the array they index:

// The table is flat: q[state * actionCount + action]. Flat because a 42 x 6
// table is a rectangle of numbers, and pretending otherwise costs an
// allocation per row.
struct QTable {
    int actionCount;
    std::vector<double> q;

    QTable(int stateCount, int actions)
        : actionCount(actions),
          q(static_cast<std::size_t>(stateCount) * actions, 0.0) {}

    double& at(int state, int action) {
        return q[static_cast<std::size_t>(state) * actionCount + action];
    }
    double at(int state, int action) const {
        return q[static_cast<std::size_t>(state) * actionCount + action];
    }
};

double maxQ(const QTable& table, int state) {
    double best = -std::numeric_limits<double>::infinity();
    for (int a = 0; a < table.actionCount; a++) best = std::max(best, table.at(state, a));
    return best;
}

// The greedy action, with ties broken by the rng rather than by always
// taking the lowest index. On a table of zeros every action ties, and a
// deterministic tie-break would send the agent marching in one direction
// for the whole of its first episode.
int greedyAction(const QTable& table, int state, std::mt19937& rng) {
    const double best = maxQ(table, state);
    std::vector<int> tied;
    for (int a = 0; a < table.actionCount; a++) {
        if (table.at(state, a) == best) tied.push_back(a);
    }
    if (tied.size() == 1) return tied[0];
    return tied[std::uniform_int_distribution<std::size_t>(0, tied.size() - 1)(rng)];
}

struct Choice { int action; bool explored; };

// Epsilon-greedy: explore with probability epsilon, otherwise take the best
// action known so far.
Choice chooseAction(const QTable& table, int state, double epsilon, std::mt19937& rng) {
    if (std::uniform_real_distribution<double>(0.0, 1.0)(rng) < epsilon) {
        return {std::uniform_int_distribution<int>(0, table.actionCount - 1)(rng), true};
    }
    return {greedyAction(table, state, rng), false};
}

// Named fields, because state and nextState are both ints and both plausible
// in either position.
struct Transition {
    int state, action;
    double reward;
    int nextState;
    double alpha, gamma;
    bool done;
};

struct Step { double before, after, target, error, bestNext; };

// One update, in place. The whole algorithm is these four lines.
//
// `done` matters: a terminal transition has no future to bootstrap from, so
// its target is the reward alone. Drop the check and the agent still finds a
// route on a task this small, but the numbers stop meaning anything.
// Delivering pays exactly 1 and ends the episode; without the check its cell
// bootstraps off the state it left behind and settles near 1.6 instead.
Step update(QTable& table, const Transition& t) {
    double& cell = table.at(t.state, t.action);
    const double before = cell;
    const double bestNext = t.done ? 0.0 : maxQ(table, t.nextState);
    const double target = t.reward + t.gamma * bestNext;
    const double error = target - before;

    cell = before + t.alpha * error;

    return {before, cell, target, error, bestNext};
}

// Floored, so the agent never stops exploring entirely.
double decayEpsilon(double epsilon, double rate, double floor = 0.02) {
    return std::max(floor, epsilon * rate);
}

// The agent never sees the world, only a state number, a reward, and a flag
// saying the episode ended. Env supplies `int reset()` and this `step`; here
// it is the pick-and-place grid above, whose twenty-one free cells times a
// carrying flag give forty-two states and six actions.
struct Outcome { int nextState; double reward; bool done; };

template <typename Env>
QTable train(Env& env, int stateCount, int actionCount, int episodes, std::mt19937& rng) {
    QTable table(stateCount, actionCount);
    double epsilon = 1.0;

    for (int e = 0; e < episodes; e++) {
        int state = env.reset();

        // Capped. An agent that has learnt nothing yet will wander for as
        // long as it is allowed to without ever reaching the bin.
        for (int t = 0; t < 200; t++) {
            const Choice choice = chooseAction(table, state, epsilon, rng);
            const Outcome out = env.step(choice.action);

            update(table, {state, choice.action, out.reward, out.nextState,
                           0.5, 0.95, out.done});

            state = out.nextState;
            if (out.done) break;
        }

        // Once per episode, not once per step.
        epsilon = decayEpsilon(epsilon, 0.99, 0.02);
    }
    return table;
}