Why cross entropy, of all things: from BCE to KL divergence

Reaching for CrossEntropyLoss when you write a classifier is close to a reflex. But explaining why it is that particular loss, why MSE (mean squared error) — which served us well in regression — will not do, what makes it different from BCELoss, and how it relates to the KL divergence that shows up in every paper turns out to be surprisingly hard to do in one breath.

The goal of this post is to connect that chain from end to end, in one pass. Starting from information content, then entropy, cross entropy, BCE, maximum likelihood estimation, and KL divergence.

The opening question: why not MSE?

Suppose in binary classification the model emits a logit $z$, passes it through a sigmoid to produce $\hat y = \sigma(z)$, and the label is $y \in {0, 1}$. With MSE the loss is

Taking the gradient with respect to $z$ exposes the problem.

There is a factor $\sigma’(z) = \hat y(1-\hat y)$ attached. It dies to zero when $\hat y$ approaches 0 or 1. But “$\hat y$ near 0 while the label is 1” is exactly the situation where the model is most badly wrong. So the MSE + sigmoid combination has its smallest gradient precisely when it is most wrong. Learning stops at the very point where learning is needed.

The numbers make the severity obvious. Take $y=1$ while the model emits $z=-8$ — completely wrong.

Same situation, gradient magnitudes about 3000x apart. MSE learns essentially nothing.

Even setting the gradient issue aside, a deeper reason remains. A classifier’s output is not a number but a probability distribution. There is no reason to measure how different two probability distributions are with Euclidean distance. Probability distributions have their own yardstick, and that yardstick comes from information theory.

Information content: the size of the surprise

“The sun rises tomorrow” carries almost zero information. “It snows tomorrow” carries a lot, if it is August. To turn the intuition that rarer events carry more information into a formula, what shape must the function have? Impose three conditions.

1. Lower probability means more information (monotonically decreasing)
2. An event with probability 1 carries zero information
3. Observing two independent events together carries the sum of their information: $I(pq) = I(p) + I(q)$

Condition 3 is the decisive one. The only continuous function that turns products into sums is the logarithm. Satisfying 1 and 2 as well requires flipping the sign.

With base 2 the unit is a bit; with the natural logarithm it is a nat. Deep learning implementations almost always use the natural logarithm.

Entropy: average surprise

Information content is a value for a single event. Averaging it over the whole distribution gives entropy.

By Shannon’s source coding theorem, this value is the minimum average number of bits needed to encode data drawn from $P$. A fair coin has $H = 1$ bit; a coin that always lands heads has $H = 0$ bits. The latter is obvious — there is no need to transmit the outcome at all.

Entropy is an intrinsic property of a single distribution. No model has entered the picture yet.

Cross entropy: the cost of using the wrong codebook

Now bring in two distributions. The real data follows $P$, but not knowing that, we believed $Q$ was correct and built our codebook from it. The average number of bits this actually costs is the cross entropy.

The expectation is taken over the true distribution $P$, while our prediction $Q$ sits inside the logarithm. That asymmetry is the whole point.

The value is minimised when $Q = P$, and that minimum is exactly $H(P)$. So always

We pay an extra cost in proportion to how far our belief diverges from reality. That excess is the KL divergence, coming up later.

Cross entropy in deep learning

Let $P$ be the label distribution and $Q$ the model’s predicted distribution, and it becomes a loss function. With a one-hot label, $P$ is 1 only at the true class $c$ and 0 elsewhere, so only one term of the sum survives.

The negative log of the probability assigned to the correct class. That is all of the cross entropy you meet in practice. Assign 1 to the correct class and the loss is 0; drive the correct-class probability toward 0 and the loss diverges to infinity.

import torch
import torch.nn.functional as F

logits = torch.tensor([[2.0, 1.0, 0.1]])
target = torch.tensor([0])

# softmax([2.0, 1.0, 0.1]) = [0.6590, 0.2424, 0.0986]
# CE = -log(0.6590) = 0.4170
F.cross_entropy(logits, target)

BCE: when there are two classes

Binary cross entropy is not a separate concept but the special case of two classes. Substituting $P = (y,\ 1-y)$ and $Q = (\hat y,\ 1-\hat y)$ straight into the definition above gives it.

So why give it its own name? Two reasons.

The first is economy of representation. With two classes, knowing $\hat y$ alone determines the rest as $1 - \hat y$. Two output nodes plus softmax means duplicated parameters. One node plus a sigmoid is enough.

The second matters more in practice: it can handle multi-label problems. Softmax forces the outputs to sum to 1, which assumes the classes are mutually exclusive. Sigmoid + BCE, by contrast, treats each class as an independent binary problem with no constraint on the sum. If “cat” and “indoors” can both be true of one photo, you need BCE.

Why this is the “right” loss (maximum likelihood estimation)

Everything so far was the information-theoretic reading. Approach it from statistics and the same formula falls out along an entirely different route.

We want to maximise the likelihood that a model with parameters $\theta$ produced the data ${(x_i, y_i)}$. If the samples are independent the likelihood is a product, and taking the logarithm turns it into a sum.

The right-hand side is exactly a sum of cross entropies.

Minimising cross entropy = maximum likelihood estimation

This is not a loss picked because it looked plausible among candidates. It is the formula that necessarily appears when you apply the standard statistical principle of “find the parameters that best explain the data” to classification.

KL divergence: what we are actually reducing

If you want to measure the difference between two distributions in its own right, cross entropy is slightly awkward — even in the perfect case $Q = P$ its value is not 0 but $H(P)$. Let us subtract that floor.

Splitting the fraction inside the logarithm reveals what it is.

Rearranged, we get the one line that runs through this whole post.

Cross entropy = the entropy inherent in the data + the excess cost of my prediction being wrong.

And a practically important conclusion follows. During training the label distribution $P$ is fixed, so $H(P)$ is a constant with respect to $\theta$. Subtracting a constant does not change the argmin.

Minimising cross entropy and minimising KL divergence are exactly the same optimisation problem. That is why we use the cheaper cross entropy rather than bothering to compute $H(P)$.

With one-hot labels we can go one step further. There is only one event with probability 1, so $H(P) = 0$, and therefore

the two values coincide. In the earlier example CE was 0.4170, and computing $D_{KL}$ directly gives $1 \cdot \log(1/0.6590) = 0.4170$ — the same value.

Two properties of KL

It is always non-negative. $D_{KL}(P\parallel Q) \ge 0$, with equality only when $P = Q$ (Gibbs’ inequality). This is another way of stating what we said earlier, $H(P,Q) \ge H(P)$.

It is not symmetric. $D_{KL}(P \parallel Q) \ne D_{KL}(Q \parallel P)$, and the triangle inequality does not hold either. Strictly speaking, then, “KL distance” is wrong. Hence divergence.

That asymmetry is not an annoying flaw but a property that is actually used.

• Forward KL $D_{KL}(P \parallel Q)$: where $P$ is large and $Q$ is small, $\log(P/Q)$ explodes. $Q$ tries to cover the whole region where $P$ has mass (mass-covering). Supervised learning falls here.
• Reverse KL $D_{KL}(Q \parallel P)$: the expectation is taken over $Q$, so regions where $Q$ is 0 are not penalised at all. $Q$ safely collapses onto one mode of $P$ (mode-seeking). Variational inference and the KL penalty in RLHF fall here.

When KL is used directly

There are situations where KL is computed explicitly even though cross entropy would do — namely when $P$ is not one-hot.

• Knowledge distillation: the teacher’s softmax distribution is used as $P$. $H(P) \ne 0$, but it is still constant with respect to the student’s parameters, so CE would in fact give the same optimum. KL is used by convention to make the intent — “match the two distributions” — explicit.
• VAE: the regularisation term that pins the latent distribution to the prior is a KL. Here it is part of the loss rather than a constant, and the value itself is genuinely needed.
• PPO and other RL: KL constrains a new policy from drifting too far from the old one.

Back to gradients: why cross entropy keeps learning alive

Return to the problem from the first section. Taking BCE’s gradient with respect to $z$:

The $\hat y(1-\hat y)$ in the denominator cancels the sigmoid’s derivative exactly. The saturating factor that killed MSE disappears, and the gradient becomes the error itself. The more wrong you are, the more you move.

The same thing happens with softmax + CE.

This is no accident. Sigmoid and softmax are built from exp, and cross entropy takes a log. The log and the exp erase each other. A loss derived from information theory and an activation that normalises into probabilities were designed to be a matched pair.

Landmines people step on when implementing this

Pass logits, not probabilities. F.cross_entropy and nn.BCEWithLogitsLoss secure numerical stability internally with the log-sum-exp trick. Applying softmax yourself and then taking a log gives -inf the moment a probability underflows to 0, and NaN follows.

nn.CrossEntropyLoss already includes the softmax. Putting a softmax at the end of the model and then using this loss applies softmax twice. No error, it just quietly fails to learn — the most irritating kind of bug to find. The relationship between nn.BCEWithLogitsLoss and nn.BCELoss is the same.

# bad: softmax twice
logits = model(x)
loss = nn.CrossEntropyLoss()(F.softmax(logits, dim=-1), target)

# good: raw logits
logits = model(x)
loss = nn.CrossEntropyLoss()(logits, target)

F.kl_div’s argument convention is not what you would guess. The first argument is not probabilities but log probabilities, and the second is probabilities (with log_target=False). Worse, the default reduction='mean' averages element-wise, which does not match the definition in the formula. To match it at the distribution level you need reduction='batchmean'.

Label smoothing is a technique that modifies $P$. It slightly blurs the one-hot into $P = (1-\epsilon, \epsilon/(K-1), \ldots)$. Now $H(P) \ne 0$, so CE and KL no longer have the same value. They still differ by a constant, though, so as optimisation targets they are identical. You only need to be careful when comparing absolute loss values against other experiments.

Summary

• Information content $-\log p$ is the form that necessarily emerges from the “products into sums” condition.
• Entropy $H(P)$ is the intrinsic minimum encoding cost of a single distribution.
• Cross entropy $H(P,Q)$ is the cost of encoding with $Q$ when reality is $P$, and it is always at least $H(P)$.
• BCE is CE with two classes. The substantive difference is that, paired with a sigmoid, it can handle multi-label problems.
• Minimising CE is maximum likelihood estimation. It is not an arbitrarily chosen loss.
• $H(P,Q) = H(P) + D_{KL}(P\parallel Q)$, and during training $H(P)$ is constant. So minimising CE and minimising KL are the same problem, and with one-hot labels even the values coincide.
• KL is asymmetric and therefore not a distance. That asymmetry is what separates mass-covering from mode-seeking.
• The log and exp cancelling so the gradient collapses to $\hat y - y$ is the practical reason to use CE instead of MSE.