Backpropagation through convolution: deriving kernel and input gradients
Derive the kernel and input gradients of a convolutional layer using the chain rule, and connect cross-correlation, full convolution, kernel flipping, and stamp accumulation.
Backpropagation through a convolutional layer computes two gradients. The kernel gradient is used to update the weights, and the input gradient is passed to the preceding layer.
Both start from the same forward equation, but they take different forms. The kernel gradient is a cross-correlation of the input with the output gradient. The input gradient is a full convolution of the output gradient with the kernel. We will derive both using the chain rule and see why the kernel sometimes needs to be flipped.
Forward pass and notation
Let the input $x$ have shape $H\times W$ and the kernel $k$ have shape $k_1\times k_2$. We use real-valued arrays, a single input channel, and a single kernel. Stride and dilation are 1, padding is 0, and bias is omitted. Assume that the kernel fits within the input.
The output $y$ has shape
\[H_{\mathrm{out}}=H-k_1+1,\qquad W_{\mathrm{out}}=W-k_2+1.\]Each output element is the sum of elementwise products between an input patch and the kernel:
\[y(r,c) = \sum_{a=0}^{k_1-1} \sum_{b=0}^{k_2-1} x(r+a,c+b)\,k(a,b),\]where the output coordinates satisfy
\[0\le r<H_{\mathrm{out}},\qquad 0\le c<W_{\mathrm{out}}\]This operation slides the kernel without flipping it, so its mathematical name is cross-correlation. Deep learning convention calls it convolution. The PyTorch Conv2d documentation also defines the forward operation as cross-correlation.
Assume that the loss $L$ depends on the input and kernel through the output $y$. The value received from the layers that follow is the gradient of the loss with respect to each output element:
\[\frac{\partial L}{\partial y(r,c)}.\]This gradient array is commonly called dout in code. Backpropagation uses it to compute
Gradient with respect to the kernel
Fix one kernel element $k(a’,b’)$. That element is reused at every output position as the kernel slides across the input. Its effect on the loss therefore includes contributions from all output positions.
Applying the chain rule gives
\[\frac{\partial L}{\partial k(a',b')} = \sum_{r=0}^{H_{\mathrm{out}}-1} \sum_{c=0}^{W_{\mathrm{out}}-1} \frac{\partial L}{\partial y(r,c)} \frac{\partial y(r,c)}{\partial k(a',b')}.\]Since $y$ is an array, the chain rule sums the contributions through its individual elements.
The term containing $k(a’,b’)$ in the forward equation is
\[x(r+a',c+b')\,k(a',b')\]so
\[\frac{\partial y(r,c)}{\partial k(a',b')} = x(r+a',c+b').\]Substituting this gives the kernel gradient:
\[\boxed{ \frac{\partial L}{\partial k(a',b')} = \sum_{r=0}^{H_{\mathrm{out}}-1} \sum_{c=0}^{W_{\mathrm{out}}-1} \frac{\partial L}{\partial y(r,c)}\,x(r+a',c+b') }\]At each output position, multiply the output gradient by the input value that was paired with this kernel element, then sum the results. Sharing a weight across positions means accumulating its gradient across those positions too.
Interpreting this as cross-correlation
Define cross-correlation for real-valued arrays $u,v$ as follows, with the sum taken over all elements of $v$:
\[(u\star v)(i,j) = \sum_r\sum_c u(i+r,j+c)\,v(r,c).\]The kernel gradient is then
\[\frac{\partial L}{\partial k} =x\star\frac{\partial L}{\partial y}\]Slide the output gradient $\frac{\partial L}{\partial y}$ across the input $x$ without flipping it, multiply overlapping elements, and sum the products.
We use valid cross-correlation, which evaluates only positions where the output gradient array fits entirely inside the input. The result has the same shape as the kernel:
\[\begin{aligned} H-H_{\mathrm{out}}+1 &= k_1,\\ W-W_{\mathrm{out}}+1 &= k_2. \end{aligned}\]Gradient with respect to the input
Now fix one input element $x(r’,c’)$. It can appear in several input patches used by the forward pass. Its gradient must therefore sum the contributions from every output position that uses it:
\[\frac{\partial L}{\partial x(r',c')} = \sum_{r=0}^{H_{\mathrm{out}}-1} \sum_{c=0}^{W_{\mathrm{out}}-1} \frac{\partial L}{\partial y(r,c)} \frac{\partial y(r,c)}{\partial x(r',c')}.\]Recall the forward equation:
\[y(r,c) = \sum_{a=0}^{k_1-1} \sum_{b=0}^{k_2-1} x(r+a,c+b)\,k(a,b).\]The term containing $x(r’,c’)$ must satisfy
\[r+a=r',\qquad c+b=c'\]or equivalently,
\[a=r'-r,\qquad b=c'-c\]If these coordinates fall within the kernel, the derivative is the kernel element at that position:
\[\frac{\partial y(r,c)}{\partial x(r',c')} = \begin{cases} k(r'-r,c'-c), & \text{if }0\le r'-r<k_1\text{ and }0\le c'-c<k_2,\\[4pt] 0,&\text{otherwise}. \end{cases}\]Change the summation indices from output coordinates $(r,c)$ to kernel coordinates $(a,b)$:
\[r=r'-a,\qquad c=c'-b.\]For valid output positions,
\[\frac{\partial y(r'-a,c'-b)}{\partial x(r',c')} =k(a,b)\]Choosing the output coordinates this way makes $k(a,b)$ the exact kernel element multiplied by $x(r’,c’)$.
Near the input boundary, however, $r’-a$ or $c’-b$ may lie outside the output. In the equations below, we treat $\frac{\partial L}{\partial y(r,c)}$ as 0 outside the output bounds. This zero extension ensures that only valid outputs contribute to the sum; it does not introduce derivatives of nonexistent output elements.
The resulting formula holds at every input position:
\[\boxed{ \frac{\partial L}{\partial x(r',c')} = \sum_{a=0}^{k_1-1} \sum_{b=0}^{k_2-1} \frac{\partial L}{\partial y(r'-a,c'-b)}\,k(a,b) }\]The kernel coordinates are subtracted in the output gradient indices. This is the characteristic form of convolution.
The stamp method: accumulating contributions at input positions
We can also compute the input gradient by stamping contributions into an array. Initialize an array with the input shape to 0. At each output position $(r,c)$, add the following contribution to each corresponding input position:
\[\frac{\partial L}{\partial x(r+a,c+b)} \mathrel{+}= \frac{\partial L}{\partial y(r,c)}\,k(a,b).\]Here $\mathrel{+}=$ means accumulating a contribution at that position. Calling the input gradient array dx, this corresponds to dx[r:r+k1, c:c+k2] += dout[r,c] * k. Multiply one element of dout by the entire kernel, then stamp and add the result into dx, starting at $(r,c)$. Repeat for every output position, adding overlapping contributions. Assignment would erase earlier contributions, so accumulation with += is essential.
The kernel keeps the same orientation as in the forward pass. The stamp method uses dout and the kernel at their original sizes, with no explicit padding.
Why full convolution?
Convolution can be computed by accumulating stamps or by sliding a kernel. Here, full specifies the output range, not the computation method. Full convolution includes every overlap position, including boundary positions where the two arrays overlap only partially.
Define convolution as follows, summing over all elements of $v$ and treating values outside either array as 0:
\[(u*v)(i,j) = \sum_a\sum_b u(i-a,j-b)\,v(a,b).\]Comparing this with the input gradient formula gives
\[\frac{\partial L}{\partial x} =\frac{\partial L}{\partial y}*k\]The stamp method computes this convolution by accumulating the contributions from each output position.
We need to include positions where the output gradient and kernel overlap only partially. Input elements at the boundary participated in the forward pass, so their gradients must be computed too. This requires full convolution.
The result has shape
\[\begin{aligned} H_{\mathrm{out}}+k_1-1 &= H,\\ W_{\mathrm{out}}+k_2-1 &= W \end{aligned}\]which matches the input shape.
Flipping the kernel to use cross-correlation
The forward operation defined earlier is cross-correlation. To use that operation to compute the input gradient, we must flip the kernel along both its height and width.
Define the flipped kernel as
\[k_{\mathrm{flip}}(a,b) = k(k_1-1-a,k_2-1-b)\]This is equivalent to rotating the kernel by $180^\circ$.
Reversing the summation indices in the input gradient formula gives
\[\frac{\partial L}{\partial x(r',c')} = \sum_{a=0}^{k_1-1} \sum_{b=0}^{k_2-1} \frac{\partial L}{\partial y\bigl( r'-(k_1-1)+a,\, c'-(k_2-1)+b \bigr)} \,k_{\mathrm{flip}}(a,b).\]This is valid cross-correlation with the flipped kernel after padding dout with $k_1-1$ zero rows on both the top and bottom, and $k_2-1$ zero columns on both the left and right.
The padding lets us compute the gradients at the boundary. Without padding, evaluating only positions where the kernel fits entirely inside dout would produce an array of shape $(H_{\mathrm{out}}-k_1+1)\times(W_{\mathrm{out}}-k_2+1)$. For a kernel larger than $1\times1$, this does not cover the whole input. If dout is smaller than the kernel in either dimension, there are no valid positions at all.
With $k_1-1$ rows of padding on each side, the height of the valid cross-correlation result is
\[H_{\mathrm{out}}+2(k_1-1)-k_1+1 =H_{\mathrm{out}}+k_1-1 =H\]The width is likewise $W$. This padding belongs to the backward computation and is consistent with using no padding in the forward pass.
The two expressions for the input gradient are therefore equivalent:
\[\begin{aligned} \frac{\partial L}{\partial x} &=\operatorname{full\_conv}\left(\frac{\partial L}{\partial y},k\right)\\[6pt] &=\operatorname{valid\_corr}\left( \operatorname{pad}\left(\frac{\partial L}{\partial y}\right),k_{\mathrm{flip}} \right). \end{aligned}\]The mathematical convolution expression uses the original kernel $k$. We flip it when implementing that convolution with cross-correlation. An operation that supports full convolution handles values outside the arrays as 0, so it needs no manual padding beforehand. Explicit padding is needed here because we are using valid cross-correlation to compute the same result.
How does one sliding step correspond to one stamp?
A sliding convolution computes a dot product at one position to complete one element of dx. A single stamp multiplies one element of dout by the entire kernel and adds contributions to several positions in dx. The equivalence means that the final arrays match after all positions have been processed.
Consider a one-dimensional example to see the correspondence:
dout = [1, 2, 3]
k = [4, 5]
To compute full convolution by sliding, pad dout with one zero on each side and take dot products with the flipped kernel [5, 4]:
padded dout = [0, 1, 2, 3, 0]
dx[0] = [0, 1] · [5, 4] = 4
dx[1] = [1, 2] · [5, 4] = 13
dx[2] = [2, 3] · [5, 4] = 22
dx[3] = [3, 0] · [5, 4] = 15
For the stamp method, initialize dx to 0, multiply each value in dout by the original kernel [4, 5], and accumulate with += starting at that position. Each row below shows the contribution from one stamp:
dout[0] = 1: [4, 5, 0, 0]
dout[1] = 2: [0, 8, 10, 0]
dout[2] = 3: [0, 0, 12, 15]
--------------
sum dx: [4, 13, 22, 15]
The value computed at one sliding position is the sum of one column in the stamp table. For example, sliding computes dx[1] in one dot product: $1\cdot5+2\cdot4=13$. Stamping adds 5 from the first stamp and 8 from the second to obtain the same 13.
The same relationship holds in two dimensions. Sliding collects the contributions for one coordinate of dx. Stamping distributes dout[r,c] * k across several input coordinates and accumulates those contributions.
Checking with a small example
Use the following input and kernel:
\[x= \begin{bmatrix} 1&2&3\\ 4&5&6\\ 7&8&9 \end{bmatrix}, \qquad k= \begin{bmatrix} 1&2\\ 3&4 \end{bmatrix}.\]The forward pass gives
\[y= \begin{bmatrix} 37&47\\ 67&77 \end{bmatrix}\]For example, the top-left output is
\[y(0,0) = 1\cdot1+2\cdot2+4\cdot3+5\cdot4 = 37\]To track the gradient contributions, define the loss as
\[L=y(0,0)+2y(1,1).\]The output gradient is then
\[\frac{\partial L}{\partial y} = \begin{bmatrix} 1&0\\ 0&2 \end{bmatrix}.\]Kernel gradient
The output gradient is nonzero only at $(0,0)$ and $(1,1)$. Multiply the input patch used at each position by its output gradient, then add the results:
\[\begin{aligned} \frac{\partial L}{\partial k} &= 1\cdot \begin{bmatrix} 1&2\\ 4&5 \end{bmatrix} + 2\cdot \begin{bmatrix} 5&6\\ 8&9 \end{bmatrix}\\[6pt] &= \begin{bmatrix} 11&14\\ 20&23 \end{bmatrix}. \end{aligned}\]For example, $k(0,0)$ multiplies $x(0,0)=1$ when computing $y(0,0)$, and $x(1,1)=5$ when computing $y(1,1)$. Thus,
\[\frac{\partial L}{\partial k(0,0)} = 1\cdot1+2\cdot5 = 11\]Input gradient
Each output sends gradient contributions back to the input patch used to compute it. The contribution to each input element is the output gradient multiplied by the corresponding kernel element.
The gradient 1 at $y(0,0)$ adds $k$ to the top-left input patch. The gradient 2 at $y(1,1)$ adds $2k$ to the bottom-right input patch:
\[\begin{aligned} \frac{\partial L}{\partial x} &= \begin{bmatrix} 1&2&0\\ 3&4&0\\ 0&0&0 \end{bmatrix} + \begin{bmatrix} 0&0&0\\ 0&2&4\\ 0&6&8 \end{bmatrix}\\[6pt] &= \begin{bmatrix} 1&2&0\\ 3&6&4\\ 0&6&8 \end{bmatrix}. \end{aligned}\]The central input $x(1,1)$ receives nonzero contributions from two outputs. It was multiplied by $k(1,1)=4$ in $y(0,0)$ and by $k(0,0)=1$ in $y(1,1)$, so
\[\frac{\partial L}{\partial x(1,1)} = 1\cdot4+2\cdot1 = 6\]Computing the same result with padding and sliding
The kernel is $2\times2$, so pad dout with one zero on every side. Rotate the kernel by $180^\circ$ for the sliding computation:
Valid cross-correlation of this $4\times4$ array with the flipped $2\times2$ kernel produces a $3\times3$ gradient array, matching the input shape. At the top-left position,
\[\frac{\partial L}{\partial x(0,0)} =0\cdot4+0\cdot3+0\cdot2+1\cdot1 =1\]and at the center,
\[\frac{\partial L}{\partial x(1,1)} =1\cdot4+0\cdot3+0\cdot2+2\cdot1 =6\]These match the values accumulated by stamping. Without padding, valid cross-correlation would produce only the central value 6, leaving the other input gradients uncomputed.
Stamping adds the contributions each output sends to the input. Sliding collects the contributions each input position receives. Both sum the same contributions in a different order and therefore produce the same result.