Hard Research

Depthwise Separable Convolution

A depthwise separable convolution splits filtering into two steps. First, each input channel uses its own spatial filter. Then a shared matrix mixes the filtered channels at each position.

Implement depthwise_separable_conv(x, dw_filters, pw_weights) for a one-dimensional signal with no batch axis.

Inputs and output

x has shape (C_in, L): input channels by signal length.
dw_filters has shape (C_in, K): one length-K filter per input channel.
pw_weights has shape (C_out, C_in): one row per output channel.
Channel counts are positive and 1 <= K <= L. Inputs share a floating-point dtype and device.
Return shape (C_out, L - K + 1), preserving the input dtype and device.

Step 1: filter each channel

Use stride 1, dilation 1 and no padding. At each position i, multiply the K values starting at x[c, i] by dw_filters[c], then sum:

depthwise[c, i] = sum(x[c, i + j] * dw_filters[c, j] for j in range(K))

Use the filter in its given order; do not reverse it. This is the cross-correlation convention used by neural-network convolution layers. Channels stay separate in this step, giving shape (C_in, L - K + 1).

For example, filtering [1, 2, 3, 4] with [1, 0, -1] gives [-2, -2]. When K equals L, there is exactly one output position.

Step 2: mix channels

At every position, combine the filtered channel values using the same pw_weights matrix:

output = pw_weights @ depthwise

This is a pointwise convolution: it mixes channels without looking at neighboring positions. Include no bias, activation or residual addition, and leave the inputs unchanged.

The window below illustrates the first step for one channel. The second step changes the number of channels, not the output length.

Loading visualization…

Understand the ideas behind this exercise. The Problem tab contains the exact implementation requirements.

Apply the same filter at many locations

A convolutional layer slides a learned kernel over its input. At each location, it multiplies the covered values by the kernel weights and sums them. Deep-learning libraries usually implement cross-correlation under the name convolution: the kernel is not reversed.

For input [1, 2, 3] and kernel [1, -1], with no padding and stride one, the outputs are 1*1 + 2*(-1) = -1 and 2*1 + 3*(-1) = -1.

Work out the output shape

For one spatial axis, the output length is

$$\left\lfloor\frac{L+2P-D(K-1)-1}{S}\right\rfloor+1.$$

L is the input length, K the kernel size, P the padding on each end, S the stride between windows, and D the spacing between kernel elements (dilation). For a 2D input, apply the formula separately to height and width. Check the channel layout too: an ordinary convolution sums over both spatial positions in the window and input channels.

A depthwise convolution filters each input channel separately. A following pointwise convolution, using a 1-by-1 spatial kernel, mixes channels. This uses a different set of parameters from a full spatial convolution; the two constructions are not generally interchangeable.

Pool nearby features

Max pooling keeps the largest value in each window. Average pooling takes the window's mean. During backpropagation, max pooling sends the gradient to the selected position, while average pooling distributes it across the values that contributed to the mean. Overlapping windows can contribute more than once to the same input position, so add those gradients.

Follow the exercise's rules for tied maxima and whether padding counts in an average. For a small CNN, also save the forward-pass values needed for activation and pooling derivatives. Add each shared kernel's gradient contributions across all locations, then update weights after computing all gradients.

Detecting runtime…