Medium Research

Squeeze-and-Excitation Block

A squeeze-and-excitation block computes one gate per channel, then uses that gate to scale every spatial position in the channel. The gates depend on the input, so different inputs can receive different channel weights.

Implement se_block(x, W1, W2) for one example, with its spatial dimensions already flattened into a single axis.

Inputs and output

x has shape (C, S): C channels and S spatial positions per channel.
W1 has shape (D, C) and W2 has shape (C, D).
C, S and D are positive integers. D is the hidden width, supplied by the weight shapes; there is no separate reduction-ratio argument.
All inputs are floating-point tensors. Return a tensor of shape (C, S).

Computation

Squeeze: average each channel over its S positions. This gives z with shape (C,). Do not average across channels.
Excitation: compute h = ReLU(W1 @ z), with shape (D,), then g = sigmoid(W2 @ h), with shape (C,). Both weight matrices belong to this step. ReLU replaces negative values with zero.
Scale: return out[c, s] = x[c, s] * g[c]. Reshape the gates to (C, 1) so each gate broadcasts across its channel's positions.

Sigmoid acts independently on each channel; the gates do not need to sum to one. A gate closer to zero suppresses that channel more strongly. There are no biases or residual additions in this exercise.

For example, if the excitation output before sigmoid is zero in every channel, every gate is 0.5 and the result is x / 2. Negative input features remain negative when multiplied by a positive gate.

This exercise implements the channel-gating operation from Squeeze-and-Excitation Networks (Hu et al., 2018).

Understand the ideas behind this exercise. The Problem tab contains the exact implementation requirements.

Add a learned change to the input

A residual connection computes y = x + F(x). The branch F learns a change, while the shortcut carries x directly to the sum. If F(x) is zero, the sum is just x. An activation applied after the sum can still change it.

For example, x = [2, 3] and F(x) = [-1, 0.5] give y = [1, 3.5]. The two branches need matching shapes for the intended elementwise addition. A projection can change the shortcut's width when necessary.

The derivative is I + dF/dx, so there is a gradient path through the shortcut as well as through the learned branch. This can help optimization, but it does not guarantee stable training. Scale, activations, and normalization placement still matter.

Use the whole image to weight each channel

A squeeze-and-excitation block computes one gate per channel in three steps:

Average each channel over its spatial positions. A channel containing [1, 3] has average 2.
Pass the channel averages through a small network to produce gates, commonly using a final sigmoid.
Multiply every spatial position in each channel by that channel's gate.

The squeeze step reduces spatial axes while preserving channels. The excitation network combines information from different channels to choose their gates. Applying the gates preserves the original spatial shape.

Check the task's equations for biases, intermediate activations, and any residual connection around the block. The block's name alone does not specify all of these choices.

Detecting runtime…

Squeeze-and-Excitation Block

Inputs and output

Computation

Add a learned change to the input

Use the whole image to weight each channel

Your code, saved here

Stuck?

PyTorch reference solution