Skip to content

← CNNs from Scratch step 7 of 15

Medium End-to-End

Skip Connection Block

A skip connection adds an input directly to a learned transformation of that input. Here, the transformation has two dense layers with a ReLU between them, and the block applies another ReLU after the addition.

Your task

Implement residual_block(x, W1, b1, W2, b2):

  • x has shape (batch, dim), with one example per row.
  • W1 and W2 each have shape (dim, dim).
  • b1 and b2 each have shape (dim,) and are shared across the batch.
  • Return shape (batch, dim), preserving x's floating-point dtype.

The supplied tensors have matching dtype and device. Compute these operations in order, using the matrices in their supplied orientation:

h = ReLU(x @ W1 + b1)
branch = h @ W2 + b2
output = ReLU(x + branch)

ReLU(z) replaces negative entries with zero. Multiply row vectors on the right by W1 and W2; do not transpose either matrix. There is no activation between the second projection and skip addition. Leave all inputs unchanged.

For x=[[1, -2]], identity W1 and W2, b1=[0, 0] and b2=[-2, 3]:

h      = [[1, 0]]
branch = [[-1, 3]]
x + branch = [[0, 1]]
output = [[0, 1]]

The skip path adds the original x, not h. If the branch is zero, the result is ReLU(x), so negative inputs still become zero. This block's final ReLU means a zero branch is not an identity mapping for arbitrary inputs.

This is a small dense residual block, not a complete convolutional ResNet block. There is no normalization, dropout or projection on the skip path.