Medium End-to-End

Adapter Layer

An adapter adds a small learned correction to an existing activation vector. It projects the vector into a bottleneck, applies a nonlinearity, then projects back to the original width. A residual connection adds the original input.

This exercise implements a simplified, bias-free ReLU adapter. Evaluate the supplied weights; no training, initialization or base model is required.

Your task

Implement adapter(x, w_down, w_up) for floating-point inputs:

x has shape (..., d).
w_down has shape (d, b).
w_up has shape (b, d).

Both widths are positive. Leading dimensions may represent examples, tokens, or both. Apply the same weights independently to every vector on the last axis and preserve all leading dimensions:

down = x @ w_down         # (..., b)
active = relu(down)       # (..., b)
correction = active @ w_up  # (..., d)
output = x + correction    # (..., d)

Return output with the same shape as x. ReLU replaces negative bottleneck values with zero. Do not apply it to x or to the final output, which may be negative. Do not add biases, normalization, scaling or dropout. Preserve the inputs.

Example

For x = [2, -1], w_down = [[1], [1]] and w_up = [[0.5, -2]], the bottleneck value is 1. ReLU leaves it unchanged, the correction is [0.5, -2], and the output is [2.5, -3].

If the correction is zero, the residual path returns x unchanged. For example, setting w_up to zero gives this identity behavior. This is a forward-pass property, not advice to initialize both matrices to zero: in this bias-free model, both-zero initialization gives zero gradients for both matrices and prevents ordinary gradient descent from learning.

Why use a bottleneck?

The two matrices contain 2*d*b parameters. When b is much smaller than d, this is fewer than the d*d parameters of a full square projection. For d = 1024 and b = 64, the counts are 131072 and 1048576. The function also accepts other positive widths; a narrow bottleneck is a modeling choice, not a shape requirement.

Unlike this nonlinear correction, a linear low-rank branch (x @ A) @ B can be written as x @ (A @ B). Here A @ B is a weight update, while x @ (A @ B) is its effect on activations. ReLU generally prevents merging an adapter's correction into a single fixed weight matrix.

For background, see Houlsby et al., Parameter-Efficient Transfer Learning for NLP. Their full setup includes biases, transformer integration and additional trainable parameters; this exercise isolates the residual bottleneck calculation.

Understand the ideas behind this exercise. The Problem tab contains the exact implementation requirements.

Train a smaller set of parameters

Low-rank adaptation, or LoRA, learns a change to a weight matrix using two smaller matrices. For row-vector inputs and weight shape (d_in, d_out), one convention is

$$W_{effective}=W+\frac{\alpha}{r}AB,$$

where A has shape (d_in, r) and B has shape (r, d_out). The base weight W can remain fixed while A and B learn.

For d_in = d_out = 100 and r = 4, the factors contain 400 + 400 = 800 parameters, compared with 10,000 in the full matrix. The saving comes with a constraint: the update has rank at most r. Follow the task's matrix orientation and scale, since implementations use different conventions.

Start with an initialization that allows learning

If both factors are zero, both receive zero gradients through their product. A common initialization makes one factor random and the other zero. The initial weight change is still zero, but the zero factor can receive a nonzero gradient and start learning.

Distinguish other adaptation methods

An adapter applies a small network to intermediate activations, often reducing then restoring the feature width in a residual branch. Prefix tuning learns context such as key/value vectors that tokens can attend to. These approaches modify different parts of the model and are not substitutes for one another.

To freeze base weights, ensure the optimizer cannot move them through momentum or weight decay as well as through current gradients. Check that base values stay unchanged after an update, while the loss still has a gradient path to the trainable factors or prefix.

Detecting runtime…