Medium End-to-End

RNN Sequence Forward Pass

A recurrent neural network carries a hidden state from one sequence position to the next. Each update combines the current input with the previous state, then applies tanh. Later states can therefore depend on earlier inputs.

Your task

Implement rnn_forward(x, W_xh, W_hh, b_h, h0) for a complete sequence:

x has shape (seq_len, input_dim).
W_xh has shape (input_dim, hidden_dim).
W_hh has shape (hidden_dim, hidden_dim).
b_h and h0 each have shape (hidden_dim,).

All arrays are floating-point and all dimensions are positive. There is no batch axis. Start with h = h0, then process rows of x from first to last:

h = tanh(x[t] @ W_xh + h @ W_hh + b_h)

The right-hand side uses the state from the previous step. Apply tanh elementwise and use the matrices in the supplied orientation. Reuse the same weights and bias at every step.

Return every updated state, stacked in order with shape (seq_len, hidden_dim). The first output row is the state after processing x[0]; do not prepend h0 or return only the final state. Keep inputs unchanged.

Example

For a one-feature sequence x = [[1], [1]], input weight 1, recurrent weight 0.5, bias 0 and initial state 0:

first state  = tanh(1 + 0.5 * 0)        = 0.761594
second state = tanh(1 + 0.5 * 0.761594) = 0.881130

Return approximately [[0.761594], [0.881130]]. The second step uses the new first state, not h0 again. If W_hh is all zeros, the previous state contributes nothing and each row depends only on that row's input and bias.

Understand the ideas behind this exercise. The Problem tab contains the exact implementation requirements.

Update a state as you read a sequence

A recurrent neural network carries a hidden state from one time step to the next. A basic cell computes

$$h_t=\tanh(x_tW_x+h_{t-1}W_h+b).$$

x_t is the current input and h_(t-1) is the previous state. The weights and bias are shared across time steps. The hidden state changes with the input; it is not a separate learned parameter at each step.

Even when the input and previous state are zero, a nonzero bias can produce a nonzero new state. For sequence classification, a classifier can read the final state. For token labeling or language modeling, it may need an output at every step instead.

Gates control how information is carried forward

An LSTM maintains a cell state as well as a hidden state. Input, forget, and output gates control how information enters, remains in, and is read from that cell state. A GRU uses update and reset gates with a candidate hidden state.

Follow the given equations for gate order and placement. In particular, applying a reset gate before a matrix multiplication is generally different from applying it afterward.

A bidirectional RNN reads the sequence in both directions. Align the two states with their original token positions before concatenating them. The reverse recurrence can use later tokens, so its representation cannot be used unchanged for a causal next-token predictor.

Add gradient contributions across time

Backpropagation through time treats the recurrence as repeated uses of the same cell. Each use of a weight contributes to that weight's total gradient. A hidden state can receive gradient from its current output and from later states.

Repeated multiplication by derivatives can shrink or amplify gradients. Gradient clipping bounds the gradient passed to the optimizer; the resulting parameter step also depends on the optimizer and its state. Gates provide other paths for information and gradients, but do not replace correct handling of padding, initial states, and sequence lengths.

Detecting runtime…

RNN Sequence Forward Pass

Your task

Example

Update a state as you read a sequence

Gates control how information is carried forward

Add gradient contributions across time

Your code, saved here

Stuck?

PyTorch reference solution