Medium End-to-End

GRU Sequence Forward Pass

A gated recurrent unit (GRU) updates a hidden state using a candidate value and two gates. The reset gate controls how much of the old state contributes to the candidate. The update gate blends that candidate with the old state.

This exercise processes a complete sequence and returns only the final state. Use the particular reset placement and update convention specified below.

Inputs and output

Implement gru_forward(x, W_xr, W_xz, W_xn, W_hr, W_hz, W_hn, b_r, b_z, b_n, h0).

x has shape (seq_len, input_dim).
W_xr, W_xz and W_xn each have shape (input_dim, hidden_dim).
W_hr, W_hz and W_hn each have shape (hidden_dim, hidden_dim).
b_r, b_z, b_n and h0 each have shape (hidden_dim,).

All arrays are floating-point and all dimensions are positive. There is no batch axis. Return shape (hidden_dim,), not a stack of intermediate states. Do not change any input.

Update one position

Start with h = h0. For each row x[t], compute the following using the same previous h throughout the step:

r = sigmoid(x[t] @ W_xr + h @ W_hr + b_r)
z = sigmoid(x[t] @ W_xz + h @ W_hz + b_z)
candidate = tanh(x[t] @ W_xn + (r * h) @ W_hn + b_n)
new_h = (1 - z) * candidate + z * h

Then set h to new_h and continue. All * operations are elementwise; @ denotes matrix multiplication. Use stable sigmoid operations.

The reset multiplication happens before W_hn: (r * h) @ W_hn. It is generally different from r * (h @ W_hn) because the matrix can mix hidden features. Do not interchange these expressions or substitute a built-in GRU with a different reset convention.

In this update convention, z near one retains more of the old state; z near zero favors the candidate. The reset gate r affects the candidate branch only. It does not directly multiply the retained z * h branch.

Example

If all weights and biases are zero, both gates equal 0.5 and the candidate is zero. Starting from h0 = [1], one step returns [0.5] and two steps return [0.25]. This shows both the carry between positions and the rule that only the final state is returned.

Understand the ideas behind this exercise. The Problem tab contains the exact implementation requirements.

Update a state as you read a sequence

A recurrent neural network carries a hidden state from one time step to the next. A basic cell computes

$$h_t=\tanh(x_tW_x+h_{t-1}W_h+b).$$

x_t is the current input and h_(t-1) is the previous state. The weights and bias are shared across time steps. The hidden state changes with the input; it is not a separate learned parameter at each step.

Even when the input and previous state are zero, a nonzero bias can produce a nonzero new state. For sequence classification, a classifier can read the final state. For token labeling or language modeling, it may need an output at every step instead.

Gates control how information is carried forward

An LSTM maintains a cell state as well as a hidden state. Input, forget, and output gates control how information enters, remains in, and is read from that cell state. A GRU uses update and reset gates with a candidate hidden state.

Follow the given equations for gate order and placement. In particular, applying a reset gate before a matrix multiplication is generally different from applying it afterward.

A bidirectional RNN reads the sequence in both directions. Align the two states with their original token positions before concatenating them. The reverse recurrence can use later tokens, so its representation cannot be used unchanged for a causal next-token predictor.

Add gradient contributions across time

Backpropagation through time treats the recurrence as repeated uses of the same cell. Each use of a weight contributes to that weight's total gradient. A hidden state can receive gradient from its current output and from later states.

Repeated multiplication by derivatives can shrink or amplify gradients. Gradient clipping bounds the gradient passed to the optimizer; the resulting parameter step also depends on the optimizer and its state. Gates provide other paths for information and gradients, but do not replace correct handling of padding, initial states, and sequence lengths.

Detecting runtime…