Medium Framework

Bidirectional RNN

A bidirectional RNN runs two independent recurrences over the same sequence. The forward state at position t has processed inputs from the start through t. The backward state at t has processed inputs from the end back through t. Joining those states gives each position information from both directions.

The backward pass needs later inputs, so this is useful when the full sequence is available. It is not a causal recurrence for predicting a next token without seeing future inputs. Both directions include the current position.

Inputs and output

Implement bidirectional_rnn(x, w_fwd, w_bwd, h0_fwd, h0_bwd).

x has shape (N, T, d_in).
w_fwd and w_bwd each have shape (d_in + d_h, d_h).
h0_fwd and h0_bwd each have shape (N, d_h).

All arrays are floating-point and all dimensions are positive. N is the batch size; each batch item has independent states. Return shape (N, T, 2 * d_h) without changing inputs. There is no bias, dropout, output projection or extra constant input feature.

Cell update

At each step, join the current input and that direction's previous state in this order, then multiply and apply tanh elementwise:

combined = concatenate([x[:, t, :], h], axis=-1)
h = tanh(combined @ w)

combined has shape (N, d_in + d_h). The first d_in rows of w act on input features, and the remaining d_h rows act on the hidden state. Reversing the concatenation order changes the calculation.

Two passes, one position order

For the forward pass, initialize h from h0_fwd and visit positions 0, 1, ..., T-1, using w_fwd. Save the updated state at every position.

For the backward pass, initialize a separate h from h0_bwd and visit T-1, T-2, ..., 0, using w_bwd. The initial backward state is used before processing the last input, not before processing the first. Save each updated state under its original position t.

Return [forward_state[t], backward_state[t]] along the feature axis for each original position t. The first d_h output features are forward states; the last d_h are backward states. Neither initial state is an output row.

Alignment example

With three input positions, the backward recurrence produces states in the order b2, b1, b0. The output still follows the input order:

position 0: [f0, b0]
position 1: [f1, b1]
position 2: [f2, b2]

If you append backward states to a list while traversing in reverse, reverse that list before joining it with the forward states. Alternatively, write each backward state directly into its original position.

With T=1, both directions process the same input once, but their results can differ because they have separate weights and initial states.

Understand the ideas behind this exercise. The Problem tab contains the exact implementation requirements.

Update a state as you read a sequence

A recurrent neural network carries a hidden state from one time step to the next. A basic cell computes

$$h_t=\tanh(x_tW_x+h_{t-1}W_h+b).$$

x_t is the current input and h_(t-1) is the previous state. The weights and bias are shared across time steps. The hidden state changes with the input; it is not a separate learned parameter at each step.

Even when the input and previous state are zero, a nonzero bias can produce a nonzero new state. For sequence classification, a classifier can read the final state. For token labeling or language modeling, it may need an output at every step instead.

Gates control how information is carried forward

An LSTM maintains a cell state as well as a hidden state. Input, forget, and output gates control how information enters, remains in, and is read from that cell state. A GRU uses update and reset gates with a candidate hidden state.

Follow the given equations for gate order and placement. In particular, applying a reset gate before a matrix multiplication is generally different from applying it afterward.

A bidirectional RNN reads the sequence in both directions. Align the two states with their original token positions before concatenating them. The reverse recurrence can use later tokens, so its representation cannot be used unchanged for a causal next-token predictor.

Add gradient contributions across time

Backpropagation through time treats the recurrence as repeated uses of the same cell. Each use of a weight contributes to that weight's total gradient. A hidden state can receive gradient from its current output and from later states.

Repeated multiplication by derivatives can shrink or amplify gradients. Gradient clipping bounds the gradient passed to the optimizer; the resulting parameter step also depends on the optimizer and its state. Gates provide other paths for information and gradients, but do not replace correct handling of padding, initial states, and sequence lengths.

Detecting runtime…