Skip to content

← Classifiers from Scratch step 3 of 5

Medium End-to-End

Sequence Classifier

Classify one token sequence by looking up embeddings, updating a recurrent hidden state and projecting the final state to class scores.

Your task

Implement sequence_classifier(indices, embedding_table, W_xh, W_hh, b_h, h0, W_out, b_out) using the supplied parameters.

  • indices has shape (seq_len,) and contains valid zero-based token IDs. The runtime delivers integer-valued floats; cast them to integers for lookup.
  • embedding_table has shape (vocab_size, embed_dim).
  • W_xh has shape (embed_dim, hidden_dim).
  • W_hh has shape (hidden_dim, hidden_dim).
  • b_h and h0 each have shape (hidden_dim,).
  • W_out has shape (hidden_dim, num_classes); b_out has shape (num_classes,).

The floating-point parameters share a dtype and device. Return logits of shape (num_classes,) with that dtype and device. Leave all inputs unchanged. There is one sequence, not a batch, and no padding mask: every supplied token is processed, including repeated IDs and ID 0.

Carry the hidden state forward

Start with h=h0. For each token in order, look up its embedding e and compute:

h_next = tanh(e @ W_xh + h @ W_hh + b_h)
h = h_next

Use the previous h for the whole update, then pass the new vector to the next token. The weights use row-vector orientation as written above. Do not reset h between tokens, apply tanh separately to the two products or average hidden states.

After the last token, compute logits = h @ W_out + b_out. Return these raw class scores, not softmax probabilities or an argmax class ID. With an empty sequence, project h0 directly. Do not apply tanh to h0 before that projection.

For a one-dimensional hidden state, embedding value 1, W_xh=1, W_hh=0.5, b_h=0 and h0=0, two repeated tokens give:

h after token 1 = tanh(1)                 = about 0.761594
h after token 2 = tanh(1 + 0.5*0.761594)  = about 0.881130

With W_out=2 and b_out=0, the single output logit is approximately 1.762259. Repetition still matters because the second update receives a different state.

The diagram below illustrates carrying a state through a sequence. Its scalar decay rule is an analogy, not the RNN equation or the numbers in this example.

Loading visualization…