One bias, every position

To add one bias value per channel, first locate the channel axis. Here the activations have shape (batch, time, channels) and the bias has shape (channels,).

Broadcasting aligns dimensions from the right:

activations: (batch, time, channels)
bias:                    (channels,)
aligned as:  (    1,    1, channels)
result:      (batch, time, channels)

The singleton dimensions let the same channel biases apply at every time step and to every batch item. No explicit reshape is needed.

import torch

x = torch.zeros(1, 2, 3)
bias = torch.tensor([1.0, 2.0, 3.0])
print((x + bias).tolist())
# [[[1.0, 2.0, 3.0], [1.0, 2.0, 3.0]]]

Check the layout, not just the dimension sizes

Features are not always on the last axis. For an image tensor shaped (batch, channels, height, width), a channel bias needs a shape such as (1, channels, 1, 1). A convolution layer handles its bias internally; adding a bare channel vector to an NCHW tensor is not generally equivalent.

The starter reshapes the bias to (1, channels, 1), placing its values on the time axis instead. If time and channels differ, this can raise an error or expand an unintended singleton dimension. If they match, the addition can run while applying the wrong bias to each channel. Matching sizes do not prove that the axes have the intended meaning.

Match shapes from the right

Broadcasting lets an operation apply across compatible shapes without explicitly constructing repeated inputs. Compare dimensions from the right: each pair must be equal, or one must be 1. Missing leading dimensions behave like dimensions of 1. At each position, equal sizes stay the same. If one size is 1, the result uses the other size. This also covers empty dimensions: (0, 3) and (1, 3) broadcast to (0, 3), not (1, 3).

For example, (4, 3) plus (3,) applies the same three-feature bias to all four rows. Adding (4, 1) instead applies a different scalar to each row. A size-one axis lets the same value be reused along that axis.

This explains a subtle centring bug: reducing (B, D) over features gives (B,). Subtracting it directly aligns B with the feature axis. If B == D, the operation runs but uses the wrong means. Retaining the reduction axis as (B, 1) expresses the intended dependence.

Check how much memory the result needs

The input need not be physically copied, but the output can still be huge: (N, 1, D) minus (1, M, D) has shape (N, M, D). An in-place operation must fit its result in the existing destination shape. Explicit expansion may also create a view whose positions refer to the same storage, making independent writes unsafe.

Before implementing an operation, write the aligned shapes and the result shape. Then check which inputs are reused and how much space the output needs.