Clear gradients between independent backward passes

backward() adds to an existing .grad rather than replacing it. When each pass should be independent, clear the previous gradient before the next pass.

import torch

x = torch.ones(3, requires_grad=True)
(x * 2).sum().backward()
print(x.grad.tolist())  # [2.0, 2.0, 2.0]
(x * 2).sum().backward()
print(x.grad.tolist())  # [4.0, 4.0, 4.0]

Each expression builds a new graph, so these are two separate backward passes. The second result includes both contributions even though x has not changed.

When accumulation is intentional

Accumulation lets several losses contribute to one update. To reproduce the mean loss over equal-sized microbatches, divide each microbatch mean by the number of microbatches before calling backward. Otherwise, summing four means gives four times the full-batch mean gradient.

For unequal microbatch sizes, weight each mean by its share of the total examples. This equivalence assumes an additive per-example loss and consistent forward behavior; layers using batch statistics can change the result. Accumulation can reduce peak activation memory, but the saving is not a fixed fraction of total training memory.

If several losses share a graph, adding them and calling backward once is often simplest. Separate backward calls may need retain_graph=True until the last call so that required saved values remain available.

Clear gradients between independent passes

x.grad = None
# For parameters managed by an optimizer:
# optimizer.zero_grad(set_to_none=True)

Setting .grad to None removes that tensor's reference to the gradient buffer. Zeroing an existing buffer is also valid, but keeps the buffer. These choices can affect optimizer behavior: a parameter with no gradient may be skipped, while one with a zero gradient may still receive momentum or weight-decay updates.

Forgetting to clear gradients changes the update direction and magnitude. It can harm training, but does not imply a fixed effective learning rate or guaranteed divergence after a particular number of steps.

def grads_per_step(x: torch.Tensor, steps: int) -> list