Updating a parameter by hand

Everything in this track so far has been about memory. This is where in-place runs into autograd, and it is the bridge to the next track.

w = torch.tensor([1., 2.], requires_grad=True)
w.add_(0.1)

RuntimeError: a leaf Variable that requires grad is being used
in an in-place operation.

Why it refuses

A leaf is a tensor you created rather than one computed from others: parameters, inputs, anything with requires_grad=True and no grad_fn. Autograd needs a leaf’s value to stay put while a backward pass is planned against it. Letting you overwrite it mid-graph would make the recorded derivative refer to a value that no longer exists.

The fix

with torch.no_grad():
    w.add_(0.1)

Inside no_grad nothing is recorded, so there is no graph for the write to invalidate. This is exactly what every optimiser’s step() does, and why you will find @torch.no_grad() on it in PyTorch’s source.

Why in place, and not w = w - 0.1 * grad

Because rebinding gives you a new tensor, and the optimiser, the module, and anything else holding the parameter still points at the old one. Same lesson as the previous problem, with worse consequences: the model does not change and nothing complains.

A parameter update has to be a write, and a write to a leaf has to be under no_grad.

Gradients afterwards

w.grad = None

rather than w.grad.zero_(), which keeps the buffer alive. zero_grad (set_to_none=True) is the default in modern PyTorch for that reason.

Your task

def sgd_step(w: torch.Tensor, grad: torch.Tensor, lr: float) -> None

Update w in place by -lr * grad, leaving it a leaf that still requires gradients. Return nothing.