Skip to content

← Memory, Copies and In-Place step 6 of 6

Medium Primitives

Accumulate into an existing tensor

Jump to the task

A function can return the correct total while leaving the caller's buffer unchanged. This happens when each addition creates a new tensor and the function only reassigns its local name:

import torch

def rebound_sum(acc, chunks):
    for chunk in chunks:
        acc = acc + chunk
    return acc

acc = torch.tensor([10.0, 20.0])
result = rebound_sum(acc, [torch.tensor([1.0, 2.0])])
print(result.tolist())  # [11.0, 22.0]
print(acc.tolist())     # [10.0, 20.0]

The name acc inside the function now refers to a different tensor. The caller still holds the original one.

Update the existing tensor

acc.add_(chunk) writes the sum into acc itself. Other code holding that tensor sees the change. For these ordinary dense tensors, it also avoids creating a new full-size result tensor for each addition. That does not mean every part of the operation is free of overhead or temporary storage.

How this relates to gradients

PyTorch accumulates gradients across backward calls until you clear them. An optimizer reads each parameter's current .grad attribute. This is related to adding contributions into an accumulator, but it is not a promise that .grad always keeps the same tensor object: the first backward can create it, and higher-order gradient calculation can replace it. Assigning a new tensor to p.grad also updates the attribute the optimizer reads. Rebinding an unrelated local variable does not.

This exercise isolates ordinary tensor mutation. It does not implement backpropagation or a full microbatch training loop.

Your task

def accumulate(acc: torch.Tensor, chunks: list) -> torch.Tensor

Add each chunk elementwise into acc, in place, and return that same tensor. Start from the values already in acc; do not reset it. All inputs are one-dimensional float32 tensors with matching shapes and no gradient tracking. If chunks is empty, return acc unchanged.