Denoising Autoencoder Lab

Your project

Build a small denoising autoencoder from its two trainable matrices. Compress noisy vectors, reconstruct clean targets, and watch mean squared error change. This is a linear autoencoder, with no activation or bias. It introduces encoder/decoder training before a VAE; it does not learn a sampling distribution.

Forward pass

noisy and clean have shape (N, D). initial_encoder has shape (D, H) and initial_decoder has shape (H, D). All dimensions are positive and all inputs are float32 tensors. Copy the initial parameters; do not mutate them. Compute latent = noisy @ encoder, then reconstruction = latent @ decoder. Record MSE as the mean of squared errors across all N*D elements.

Backpropagation

For error = reconstruction - clean, compute g = 2*error/(N*D). Then d_decoder = latent.T @ g and d_encoder = noisy.T @ (g @ decoder.T). Calculate both gradients from the same pre-update parameters, then subtract lr times each gradient. Use full-batch gradient descent with no regularisation or optimiser state.

Return contract

Return ordinary Python values: encoder (D by H), decoder (H by D), latent (N by H), reconstruction (N by D), and losses (steps + 1 floats). Latent vectors and reconstructions must use the final parameters. Record the initial MSE even when steps is zero. steps and lr are nonnegative.

Worked example

For noisy=[[1]], clean=[[0]], encoder=[[1]], decoder=[[1]], steps=1 and lr=0.1, both gradients are [[2]], so both parameters become [[0.8]]. Final latent is [[0.8]], reconstruction is [[0.64]], and losses are [1, 0.4096]. Updating the decoder before computing the encoder gradient gives a different, wrong result.

Experiment after you pass

Export the experiment and perturb a new pattern. Vary bottleneck width and noise strength; compare reconstruction errors on separate clean targets. Interpolate between two latent vectors and decode them. This produces blends, but arbitrary random latent samples need not resemble the data: a VAE adds a probabilistic objective to address that gap. The displayed training loss is not a held-out metric.