Easy End-to-End

Logistic Regression From Scratch

Logistic regression uses a weighted sum of features to estimate the probability of class 1. Applying a sigmoid turns the score z into a probability: sigmoid(z) = 1 / (1 + exp(-z)).

In this exercise, train the weights using full-batch gradient descent. Every step uses all examples, and training stops after the requested number of steps.

Your task

Implement fit_logistic(x, y, lr, n_steps):

x is a floating-point feature matrix of shape (N, d), with N, d >= 1.
y has shape (N,) and contains labels 0 or 1.
lr is a nonnegative learning rate.
n_steps is a nonnegative integer.

Start with a zero weight vector of shape (d,). Repeat these operations exactly n_steps times:

p = sigmoid(x @ w)           # shape (N,)
grad = x.T @ (p - y) / N     # shape (d,)
w = w - lr * grad

Compute each gradient using the current weights, then update all weights together. Use these explicit operations rather than an optimizer or automatic differentiation. Return only the final weight vector, preserving the inputs. For zero steps, return zeros. With lr = 0, the weights also stay zero.

Why this update?

The training objective is mean binary cross-entropy:

loss = -mean(y * log(p) + (1 - y) * log(1 - p))

Its gradient with respect to the weights is x.T @ (p - y) / N. The division by N averages across examples. You do not need to compute or return the loss itself, and you should not multiply this gradient by another sigmoid derivative. The formula already accounts for it.

Intercept and example

Use x exactly as supplied. There is no separate bias parameter and your function must not add a column. When the caller supplies a column of ones, its weight acts as the intercept.

For x = [[1, 2], [1, -1], [1, 0]] and y = [1, 0, 1], the initial weights [0, 0] give probabilities [0.5, 0.5, 0.5]. The first gradient is [-1/6, -1/2]. With lr = 0.6, one step returns [0.1, 0.3]. The first weight is the intercept because this input already has a column of ones. Subsequent steps must recompute the probabilities.

A fixed number of steps does not guarantee convergence. Follow the given step count even if the weights are still changing.

Understand the ideas behind this exercise. The Problem tab contains the exact implementation requirements.

Predict the probability of one of two outcomes

A linear binary classifier first computes a score, called a logit:

$$z=w^Tx+b.$$

The weights w combine the input features x. The bias b shifts the score. The sigmoid function converts it to the probability of class 1:

$$p=\frac{1}{1+e^{-z}}.$$

A logit of zero gives probability 0.5. Positive logits give probabilities above 0.5; negative logits give probabilities below it. Predicting class 1 when p >= 0.5 is equivalent to checking z >= 0. Without a bias, the boundary w^T x = 0 passes through the origin.

Train with probabilities, not thresholded predictions

For the true label y, which is either zero or one, binary cross-entropy is

$$\ell=-y\log p-(1-y)\log(1-p).$$

If y = 1, only -log(p) remains. If y = 0, only -log(1-p) remains. In both cases, assigning a higher probability to the true class reduces the loss.

A numerically stable form works directly with the logit:

$$\ell=\max(z,0)-yz+\log(1+e^{-|z|}).$$

This avoids taking logs of probabilities that have rounded to zero or one. The derivative with respect to z is p - y. For mean loss, average this error to get the bias gradient, and average its product with each feature to get that feature's weight gradient.

At zero weights and bias, all probabilities are 0.5 and mean loss is log(2). Lower training loss means the model assigns better probabilities to the training labels. It does not guarantee better results on new data, and a linear boundary cannot fit every pattern.

Apply the decision threshold when making predictions. Replacing probabilities with hard class labels before computing the training loss discards information about how confident the model is.

Detecting runtime…

Logistic Regression From Scratch

Your task

Why this update?

Intercept and example

Predict the probability of one of two outcomes

Train with probabilities, not thresholded predictions

Your code, saved here

Stuck?

PyTorch reference solution