Skip to content
← All PyTorch tracks

Activations

PyTorch

From the classic non-linearities to modern gated activations. End with implementing a custom gradient.

9 exercises

Explore freely. Sign in to keep your progress.

Start track

Your learning sequence

  1. 1 Not solved yet. Implement ReLU Practice · Easy
  2. 2 Not solved yet. Implement Sigmoid Practice · Easy
  3. 3 Not solved yet. Implement Tanh Practice · Easy
  4. 4 Not solved yet. Implement Leaky ReLU Practice · Easy
  5. 5 Not solved yet. GELU Activation Practice · Easy
  6. 6 Not solved yet. SwiGLU Activation Practice · Medium
  7. 7 Not solved yet. GLU Activation Practice · Easy
  8. 8 Not solved yet. Implement Softmax Practice · Easy
  9. 9 Not solved yet. Custom Activation with Gradient Practice · Hard

Check yourself

4 questions · not saved while you're signed out

Use these questions to check your understanding and compare explanations. Your score does not affect track completion.

0 / 4 correct

What gradient problem can repeated sigmoid activations cause in a deep network?

Choose one answer to reveal the explanations. Answers are kept only while this page is open. 

import torch

x = torch.tensor([-10., 0., 10.], dtype=torch.float64,
                 requires_grad=True)
y = torch.sigmoid(x)
y.sum().backward()
print("sigmoid:", [f"{v:.6f}" for v in y.tolist()])
print("gradient:", [f"{v:.7f}" for v in x.grad.tolist()])
Question 1 of 4