Skip to content
← All tracks

Activations

PyTorch

From the classic non-linearities to modern gated activations. End with implementing a custom gradient.

0 / 9 solved
Continue →
  1. 1. Not solved yet. Implement ReLU
  2. 2. Not solved yet. Implement Sigmoid
  3. 3. Not solved yet. Implement Tanh
  4. 4. Not solved yet. Implement Leaky ReLU
  5. 5. Not solved yet. GELU Activation
  6. 6. Not solved yet. SwiGLU Activation
  7. 7. Not solved yet. GLU Activation
  8. 8. Not solved yet. Implement Softmax
  9. 9. Not solved yet. Custom Activation with Gradient

Check yourself

4 questions · one attempt each

These do not count toward finishing the track. They are here to catch the things that are easy to read past.

0 / 4

Why do deep networks avoid sigmoid in hidden layers? 

torch.sigmoid(torch.tensor([-10., 0., 10.]))
# and its gradient
Question 1 of 4