Medium Research

SwiGLU Activation

SwiGLU combines two projections of the same input. One projection goes through SiLU (also called Swish); its result multiplies the other projection element by element. This exercise starts after the projections are computed.

Your task

Implement swiglu(gate, value) for two floating-point arrays of the same shape (batch, d), with positive dimensions. Return the same shape without changing either input:

silu(gate) = gate * sigmoid(gate)
output = silu(gate) * value

Both multiplications are elementwise. Do not split either input, add weights or biases, or apply a final projection. The two arguments already contain the gate and value projections. Use a stable sigmoid or a framework SiLU operation, including for large negative gate values.

Example

For gate = [[1, 2]] and value = [[3, 4]], SiLU gives approximately [[0.731059, 1.761594]]. Multiplying by value gives [[2.193176, 7.046377]].

A zero gate produces zero output regardless of value. Negative gates can produce negative SiLU values, so this is not a multiplier constrained to [0, 1]. For example, gate -1 and value 2 give approximately -0.537883. If value is negative, the final sign can change again.

The chart shows the SiLU part and compares it with ReLU. It does not include the second input: each plotted SiLU result would still be multiplied by value.

Loading visualization…

Understand the ideas behind this exercise. The Problem tab contains the exact implementation requirements.

Multiply each feature by a learned gate

A gated layer computes two branches from the same input: values to carry forward and coefficients that control their contribution. With a sigmoid gate,

$$g=\sigma(xW_g),\qquad v=xW_v,\qquad h=g\odot v.$$

The symbol odot means elementwise multiplication. The branch shapes must be compatible; this is not a matrix product between g and v.

For values [2, -3] and gates [0.25, 0.8], the output is [0.5, -2.4]. A sigmoid gate near zero suppresses its feature. A gate near one passes most of that feature through. Each feature has its own gate value rather than sharing one decision for the entire example.

SwiGLU uses a different gate function

SwiGLU uses SiLU, a * sigmoid(a), in its gate branch. Unlike sigmoid, SiLU is not limited to [0, 1]. Its output can amplify a value or reverse its sign.

Some tasks ask you to project an input into two branches. Others supply a tensor that already contains both branches and ask you to split the last axis. Follow the Problem tab's input contract and branch order.

When computing derivatives, include both branches: changing the input can change both the gate and the value. In a transformer's feed-forward layer, an output projection usually converts the gated hidden features back to the width expected by the residual connection.

Detecting runtime…

SwiGLU Activation

Your task

Example

Multiply each feature by a learned gate

SwiGLU uses a different gate function

Your code, saved here

Stuck?

PyTorch reference solution