SwiGLU Activation

SwiGLU combines two projections of the same input. One projection goes through SiLU (also called Swish); its result multiplies the other projection element by element. This exercise starts after the projections are computed.

Your task

Implement swiglu(gate, value) for two floating-point arrays of the same shape (batch, d), with positive dimensions. Return the same shape without changing either input:

silu(gate) = gate * sigmoid(gate)
output = silu(gate) * value

Both multiplications are elementwise. Do not split either input, add weights or biases, or apply a final projection. The two arguments already contain the gate and value projections. Use a stable sigmoid or a framework SiLU operation, including for large negative gate values.

Example

For gate = [[1, 2]] and value = [[3, 4]], SiLU gives approximately [[0.731059, 1.761594]]. Multiplying by value gives [[2.193176, 7.046377]].

A zero gate produces zero output regardless of value. Negative gates can produce negative SiLU values, so this is not a multiplier constrained to [0, 1]. For example, gate -1 and value 2 give approximately -0.537883. If value is negative, the final sign can change again.

The chart shows the SiLU part and compares it with ReLU. It does not include the second input: each plotted SiLU result would still be multiplied by value.