← All parts of this equation

Equation 16 · Part 8 · How Llama's Architecture Actually Works, Generation by Generation

multiplication

FFNSwiGLU(x)=(Swish1(xW1)⊙xW3) W2,Swish1(z)=z⋅σ(z).\mathrm{FFN}_{\mathrm{SwiGLU}}(x) = \big(\mathrm{Swish}_1(xW_1) \odot xW_3\big)\,W_2, \qquad \mathrm{Swish}_1(z) = z \cdot \sigma(z).
multiplication

What this part means

Multiply the quantities on either side.

Its job in the formula

Multiply the quantities on either side.

The passage around this formula

Shazeer’s paper on gated linear unit variants tested several replacements for the standard ReLU or GELU feed-forward block, in which two linear projections are combined so that one, passed through a nonlinearity, gates the other [ 6 ] . The variant that stuck across the field, and that Llama adopts in every generation, uses the Swish (SiLU) nonlinearity as the gate: FFNSwiGLU(x)=(Swish1(xW1)⊙xW3) W2,Swish1(z)=z⋅σ(z)\mathrm{FFN}_{\mathrm{SwiGLU}}(x) = \big(\mathrm{Swish}_1(xW_1) \odot xW_3\big)\,W_2, \qquad \mathrm{Swish}_1(z) = z \cdot \sigma(z). Two independent linear projections of the residual stream are formed; one is passed through the gate and multiplied element-wise ( ⊙\odot ) against the other; the product is projected back down. Shazeer’s own results reported the gated variants outperforming the plain ReLU feed-forward block across the…

Read this part in the article →

Learn the underlying idea

Multiplication scales one quantity by another. A dot, a cross, or adjacent symbols can indicate a product.

Open the illustrated multiplication: combining factors guide →

Sources cited in the surrounding passage

These citations provide research context; check each source for the exact claim it supports.