← Mathematical compendium

Published equation contexts

FFNSwiGLU(x)=(Swish1(xW1)⊙xW3) W2,Swish1(z)=z⋅σ(z)\mathrm{FFN}_{\mathrm{SwiGLU}}(x) = \big(\mathrm{Swish}_1(xW_1) \odot xW_3\big)\,W_2, \qquad \mathrm{Swish}_1(z) = z \cdot \sigma(z)

Why this formula appears here

Shazeer’s paper on gated linear unit variants tested several replacements for the standard ReLU or GELU feed-forward block, in which two linear projections are combined so that one, passed through a nonlinearity, gates the other [ 6 ] . The variant that stuck across the field, and that Llama adopts in every generation, uses the Swish (SiLU) nonlinearity as the gate: FFNSwiGLU(x)=(Swish1(xW1)⊙xW3) W2,Swish1(z)=z⋅σ(z)\mathrm{FFN}_{\mathrm{SwiGLU}}(x) = \big(\mathrm{Swish}_1(xW_1) \odot xW_3\big)\,W_2, \qquad \mathrm{Swish}_1(z) = z \cdot \sigma(z). Two independent linear projections of the residual stream are formed; one is passed through the gate and multiplied element-wise ( ⊙\odot ) against the other; the product is projected back down. Shazeer’s own results reported the gated variants outperforming the plain ReLU feed-forward block across the…

Read the full article-specific guide →

Read the representative guide

How to interpret it

Read it with the definitions, units, and assumptions supplied by the article.

Research cited beside this formula

Learn the mathematics behind it

Published contexts (1)

A symbol can carry a different meaning in another article. Each occurrence keeps its own guide and term definitions.

FFNSwiGLU(x)=(Swish1(xW1)⊙xW3) W2,Swish1(z)=z⋅σ(z).\mathrm{FFN}_{\mathrm{SwiGLU}}(x) = \big(\mathrm{Swish}_1(xW_1) \odot xW_3\big)\,W_2, \qquad \mathrm{Swish}_1(z) = z \cdot \sigma(z).

Equation 16 · Open Models

How Llama's Architecture Actually Works, Generation by Generation

This equation states an equality: the expressions on both sides have the same value under the article’s assumptions.

Shazeer’s paper on gated linear unit variants tested several replacements for the standard ReLU or GELU feed-forward block, in which two linear projections are combined so that one, passed through a nonlinearity, gates the other [ 6 ] . The variant that stuck across the field, and that Llama adopts in every generation, uses the Swish (SiLU) nonlinearity as the gate: FFNSwiGLU(x)=(Swish1(xW1)⊙xW3) W2,Swish1(z)=z⋅σ(z)\mathrm{FFN}_{\mathrm{SwiGLU}}(x) = \big(\mathrm{Swish}_1(xW_1) \odot xW_3\big)\,W_2, \qquad \mathrm{Swish}_1(z) = z \cdot \sigma(z). Two independent linear projections of the residual stream are formed; one is passed through the gate and multiplied element-wise ( ⊙\odot ) against the other; the product is projected back down. Shazeer’s own results reported the gated variants outperforming the plain ReLU feed-forward block across the…

Equation guide → · Article →