← All parts of this equation

Equation 16 · Part 1 · How Llama's Architecture Actually Works, Generation by Generation

Symbol x

FFNSwiGLU(x)=(Swish1(xW1)⊙xW3) W2,Swish1(z)=z⋅σ(z).\mathrm{FFN}_{\mathrm{SwiGLU}}(x) = \big(\mathrm{Swish}_1(xW_1) \odot xW_3\big)\,W_2, \qquad \mathrm{Swish}_1(z) = z \cdot \sigma(z).
xx

What this part means

x is an argument of the function-like quantity on the left; its role is set by that function’s stated inputs.

Its job in the formula

x is an argument of the function-like quantity on the left; its role is set by that function’s stated inputs.

The passage around this formula

Shazeer’s paper on gated linear unit variants tested several replacements for the standard ReLU or GELU feed-forward block, in which two linear projections are combined so that one, passed through a nonlinearity, gates the other [ 6 ] . The variant that stuck across the field, and that Llama adopts in every generation, uses the Swish (SiLU) nonlinearity as the gate: FFNSwiGLU(x)=(Swish1(xW1)⊙xW3) W2,Swish1(z)=z⋅σ(z)\mathrm{FFN}_{\mathrm{SwiGLU}}(x) = \big(\mathrm{Swish}_1(xW_1) \odot xW_3\big)\,W_2, \qquad \mathrm{Swish}_1(z) = z \cdot \sigma(z). Two independent linear projections of the residual stream are formed; one is passed through the gate and multiplied element-wise ( ⊙\odot ) against the other; the product is projected back down. Shazeer’s own results reported the gated variants outperforming the plain ReLU feed-forward block across the…

Read this part in the article →

Learn the underlying idea

A variable is a named place for a value. Its letter is a local label: x can mean position in one formula and a data point in another.

Open the illustrated variables: a letter stands for a value guide →

See this notation across published equations →

Sources cited in the surrounding passage

These citations provide research context; check each source for the exact claim it supports.