← All parts of this equation

Equation 16 · Part 2 · How Llama's Architecture Actually Works, Generation by Generation

Symbol W_1

FFNSwiGLU(x)=(Swish1(xW1)⊙xW3) W2,Swish1(z)=z⋅σ(z).\mathrm{FFN}_{\mathrm{SwiGLU}}(x) = \big(\mathrm{Swish}_1(xW_1) \odot xW_3\big)\,W_2, \qquad \mathrm{Swish}_1(z) = z \cdot \sigma(z).
W1W_1

What this part means

W1W_1 is one factor in the product that computes the quantity on the left.

Its job in the formula

W1W_1 is one factor in the product that computes the quantity on the left.

The passage around this formula

Shazeer’s paper on gated linear unit variants tested several replacements for the standard ReLU or GELU feed-forward block, in which two linear projections are combined so that one, passed through a nonlinearity, gates the other [ 6 ] . The variant that stuck across the field, and that Llama adopts in every generation, uses the Swish (SiLU) nonlinearity as the gate: FFNSwiGLU(x)=(Swish1(xW1)⊙xW3) W2,Swish1(z)=z⋅σ(z)\mathrm{FFN}_{\mathrm{SwiGLU}}(x) = \big(\mathrm{Swish}_1(xW_1) \odot xW_3\big)\,W_2, \qquad \mathrm{Swish}_1(z) = z \cdot \sigma(z). Two independent linear projections of the residual stream are formed; one is passed through the gate and multiplied element-wise ( ⊙\odot ) against the other; the product is projected back down. Shazeer’s own results reported the gated variants outperforming the plain ReLU feed-forward block across the…

Read this part in the article →

Learn the underlying idea

A subscript is a label attached below a symbol. It often selects a time step, component, category, or member of a sequence.

Open the illustrated subscripts: which member of a family? guide →

See this notation across published equations →

Sources cited in the surrounding passage

These citations provide research context; check each source for the exact claim it supports.