← All symbols

Symbol in context

W3W_3

This entry links every published equation using this exact notation. Its meaning may change between equations.

Used in 1 equation

FFNSwiGLU(x)=(Swish1(xW1)⊙xW3) W2,Swish1(z)=z⋅σ(z).\mathrm{FFN}_{\mathrm{SwiGLU}}(x) = \big(\mathrm{Swish}_1(xW_1) \odot xW_3\big)\,W_2, \qquad \mathrm{Swish}_1(z) = z \cdot \sigma(z).

How Llama's Architecture Actually Works, Generation by Generation · Equation 16

This equation states an equality: the expressions on both sides have the same value under the article’s assumptions.

Equation guide → · This term → · Article →