Symbol d
d is a part of this expression. Its role is fixed by the surrounding article and by the operations shown in the formula.
Read this term in its guide →Published equation contexts
Why this formula appears here
Two independent linear projections of the residual stream are formed; one is passed through the gate and multiplied element-wise ( ) against the other; the product is projected back down. Shazeer’s own results reported the gated variants outperforming the plain ReLU feed-forward block across the language modelling and fine-tuning tasks tested, at matched parameter and compute budgets [ 6 ] . Because the gate consumes an extra weight matrix, Llama’s feed-forward hidden dimension is set below the naive 4d multiplier used in the original transformer, to hold total parameters roughly constant against a non-gated block of the same width — an accounting detail visible in the hyperparameter…
Read the representative guide
How to interpret it
Read this expression with the definitions, units, and assumptions supplied by the article.
Research cited beside this formula
Published contexts (1)
A symbol can carry a different meaning in another article. Each occurrence keeps its own guide and term definitions.
Equation 18 · Open Models
How Llama's Architecture Actually Works, Generation by Generation
This mathematical expression combines the displayed quantities; its precise role follows from the surrounding article text.
Two independent linear projections of the residual stream are formed; one is passed through the gate and multiplied element-wise ( ) against the other; the product is projected back down. Shazeer’s own results reported the gated variants outperforming the plain ReLU feed-forward block across the language modelling and fine-tuning tasks tested, at matched parameter and compute budgets [ 6 ] . Because the gate consumes an extra weight matrix, Llama’s feed-forward hidden dimension is set below the naive 4d multiplier used in the original transformer, to hold total parameters roughly constant against a non-gated block of the same width — an accounting detail visible in the hyperparameter…
Equation guide → · Article →