=

The passage around this formula

Here is the modelling error to avoid. Suppose a request passes n independent detection layers, each of which fails to catch a given malicious input with probability $p_i$ . It is tempting to write the bypass probability of the stack as $P_{\mathrm{bypass}} = \prod_{i=1}^{n} p_i$. and conclude that four layers at ten percent leakage each give one bypass in ten thousand. That number is an artefact of the independence assumption, and the assumption is the weakest part of the model. The layers are typically built from the same model family, trained on overlapping data, and sensitive to the same features of an input; their failures are positively correlated, so the true joint failure probability is larger than the…

Read this part in the article →