
Two currently accessible figures in Jonathan Fulton’s “AB Testing 101” sit awkwardly together. The first defines the treatment and control means, labels and as sample standard deviations, and gives and as the group counts. The next figure prints a variance for relative lift without either sample-count divisor.
That matters because a sample standard deviation describes variation among observations; it is not the standard error of the sample mean. Under the ordinary independent-groups model, the variance of each group mean is estimated by its sample variance divided by that group’s count.
For population lift , the sample estimator is . Evaluating the first-order delta-method gradient at those sample means gives . Substituting the sample-mean variances produces:
This plug-in approximation assumes independent treatment and control means, an unclustered sample-mean variance model, and a nonzero control mean. It can be unreliable when the control mean is near zero or the ratio is poorly approximated by a normal distribution. Dependence adds a covariance contribution, ; clustering, weighting and repeated measures also require their own variance calculations.
Where the displayed factored expression is defined ( and ), if both groups have the same count , its variance is times the first-order estimate implied by the preceding definitions. For positive variances, the corresponding standard-error factor is . With unequal group counts, there is no single shared factor that repairs both terms.
The same article figure could be read as using standard errors in place of standard deviations, which would remove the count divisors. That reading conflicts with the labels in the preceding figure. Eppo’s currently accessible documentation provides a useful current comparator: its relative-lift calculation includes both sample-size terms. That comparison does not establish what an earlier version of Eppo’s documentation said.
The finding is limited to the figures currently served with Fulton’s article. The separately archived course PDF was not recovered or authenticated, so this review says nothing about its contents, wording or pagination. It does not establish why the displayed figure takes this form, whether any experiment or implementation used it, or what happened downstream. No data, code, experiment, estimator implementation, novelty claim or new statistical theorem is presented.
The pinned Research Artifact Ledger review records the source boundary, derivation and independent AI challenge scope. The currently accessible article and its notation figure and variance figure are linked rather than reproduced. The current Eppo statistical documentation is a present-day comparator only.
AI-use disclosure
OpenAI GPT models, accessed through Codex, assisted with source review, the standard delta-method derivation, claim checking, and drafting and editorial revision. Two AI public-source challenge passes checked the figure transcription, calculation and scope; serving-model telemetry is unavailable. The featured illustration was AI-generated and is conceptual, not data. No data, code, experiment, implementation or downstream experiment was reviewed. No human statistical specialist review or peer review was conducted for this article. The author is responsible for the work.



Loading published comments…