Friday, October 9, 2026

Power and Sample Size: The Under-Appreciated Influence of "Noisy" Data

While problems with under-powered studies are well-understood, one under-appreciated aspect involves data noise. This subtle wrinkle is discussed in a recent (and helpful) post by Andrew Gelman (click here). Gelman's main point is that, when it comes to addressing under-powered studies, "the real problem is noise, not N."

Gelman's point is well-taken, if oft-ignored. For many researchers, the go-to remedy for under-powered studies is to simply increase sample size (N). Gelman's post underscores that "[n]oise (variation in effect size, variation in outcomes, and measurement error) is more of an issue than sample size because lots of bad studies are so noisy that, to get a large enough sample size to detect a reliable effect, you’d need N = a zillion, and at that point you’d be averaging over such a wide range of conditions that it’s not clear what you’re estimating anyway." Thus, possible solutions to problems attributable to under-powered studies benefit from multivariate rather than univariate approaches.