While the specific context discussed below is not law-related, per se, the degree to which it illustrates the potentially corrosive effects of selection effects remains germane to ELS scholars. Indeed, in much (if not most) of research on legal systems or institutions, empirical or other, it remains difficult to over-emphasize the threat posed by selection effects.
Recent media and scholarly attention has focused on a (so-called) "Mississippi education miracle." The purported "miracle" involves National Assessment of Educational Progress (NAEP) reading test score results for Mississippi 4th graders. Notably (and, perhaps, incredibly), between 2013 and 2024 Mississippi's ranking moved from 49th to a tie for 8th place among 53 US states and territories. From a purely psychometric perspective such a dramatic turnaround clearly marks a sharp deviation from what one would expect given the "laws of nature" as well as more than a century of empirical experience in the education setting.
While most of the initial public and scholarly attention was celebratory, more recent and emerging attention evidences more skepticism and focuses on the likely influence of a key change in Mississippi education policy. Specifically, since 2013 (when Mississippi's test score ascent began), only those third-graders who demonstrated acceptable reading skills were permitted to progress to the 4th grade and sit for the NAEP tests in Mississippi. Thus, Mississippi selected an outcome variable to help measure its education progress (NAEP reading score) and then instituted a compound treatment (deciding which 3rd grades students would move onto 4th grade) on the basis of that outcome variable (a student's reading skills).
While the entire episode is a bit more complicated (click here for a lengthy description), as Andrew Gelman observes, there are "lots of moving parts." Complications notwithstanding, one perspective (Gelman's) is that: "On statistical grounds, it would seem undeniable that some large chunk of the improved test scores in Mississippi come from the selection effect of delaying the students who were going to perform the worst, but it seems hard to put a number on this."