Helpful Reminders From the (Recent) Past

While Prof. Lawless' (Ill.) essay, What Empirical Legal Scholars Do Best, is over one decade old, it is surviving the test of time well. At its core, the essay advances the claim that one comparative advantage for legally-trained ELS scholars is that they can more easily lever "in-depth knowledge of fine-grained institutional detail that can unlock patterns that otherwise might remain hidden." At the same time, Bob is also quick (and correct) to note that formal legal training is far from a necessary precondition for generating the helpful in-depth knowledge base. Relatedly, even if legal scholars benefit from comparative advantage when it comes to studying legal institutions or doctrines, many law-trained scholars (those without quantitative Ph.D.s) must overcome other traditional comparative disadvantages, typically involving methodology and empirical execution. The essay's final claim also resonates: "empirical legal scholars should be developing new methodologies that are particularly suited to the study of the legal system."

Even if the essay reminds many of what is already well-understood, periodic reminders can be helpful as well.

An Updated (v.2.2) Stata Command for Structural Breaks in Time Series and Panel Data

In time series or panel data structural breaks (or change points) in relations between (or among) key variables can occur. If they do occur, researchers need to be aware of and adjust to such breaks. To make an obvious point clear, the applications to ELS are significant (e.g., Financial Crisis, Brexit Referendum, COVID19, to name but a few).

An updated version (v.2.2) of a user-written command, xtbreak, was recently released to the Stata platform that helps estimate and test for known and unknown structural breaks. For known breakpoints, xtbreak can assess whether the break occurs at a specific point in time. For unknown breaks, xtbreak tests three different hypotheses. First, no break against the alternative of s breaks; second, no breaks against a lower and upper limit of breaks; third, the null of s breaks against the alternative of one more break (s+1).

For those interested, information on the xtbreak program (v.2.2), as well as download information, is here. Additional info accompanied by a brief tutorial is found here.

More Post-Estimation Commands: Contrast

For those who include factor variables on the right-side of a regression equation, a more robust set of post-estimation commands in Stata unlocks additional information on, e.g., how various factor variable outcomes differ from, e.g., the factor variable's designated reference category. Stata's contrast command provides a set of contrast operators that make it easy to specify named contrasts such as reference-level contrasts, adjacent contrasts, Helmert contrasts, and orthogonal polynomial contrasts. For anyone interested, a Stata instructional video walks you through the basics (with a helpful on-line example, here) and additional information in included at UCLA's resource (click here).

Selection Effects: One Cautionary Tale

While the specific context discussed below is not law-related, per se, the degree to which it illustrates the potentially corrosive effects of selection effects remains germane to ELS scholars. Indeed, in much (if not most) of research on legal systems or institutions, empirical or other, it remains difficult to over-emphasize the threat posed by selection effects.

Recent media and scholarly attention has focused on a (so-called) "Mississippi education miracle." The purported "miracle" involves National Assessment of Educational Progress (NAEP) reading test score results for Mississippi 4th graders. Notably (and, perhaps, incredibly), between 2013 and 2024 Mississippi's ranking moved from 49th to a tie for 8th place among 53 US states and territories. From a purely psychometric perspective such a dramatic turnaround clearly marks a sharp deviation from what one would expect given the "laws of nature" as well as more than a century of empirical experience in the education setting.

While most of the initial public and scholarly attention was celebratory, more recent and emerging attention evidences more skepticism and focuses on the likely influence of a key change in Mississippi education policy. Specifically, since 2013 (when Mississippi's test score ascent began), only those third-graders who demonstrated acceptable reading skills were permitted to progress to the 4th grade and sit for the NAEP tests in Mississippi. Thus, Mississippi selected an outcome variable to help measure its education progress (NAEP reading score) and then instituted a compound treatment (deciding which 3rd grades students would move onto 4th grade) on the basis of that outcome variable (a student's reading skills).

While the entire episode is a bit more complicated (click here for a lengthy description), as Andrew Gelman observes, there are "lots of moving parts." Complications notwithstanding, one perspective (Gelman's) is that: "On statistical grounds, it would seem undeniable that some large chunk of the improved test scores in Mississippi come from the selection effect of delaying the students who were going to perform the worst, but it seems hard to put a number on this."

Post-Estimation Tests of Competing (or Alternative) Models

In various literatures I'm seeing a palpable trend of reporting results from various competing models, alternative specifications, or robustness checks, in a single table. This trend, in turn, helps motivate another, related emerging trend of reporting results from various postestimation tests designed to facilitate comparisons between or among competing models for overall efficacy (or balance between goodness-of-fit and complexity).

One popular test includes the Akaike Information Criterion ("AIC") which provides one perspective on a model's performance. A brand new Stata video (click here) provides a quick-and-easy demonstration on how to generate AIC (and BIC) results in Stata (using the estat ic Stata command).

Trusts, Litigation & Judicial Engagement

Many wealth transfers occur privately and through various trust instruments. For many, trusts are the legal instruments of choice and have persisted as the "centerpiece of professionally drafted estate plans since the 1960s" partly as they are thought to "cut courts out of the loop." Conventional wisdom implies that judicial intervention and engagement are required only if litigation emerges.

A recent paper by Christopher J. Ryan, Jr., et al., Uncontested Trusts in Courts, notes an anomaly. Specifically, their study of trust filings (N=1,431 cases; N=1,660 court petitions) in the San Francisco Superior Court over a seven-year period (Jan. 1, 2014, through December 31, 2020), finds that (and the figure reprinted below illustrates) 68 percent of these cases in the judicial system and 61 percent of the petitions involved uncontested (or non-litigated) trusts.

According to the paper, explanations for why this is so include "human error, changed circumstances, and the need to protect vulnerable parties." To be sure, and as the authors duly note, any such study restricted to a single jurisdiction and a relatively small time frame invites obvious generalizability concerns. Limitations aside, the findings may inform conventional wisdom about trust instruments and better instruct how to keep them out of courts.

Experimental Evidence of Type I and Type II Error Tradeoffs

In the criminal law context, one central tradeoff incident to the "beyond all reasonable doubt" threshold involves a core tradeoff between Type I (wrongful conviction) and II (wrongful acquittal) errors. Helping frame conventional wisdom surrounding this tradeoff includes Blackstone's admonition that “it is better that ten guilty persons escape than that one innocent suffer."

Much of the experimental research on this topic, according to a recent paper by Stanton Hudja (IIT-business) et al., Errors in the Pursuit of  Justice: An Experimental Study of Type I and Type II Tradeoffs, focuses on placing participants within the legal "beyond a reasonable doubt" standard. Doing so, however, and as the papers notes, "limits the ability to observe how people make justice-related decisions when left to their own judgment." This study, by contrast, explores how people make such Type I and II tradeoffs outside of the "beyond all reasonable doubt" frame.

To do so, the authors analyze how participants allocated a fixed number of points to "toward reducing the likelihood of either error, revealing their implicit priorities over fairness and punishment. By removing legal thresholds or normative framing, our design isolates participants’ underlying preferences over judicial outcomes.” Respondents in this study include university students (in-person) as well as an online general population sample.

Standard external validity limitations notwithstanding, the paper's main findings include that "most individuals prioritize reducing Type I error over Type II error, without legal or moral instruction." In addition, the results also reflect "meaningful heterogeneity: some participants focus on minimizing Type II error, while others seek a balance between the two." An excerpted abstract follows.

"... Across all treatments, participants consistently prioritize reducing Type I error, without the “beyond a reasonable doubt” standard. Resource effectiveness significantly influences allocation behavior, and we observe systematic demographic differences: men allocate more resources to reducing Type I error than women. These findings shed light on how individuals approach moral trade-offs in legal contexts and offer implications for institutional design and public support for criminal justice policies."