The Test-Optional Puzzle

Even prior to the Covid-19 pandemic, there was a trend in US colleges making standardized test scores optional for undergraduate applicants. In 2019, one third of the 900+ colleges that accepted the Common Application did not require test scores.

Colleges often state that the purpose of test-optional policies is to help enroll a more diverse class. They argue that de-emphasizing tests avoids disadvantaging students groups—including underrepresented minorities—who score lower on standardized tests. For instance, when University of Michigan announced in February 2024 that it would adopt a test-optional policy for the indefinite future, a vice provost stated: “Our commitment today to a test-optional policy for undergraduate admissions demonstrates our focus on providing access to high-achieving students from all backgrounds.”

We find this argument puzzling. Regardless of a (rational) economic agent’s objectives, more information should help their decision making. In particular, colleges can choose how much weight to put on tests, and how to adjust test scores for students from different circumstances and backgrounds. Wouldn’t a college be better off observing students’ scores and deciding how to use them, rather than letting some students not submit scores?

This article formalizes the point via a simple “impossibility result”: under certain assumptions, a test-mandatory policy is always (weakly) better for the college. In fact, we show that a test-mandatory college can replicate any outcome it could obtain under test optional. Our argument will be straightforward to an economic theorist, as it has the flavor of a “revelation principle” familiar from mechanism design. But we believe there is value in articulating a set of assumptions that assure the result in the present context. In particular, the result holds even when score distributions differ across student groups, students can exert hidden effort towards improving their scores, and when effort costs are unobservably heterogeneous across students.

One interpretation of our impossibility result is that test-optional policies are a mistake (pandemics notwithstanding). Indeed, some colleges have reverted to test mandatory after the Covid-19 pandemic. Another interpretation is that the result sheds light on what is needed to rationalize such policies. We discuss a few such considerations in the latter part of the paper.

1 The Impossibility Result

For clarity, we formulate the problem as one of a single student applying to a single college. The logic extends to many students applying to many colleges.

Model.

We are interested in three alternative testing regimes: test mandatory, test optional, and test blind. Under test mandatory, if the student applies to the college, they must submit their standardized test score. Under test optional, the student chooses whether to submit their score. Under test blind, the student cannot submit their score. Given a testing regime, the college publicly commits to an admission policy, which—as detailed below—consists both of a message space and a mapping from everything it observes about the student to a probability of admission.

Given a testing regime and admission policy, the student faces the following decision problem:

1. The student’s exogenous characteristics, denoted by $x\equiv (z,q)$, are stochastically realized. We assume the student observes only the component $z$ while the college does not observe either component. However, the college will obtain a signal as described in point (5) below, which may reveal components of $x$.

   We think of $z$ itself as having both “public” and “private” characteristics. That is, some student features will be seen by the college in the student’s application: GPA and other measures of classroom performance, extra-curricular achievements, legacy status, etc. Others are never directly observed by the college: family wealth or aspects of the student’s interests and upbringing, say. Features such as race or socioeconomic status might or might not be seen by the college, depending on what information it collects on its application.

   We view the variable $q$ as quality/ability, which is not directly observable to either the college or the student, although each may assess that through correlation with other variables they do observe.

2. The student chooses effort $e$, which is not directly observed by the college. We think of $e$ as effort towards preparation for a standardized test, although it can also capture actions that affect other aspects of a student’s application.

3. The student privately obtains a test score $t$, which can stochastically depend on the characteristics $x$ and effort $e$.

4. The student chooses whether to apply to the college, $a\in \{0,1\}$, where $1$ denotes applying. If the student applies, then in the test-optional regime, the student chooses whether to disclose their test score $t$. The test score is automatically disclosed under test mandatory, and cannot be disclosed under test blind. The student may also send the college a supplementary “cheap talk” or payoff-irrelevant message $m$; the set of available messages is chosen by the college as part of its admission policy. We think of $m$ as capturing responses to application questions concerning “soft” information.

5. If the student applies, the college observes a stochastic signal $h$—which we refer to as a “holistic” signal—that can depend on the preceding exogenous and endogenous variables ($x$, $e$, $t$, and $m$). We think of $h$ as capturing everything the college observes about the student except for the test score and the student’s supplementary message. Among other things, $h$ can subsume recommendation letters, the college’s assessment of the student’s match, etc.

6. Finally, if the student applies, admission is determined by the college’s admission policy, which maps $h$, $t$ (if disclosed), and $m$ into an admission probability.

Let $o$ be a binary variable indicating whether the student is admitted by the college. The student’s preferences are given by two utility functions: one if they apply, which can depend on $(x,e,t,h,o)$, and another if they don’t, which can depend on $(x,e,t)$. The college’s utility is normalized to $0$ if the student does not apply or is not admitted, and is some function of $(x,e,t,h)$ if the student is admitted.

These preference assumptions rule out either party’s utility depending directly on the testing regime or admission policy, and on whether the student discloses their score or which supplementary message they send. There is also, implicitly, no cost to generating a (possibly very low) test score. Otherwise, our formulation is quite general. In particular, the student’s cost of improving their test score (i.e., exerting effort $e$) and the score distribution conditional on effort can vary with the student’s characteristics ($x$), in ways that are either observable or unobservable to the college. On the college’s side, factors like preferences over yield (how likely the student is to matriculate) can be subsumed by making yield a component of the holistic signal $h$.

We assume the student knows all the relevant stochastic distributions (i.e., of $x$, $t$, and $h$), and that none of those distributions depends on either the testing regime or the admission policy. Given any regime and policy, the student best responds, i.e., chooses their (possibly randomized) actions at steps (2) and (4) in the sequence above to maximize their expected utility.

Result.

We say that an admission policy ${\alpha }^{\prime }$ (in some testing regime) replicates another admission policy $\alpha$ (in a possibly different regime) if, roughly speaking, optimal student behavior results in the same payoff-relevant outcomes under both policies. More precisely, ${\alpha }^{\prime }$ replicates $\alpha$ if:

1. the set of student-optimal effort and application decision pairs $(e,a)$ are the same under both policies; and

2. given any optimal $(e,a)$, and any optimal supplementary message $m$ under $\alpha$, there is an optimal message $m^{\prime }$ under ${\alpha }^{\prime }$ such that for any realizations of the student characteristics $x$, test score $t$, and holistic signal $h$, the student’s probability of admission under both policies is the same.

If every admission policy under one testing regime can be replicated by a policy under another regime, then the college is (weakly) better off in the latter regime.

Proposition 1.

For any test-optional admission policy, there is a replicating test-mandatory policy. For any test-blind admission policy, there is a replicating test-optional policy.

• We show how any test-optional admission policy can be replicated by some test-mandatory policy. An analogous argument, which we omit, shows that test optional can replicate test blind.

  Take any test-optional admission policy with a message space $M$ and admission probability function $\alpha (h,\widehat{t},m)$. Here, $\widehat{t}$ is the test score $t$ if submitted and null, denoted $\phi$, otherwise. Consider a test-mandatory policy with message space $M^{\prime }:=M\times \{0,1\}$ and admission probability function ${\alpha }^{\prime }(h,t,m^{\prime })$ given by ${\alpha }^{\prime }(h,t,(m,0)):=\alpha (h,\phi ,m)$ and ${\alpha }^{\prime }(h,t,(m,1)):=\alpha (h,t,m)$. In other words, the college sets its test-mandatory admission probability equal to what a student with the same holistic signal, test score, and supplementary message would have gotten under the test-optional policy. The nuance is that the student is now also asked to indicate whether they would have submitted their score under test optional. Formally, this is the second component of their test-mandatory message $m^{\prime }$, with a $0$ indicating nonsubmission under test optional and $1$ indicating submission. The college’s test-mandatory policy mimics the corresponding test-optional outcome.

  Thus defined, the test-mandatory policy ensures that the incentives for the student to choose effort, to apply, to choose the first component of their message $m^{\prime }$, and to (indicate whether they would) submit a test score are identical to those under the test-optional policy. It is thus a best response for the student to act the same as under test optional, and the outcome is replicated. ∎

Note that a test-mandatory admissions policy might also be replicated by a test-optional policy that simply rejects any student who does not submit a test score. We do not consider this interesting because always rejecting students who do not submit scores defeats the spirit of test optional, as it is de facto test mandatory.

Although Proposition 1 only implies that the college cannot do worse under test mandatory than test optional (and under test optional rather than test blind), one would expect that observing more information about the test score would allow the college to generally do strictly better (so long as the test-optional policy is not replicating test mandatory).

We stress that we do not interpret Proposition 1 as implying that colleges in the real world cannot benefit from going test optional or test blind. Rather, the model and result point us to the assumptions that must be violated for a college to in fact benefit from not observing test scores.

2 Ways Out of the Puzzle

We now turn to discussing how breaking some of Proposition 1’s underlying assumptions might—or might not—lead the college to prefer test optional to test mandatory.

Lack of Commitment.

Suppose the college lacks commitment power, and instead admits students according to what it finds ex-post optimal given the information provided. We expect that under reasonable monotonicity assumptions, the college still cannot do better under test optional than test mandatory. The logic is now quite different from that of Proposition 1, however. The issue now is that a test-optional policy unravels, as in classic voluntary-disclosure models (e.g., Milgrom, 1981). Despite the college being nominally test optional, all students end up submitting their scores because not submitting will, in equilibrium, be met with the skepticism of a low score and hurt admission. Such unraveling suggests that a lack of college commitment power is unlikely, on its own, to explain why colleges might go test optional.

Additional Costs.

We allowed for the students to have arbitrary type-dependent costs of studying for the test and of applying to the college. However, we did not allow for a direct cost of sitting for the test—formally, there was no (lower cost) option to not generate a test score—nor of submitting a test score. Adding either of these costs breaks the replication argument and can flip the result. A student who applies can avoid these costs only if the college is test optional or test blind. In the presence of such costs, colleges potentially face a genuine tradeoff: requiring test scores deters applications while yielding more information about students who do apply (cf.  Garg et al., 2021).

There is, of course, a large cost of sitting for the test during a pandemic—even infinite, when test centers are shut down. Our view is that this cost is not otherwise particularly large. (Let us reiterate that this test-sitting cost would be separate from any costs of studying and preparing for test, which we view as significant but are already part of the model.) That said, students may still perceive these costs as significant (cf.  Goodman, 2016).

Non-equilibrium Behavior.

The impossibility result can fail if students don’t follow our predictions of equilibrium behavior. Students may make different application or test-preparation decisions when facing a test-optional college rather than a test-mandatory college with the same acceptance probabilities. For instance, colleges would certainly want to switch from test mandatory to test optional if many students happened to follow the behavioral rule that they will not apply to test-mandatory colleges. Even a student who plans on taking the test and submitting their score to test-optional colleges might, for reasons of principle, be unwilling to apply to a test-mandatory college.

Signaling.

In practice, students may try to infer a college’s values from its admission policy. Being test optional could then be a credible signal that the college cares about diversity or about aspects of a student’s character that can’t be measured by tests. Likewise, if students do not blindly trust admission policy pronouncements, then test optional could serve as a credible signal that the college will not put much weight on test scores.

Constraints on Admission Rules.

Our replication argument assumes that the college is able to choose any admission policy. If the college is instead constrained, the impossibility result could fail. As an extreme case, imagine that the college has no flexibility at all: it is required to evaluate students with a test score by one rule, and students without a test score by another rule. If the admission rule for students with test scores were to put too much weight on tests, the college might very well prefer not to see tests at all. Such an exogenous admission rule might be imposed by the government or by stakeholders with different preferences from the college.

The approach in Dessein et al. (2024) of college decisions under social pressure is a milder form of constraints on admissions. In that model, the college is free to choose any admission policy it wants, but it faces costs of making admission decisions that a third party disagrees with. Not seeing scores allows the college to reduce disagreement, which can outweigh the loss of information.

Agency Issues.

Admission policies may also be limited by a college’s admissions officers. These officers might put more weight on test scores than the college seeks, owing to differences in preferences or beliefs. The salience of test scores may also subconsciously influence how officers process, interpret, or attend to other more subjective parts of an application. Finally, officers may put excess weight on test scores due to incentives and moral hazard: officers may only find it worthwhile to conduct a costly holistic assessment when they lose access to test scores.

Gaming Rankings.

Finally, some commentators have suggested that colleges may go test optional simply to report higher average (submitted) standardized test scores, which mechanically improves their rankings from organizations like the U.S. News. Conlin et al. (2013) provide evidence of strategic admission decisions by test-optional colleges to manipulate their rankings.