In their efforts to assess whether people harbor implicit biases, psychologists have used a variety of methods.¹⁵ Standing front and center among these methods, however, is the Implicit Association Test (IAT).¹⁶ Developed by a research group led largely by Tony Greenwald, Mahzarin Banaji, and Brian Nosek, the IAT is the product of decades of research on the study of bias and stereotypes¹⁷ and has attracted enormous scholarly and popular attention.¹⁸ More than 4.5 million people have taken the IAT.¹⁹ The test takes different forms, but most commonly, it consists of a computer-based sorting task in which study participants pair words and faces. A typical administration of the “Race IAT” proceeds as follows.²⁰

First, researchers present participants with a computer screen that has the words “white or good” in the upper left-hand corner of the screen and “black or bad” in the upper right. The researchers then inform the participants that one of four types of stimuli will appear in the center of the screen: white people’s faces, black people’s faces, good (positive) words, or bad (negative) words. The researchers then explain that the participants should press a designated key on the left side of the computer when a white face or a good word appears and press a designated key on the right side of the computer when a black face or a bad word appears. Researchers refer to the white/good and black/bad pairings as “stereotype congruent”, because they are consistent with negative stereotypes associated with black Americans.²¹ The participants complete several trials of this first task.

Then, the computer is programmed to switch the spatial location of “good” and “bad” so that the words “white or bad” appear in the upper left-hand corner and “black or good” appear in the upper right. The researchers explain to the participants that they are now supposed to press a designated key on the left side of the keyboard when a white face or a bad word appears and press a designated key on the right side of the keyboard when a black face or a good word appears. Researchers refer to these white/bad and black/good pairings as “stereotype-incongruent”, because they are inconsistent with the negative stereotypes associated with black Americans. The participants then complete several trials of this second task.²²

Researchers have consistently found that white Americans express a strong “white preference” on the IAT.²³ They make this determination by comparing the amount of time it takes respondents to complete the two tasks identified above — that is, their “response latency”.²⁴ Most white Americans complete the first task (in which they sort white and good from black and bad) more quickly than the second (in which they sort black and good from white and bad).²⁵ In other words, most white Americans produce higher response latencies when faced with the stereotype-incongruent pairing (white/bad or black/good) than when faced with the stereotype-congruent pairing (white/good or black/bad).

Researchers have observed a different pattern of implicit biases among black Americans. Black Americans do not exhibit the same white preference that whites express, but neither do they show a mirror-image black preference.²⁶ Rather, black Americans express a much greater variation, with many expressing moderate to strong black preferences that are rarely found in white Americans.²⁷ But some also express white preferences — sometimes even strong ones.²⁸ On average, black Americans express a slight white preference, but the average masks wide variation in response.²⁹ Latinos also express a small white preference. Asian Americans show a white preference that is comparable to but somewhat weaker than that found in white Americans.³⁰

The implications of the research using the IAT are a matter of some debate,³¹ but the cognitive mechanisms underlying the research are clear enough. The white preference arises from well-established mnemonic links. Whites more closely associate white faces with positive words and black faces with negative words than the opposite. Thus, when they complete the white/good versus black/bad trials, they need only make a judgment about whether the stimulus that appears in the middle of the screen is positive or negative. The incongruent association, in contrast, requires that they first judge whether the stimulus is a word or a face and then decide on which side it belongs. Stereotype-incongruent associations interfere with the sorting task in much the same way that the use of green ink can make the word “blue” hard to read.³²

The white preference on the IAT is well-documented among white Americans.³³ Researchers have conducted and published hundreds of academic studies, and several million people have participated in IAT research.³⁴ They have determined that the implicit biases documented through IAT research are not the product of the order in which people undertake the tasks, their handedness, or any other artifact of the experimental method.³⁵ The prevailing wisdom is that IAT scores reveal implicit or unconscious bias.³⁶

Even if implicit bias is as widespread as the IAT studies suggest, it does not necessarily lead to, or explain, racially disparate treatment. Only if researchers can show that implicit bias influences decision makers can we infer that implicit bias is a cause of racial disparities.

Implicit bias, at least as measured by the IAT, appears to correlate with behavior in some settings. In a recent review, Greenwald and his colleagues identified 122 research reports assessing the relationship between IAT scores and observable behaviors;³⁷ of these, 32 involved “White-Black interracial behavior”.³⁸ Across these 24 studies, the researchers found a modest correlation of 0.24 between the implicit bias measures and the observed behaviors tested in the studies.³⁹ This means that implicit bias accounted for roughly 6% of the variation in actual behavior.⁴⁰

Six percent might not sound like much, but a 6% disparity could have an enormous impact on outcomes in the criminal justice system. In a typical year, judges preside over approximately 21 million criminal cases in state courts⁴¹ and 70 000 in federal courts,⁴² many of which involve black defendants. Throughout the processing of these cases, judges make many judgments concerning bail, pretrial motions, evidentiary issues, witness credibility, and so forth. Each of these judgments could be influenced by implicit biases, so the cumulative effect on bottom-line statistics like incarceration rates and sentence length is much larger than one might imagine.⁴³ Furthermore, 6% is only an average. Some judges likely hold extremely strong implicit biases. And some defendants are apt to trigger an unconscious bias to a much greater extent than others.⁴⁴ Even this seemingly small effect might harm tens or even hundreds of thousands of black defendants every year.

Researchers have found, however, that people may have the ability to compensate for the effects of implicit bias.⁴⁵ If they are internally driven or otherwise motivated to suppress their own biases, people can make judgments free from biases,⁴⁶ even implicit ones.⁴⁷ In one recent study,⁴⁸ for example, a team of researchers administered the IAT to a group of physicians and asked them to diagnose and treat a hypothetical patient — identified to some of the physicians as a white man and to others as a black man — based on a description of symptoms.⁴⁹ The researchers found a correlation between IAT scores and treatment; the physicians with higher IAT scores were more likely to offer appropriate treatment to white patients than to black patients diagnosed with the same condition.⁵⁰ But among the 67 physicians who reported some awareness of the purpose of the study, those with higher IAT scores were more likely to recommend the treatment to black patients.⁵¹ In other words, the doctors who were aware of the purpose of the study compensated for their implicit biases when the situation made them sensitive to the risk of behaving — or being observed to behave — in a biased way. “This suggests”, argue the authors, “that implicit bias can be recognised and modulated to counteract its effect on treatment decisions.”⁵²

Jack Glaser and Eric Knowles found similar results in a study using the so-called “Shooter Task”.⁵³ In research of this type, subjects participate in a simulation akin to a video game in which they watch a person on screen pull either a gun or an innocent object, like a wallet, out of his pocket.⁵⁴ If he pulls a gun, the participants are instructed to “shoot” by pushing a button on a joystick; if he pulls a benign object, they are instructed to refrain from shooting.⁵⁵ Researchers have found that most white adults exhibit a “shooter bias” in that they are more likely to shoot a black target — regardless of what object the on-screen target pulls out of his pocket,⁵⁶ and that this effect correlates with a white preference on the IAT.⁵⁷ Glaser and Knowles found in their study, however, that those rare individuals with a white preference on the IAT and who are highly motivated to control prejudice were able to avoid the shooter bias.⁵⁸ In short, “those high in an implicit negative attitude toward prejudice show less influence of implicit stereotypes on automatic discrimination”.⁵⁹

In sum, the research on implicit bias suggests that people exhibit implicit biases, that there is some evidence that implicit bias can influence behavior, and that people can overcome or compensate for implicit biases if properly motivated and if the racial context is made sufficiently salient. Whether and how this research applies to judges and the criminal justice system is an open question and one to which we turn in “The study design”.

We recruited judges to participate in our study at judicial education conferences, as we have in our prior work.⁶⁴ The 133 judges who participated in our study came from three different jurisdictions.⁶⁵ The judges asked us not to identify their jurisdictions,⁶⁶ but we can describe the basic characteristics of each of the three. We recruited 70 judges from a large urban center in the eastern United States.⁶⁷ These 70 judges, who are appointed to the bench for renewable terms, constitute roughly three-quarters of the judges who sit in this jurisdiction. We recruited 45 judges from a large urban center in the western United States.⁶⁸ These 45 judges, who are appointed to the bench but then stand for election, make up roughly half of the judges in their jurisdiction. We recruited our final group of judges at an optional session at a regional conference. These 18 judges, who sit in various towns and cities throughout the state in which the conference was held, are appointed to the bench but are then required to stand for election.⁶⁹

We did not ask the judges to identify themselves by name, but we did ask them to identify their race, gender, exact title, political affiliation, and years of experience on the bench.⁷⁰ Table 1 summarises the demographic information that the judges provided. As Table 1 indicates, our sample of judges, particularly those from the eastern jurisdiction, is fairly diverse, at least in terms of gender and race.

To explore the two questions animating this article — that is, whether judges hold implicit racial biases, and if so, whether those biases produce biased judicial decisions — we designed a multipart study requiring the participating judges to complete computer tasks⁷¹ and then to respond to a paper questionnaire.

We proceeded as follows. We placed in front of each judge a laptop computer and a questionnaire. The computer screen and the front page of the questionnaire introduced the study and asked the judges to await instruction before beginning.⁷² Once the judges were fully assembled, we announced, “Today, we shall ask you to participate actively in your own education”.⁷³

We asked the judges to complete the computer tasks and to respond to the questionnaire according to the instructions provided. We assured the judges that their responses were anonymous and that we had no way of identifying them individually, but we also made clear that participation was entirely voluntary and that any judge who wanted to exclude her results from the study could do so. (Only one judge chose to do so.) We informed the judges that we would compile their cumulative results and share them with the group at the end of the session.

With these important preliminaries out of the way, we then asked the judges to begin the study. The study included a race IAT;⁷⁴ two hypothetical vignettes in which the race of the defendant was not explicitly identified but was subliminally primed; and another hypothetical vignette in which the race of the defendant was made explicit.⁷⁵ The final page of the questionnaire asked judges to provide the basic demographic information identified above.⁷⁶

To measure implicit associations involving race, we gave the judges a computer-based-race IAT comparable to the race IAT given to millions of study participants around the world.⁷⁸ We asked the judges to perform two trials of the IAT, as described above. The first required them to pair white faces with positive words and black faces with negative words. In other words, the first trial required them to select stereotype-congruent pairings. The second required them to pair white faces with negative words and black faces with positive words. In other words, the second trial required them to select stereotype-incongruent pairings.⁷⁹

To determine each judge’s implicit bias score, we performed two calculations. First, we subtracted each judge’s average response latency in the stereotype-congruent round from the stereotype-incongruent round to calculate the IAT measure. This measure reflects the most commonly used scoring method for large samples of data collected on the Internet, and hence allows us to compare judges to ordinary adults.⁸⁰ Second, we constructed a standardised measure consisting of the average difference in response latencies for each judge divided by the standard deviation of that judge’s response latencies in the target rounds. This measure is less commonly reported, but more stable, and produces higher correlations with other behaviors.⁸¹

We found a strong white preference among the white judges, as shown in Table 2. Among the 85 white judges, 74 (or 87.1%) showed a white preference on the IAT. Overall, the white judges performed the stereotype-congruent trial (white/good and black/bad) 216 milliseconds faster than the stereotype-incongruent trial (black/good and white/bad). The black judges, by contrast, demonstrated no clear preference overall. Although 14 of 43 (or 44.2%) showed a white preference, the black judges performed the stereotype-congruent trial (white/good and black/ bad) a mere 26 milliseconds faster than the stereotype-incongruent trial (black/good and white/bad). Comparing the mean IAT scores of the white judges with those of the black judges revealed that the white judges expressed a significantly larger white preference.⁸²

Note:

Positive numbers indicate lower latencies on the white/good versus black/bad round

Because we used a commonly administered version of the IAT, we are able to compare the results of our study to the results of other studies involving ordinary adults. We found that the black judges produced IAT scores comparable to those observed in the sample of black subjects obtained on the Internet.⁸³ The white judges, on the other hand, demonstrated a statistically significantly stronger white preference than that observed among a sample of white subjects obtained on the Internet.⁸⁴ For two reasons, however, this does not necessarily mean that the white judges harbor more intense white preferences than the general population. First, we did not vary the order in which we presented the materials, and this order effect could have led to artificially higher IAT scores.⁸⁵ Second, the judges performed both trials much more slowly than the other adults with whom we are making this comparison, and this, too, could have led to artificially higher IAT scores.⁸⁶ We also suspect that the judges were older, on average, than the Internet sample. To the extent that implicit racial bias is less pronounced among younger people, we would expect the judges to exhibit more implicit bias than the Internet sample.

To assess the impact of implicit bias on judicial decision making, we gave the judges three hypothetical cases: the first involving a juvenile shoplifter, the second involving a juvenile robber, and the third involving a battery. We speculated that the judges might respond differently depending upon whether we made the race of the defendant salient, so in the first two cases, we did not identify the race of the defendant explicitly, but we did so implicitly through a subliminal priming technique described below. In the third case, we made race explicit, informing some of the judges that the defendant was “Caucasian” and others that he was “African American”.⁸⁷ By comparing the judges’ individual IAT scores with their judgments in these hypothetical cases, we are able to assess whether implicit bias correlates with racially disparate outcomes in court.

Our research supports three conclusions. First, judges, like the rest of us, carry implicit biases concerning race. Second, these implicit biases can affect judges’ judgment, at least in contexts where judges are unaware of a need to monitor their decisions for racial bias. Third, and conversely, when judges are aware of a need to monitor their own responses for the influence of implicit racial biases, and are motivated to suppress that bias, they appear able to do so.

Our first conclusion was perhaps the most predictable, though it is still troubling. Given the large number of Americans who have taken the IAT, and given the frequency with which white Americans display at least a moderate automatic preference for white over black, it would have been surprising if white judges had failed to exhibit the same automatic preference. Similarly, the black judges carry a more diverse array of implicit biases, just like black adults generally: some exhibit a white preference just like the white judges; others exhibit no preference; and some exhibit a black preference. Overall, like adults, most of the judges — white and black — showed a moderate-to-large degree of implicit bias in one direction or the other. If ordinary adults carry a “bigot in the brain”, as one recent article put it,¹²² then our data suggest that an invidious homunculus might reside in the heads of most judges in the United States, with the potential to produce racially biased distortions in the administration of justice.

It is worth noting, however, that the research on so-called “chronic egalitarians” suggests that this result was not inevitable. Some whites with longstanding and intense personal commitments to eradicating bias in themselves — chronic egalitarians — do not exhibit the preference for whites over blacks on the IAT that most white adults show.¹²³ Despite their professional commitment to the equal application of the law, judges do not appear to have the same habits of mind as the chronic egalitarians. The proportion of white judges in our study who revealed automatic associations of white with good and black with bad was, if anything, slightly higher than the proportion found in the online surveys of white Americans. Thus, a professional commitment to equality, unlike a personal commitment to the same ideal, appears to have limited impact on automatic racial associations, at least among the judges in our study. Alternatively, the over-representation of black Americans among the criminal defendants who appear in front of judges might produce invidious associations that overwhelm their professional commitment. In either case, our findings are consistent with the implicit associations found among capital defence attorneys. White capital defence attorneys, another group which might be expected to have strong professional commitments to the norm of racial equality,¹²⁴ exhibit the same automatic preference for whites as the general population.¹²⁵

Taken together, then, the research on judges and capital defence attorneys raises serious concerns about the role that unconscious bias might play in the criminal justice system. Jurors are drawn from randomly selected adults, and a majority of white jurors will harbor implicit white preferences. If police, prosecutors, jurors, judges, and defence attorneys all harbor anti-black preferences, then the system would appear to have limited safeguards to protect black defendants from bias. Based on IAT scores alone, both black judges and black jurors seem to be less biased than either white judges or white jurors, because black Americans show less implicit bias than white Americans. But even considerable numbers of blacks express implicit biases. Perhaps the only entity in the system that might avoid the influence of the bigot in the brain is a diversely composed jury.

That said, the rest of our results call into question the importance of IAT scores alone as a metric to evaluate the potential bias of decision makers in the legal system. Our second and third conclusions show that implicit biases can translate into biased decision making under certain circumstances, but that they do not do so consistently.

Implicit associations influenced judges — both black judges and white judges — when we manipulated the race of the defendant by subliminal methods. Judges with strong white preferences on the IAT made somewhat harsher judgments of the juvenile defendants after being exposed to the black subliminal prime, and judges with strong black preferences on the IAT were somewhat more lenient after exposure to the black subliminal prime. In effect, the subliminal processes triggered unconscious bias, and in just the way that might be expected.

The story for the explicit manipulation of race is more complicated, however. The white judges, unlike the white adults in the Sommers and Ellsworth study,¹²⁶ treated African American and Caucasian defendants comparably. But the proper interpretation of this finding is unclear. We observed a trend among the white judges in that the higher their white preference, the more favorably they treated the African American defendant in the battery case. Thus, among the white judges, implicit bias did not translate into racial disparities when the race of the defendant was clearly identified in an experimental setting.

We believe that the data demonstrate that the white judges were attempting to compensate for unconscious racial biases in their decision making. These judges were, we believe, highly motivated to avoid making biased judgments, at least in our study. Codes of judicial conduct demand that judges make unbiased decisions, at least in our study.¹²⁷ Moreover, impartiality is a prominent element in almost every widely accepted definition of the judicial role.¹²⁸ Judges take these norms seriously. When the materials identified the race of the defendant in a prominent way, the white judges probably engaged in cognitive correction to avoid the appearance of bias.

The white judges in our study behaved much like the subjects in other studies who were highly motivated to avoid bias in performing an assigned task.¹²⁹ What made our white judges different from the subjects studied by these other researchers is that most of the judges reported that they suspected racial bias was being studied, despite the fact that the only cue they received was the explicit mention of the defendant’s race.¹³⁰ We think this report was truthful, given that the judges behaved the same way as other white subjects who attempted to avoid the influence of implicit bias.

The black judges responded somewhat differently to the overt labeling of the defendant’s race. Like the white judges, the black judges in our study also reported being aware of the subject of the study, yet they showed a correlation between implicit associations and judgment when race was explicitly manipulated. Among these judges, a greater white preference produced a greater propensity to convict the African American defendant. In other words, the black judges clearly reacted differently when they were conscious that race was being manipulated — a difference that correlated with their score on the race IAT.

We do not conclude, however, that black judges are less concerned about avoiding biased decision making than white judges. We have no doubt that the professional norms against bias concern the black judges just as deeply as their white counterparts — if not more so. And we are mindful that research on the effect of race on judges’ decisions in actual cases demonstrates no clear effects.¹³¹ We believe that both white and black judges were motivated to avoid showing racial bias.

Why then did the black judges produce different results? We can only speculate, but we suspect that both groups of judges were keen to avoid appearing to favor the white defendant (or conversely, wanted to avoid appearing to disfavor the black defendant). Black judges, however, might have been less concerned with appearing to favor the black defendant than the white judges. Those black judges who expressed a white preference, however, behaved more like their white counterparts in this regard, thereby producing a correlation between verdict and IAT score among black judges.

We also cannot ignore the possibility that the judges were reacting to the race of the victim, rather than (or in addition to) the race of the defendant. In all cases, we identified the victim as the opposite race as the defendant. Furthermore, black judges might have reacted differently to the fact that the case involved a cross-racial crime.

Given our results, we cannot definitively ascribe continuing racial disparities in the criminal justice system to unconscious bias. We nevertheless can draw some firm conclusions. First, implicit biases are widespread among judges. Second, these biases can influence their judgment. Finally, judges seem to be aware of the potential for bias in themselves and possess the cognitive skills necessary to avoid its influence. When they are motivated to avoid the appearance of bias, and face clear cues that risk a charge of bias, they can compensate for implicit bias.

Whether the judges engage their abilities to avoid bias on a continual basis in their own courtrooms, however, is unclear. Judges are subject to the same significant professional norms to avoid prejudice in their courtrooms that they carried with them into our study. And judges might well point to our study as evidence that they avoid bias in their own courtrooms, where the race of defendants is often reasonably clear, and they never face subliminal cues. But courtrooms can be busy places that do not afford judges the time necessary to engage the corrective cognitive mechanisms that they seem to possess. And even though many decisions are made on papers only, judges might unwittingly react to names or neighborhoods that are associated with certain races. Control of implicit bias requires active, conscious control.¹³² Judges who, due to time pressure or other distractions, do not actively engage in an effort to control the “bigot in the brain” are apt to behave just as the judges in our study in which we subliminally primed with race-related words. Moreover, our data does not permit us to determine whether a desire to control bias or avoid the appearance of bias motivates judges in their courtrooms the way it seemed to in our study.

Furthermore, judges might be over-confident about their abilities to control their own biases. In recently collected data, we asked a group of judges attending an educational conference to rate their ability to “avoid racial prejudice in decision making” relative to other judges who were attending the same conference. Ninety-seven percent (35 out of 36) of the judges placed themselves in the top half and 50% (18 out of 36) placed themselves in the top quartile, even though by definition, only 50% can be above the median, and only 25% can be in the top quartile.¹³³ We worry that this result means that judges are over-confident about their ability to avoid the influence of race and hence fail to engage in corrective processes on all occasions.

To be sure, this is only one study, and it has its limitations. The results might be the product of the particular judges who participated in our study, or the materials we used, or even the fact that hypothetical scenarios were used. Most importantly, we cannot determine whether the mental processes of judges on the bench more closely resemble those of judges subliminally primed with race or those for whom race was explicitly manipulated. Thus, it is not clear how implicit racial bias influences judicial decision making in court, but our study suggests, at a minimum, that there is a sizeable risk of such influence, so we turn in “Mitigating implicit bias in court” to reforms the criminal justice system might consider implementing.

Several scholars have suggested that society might try to reduce the presence of unconscious biases by exposing decision makers to stereotype-incongruent models.¹³⁴ This suggestion, in fact, probably represents the dominant policy proposal among legal scholars who write about unconscious bias.¹³⁵ We certainly agree, for example, that posting a portrait of President Obama alongside the parade of mostly white male judges in many courtrooms would be an inexpensive, laudable intervention.

Our results, however, also raise questions about the effectiveness of this proposal. The white judges from the eastern jurisdiction in our study showed a strong set of implicit biases, even though the jurisdiction consists of roughly half white judges and half black judges. Indeed, the level of implicit bias in this group of judges was only slightly smaller than that of the western jurisdiction, which included only two black judges (along with 36 white, five Latino, and two Asian judges). Exposure to a group of esteemed black colleagues apparently is not enough to counteract the societal influences that lead to implicit biases.

Consciously attempting to change implicit associations might be too difficult for judges. Most judges have little control over their dockets, which tend to include an over-representation of black criminal defendants.¹³⁶ Frequent exposure to black criminal defendants is apt to perpetuate negative associations with black Americans. This exposure perhaps explains why capital defence attorneys harbor negative associations with blacks,¹³⁷ and might explain why we found slightly greater negative associations among the white judges than the population as a whole (although as we noted above, the latter finding might have other causes).

The criminal justice system might test candidates for judicial office using the IAT or other devices to determine whether they possess implicit biases. We do not suggest that people who display strong white preferences on the IAT should be barred from serving as judges, nor do we even support using the IAT as a measure of qualification to serve on the bench.¹³⁸ The direct link between IAT score and decision making is far too tenuous for such a radical recommendation. And our data shows that judges can overcome these implicit biases at least to some extent and under some circumstances. Rather, knowing a judge’s IAT score might serve two other purposes. First, it might help newly elected or appointed judges understand the extent to which they have implicit biases and alert them to the need to correct for those biases on the job.¹³⁹ Second, it might enable the system to provide targeted training about bias to new judges.¹⁴⁰

Judicial training should not end with new judges, however. Training for sitting judges is also important. Judicial education is common these days, but one problem with it, at least as it exists at this time, is that it is seldom accompanied by any testing of the individual judge’s susceptibility to implicit bias, or any analysis of the judge’s own decisions, so the judges are less likely to appreciate and internalise the risks of implicit bias.¹⁴¹ As Timothy Wilson and his colleagues have observed, “people’s default response is to assume that their judgments are uncontaminated”.¹⁴² Surely this is true of judges as well. Moreover, because people are prone to egocentric bias, they readily assume that they are better than average, or the factors that might induce others to make poor or biased decisions would not affect their own decisions. Our research demonstrates that judges are inclined to make the same sorts of favorable assumptions about their own abilities that non-judges do.¹⁴³ Therefore, while education regarding implicit bias as a general matter might be useful, specific training revealing the vulnerabilities of the judges being trained would be more useful.¹⁴⁴

Another problem with training is that although insight into the direction of a bias frequently can be gained, insight into the magnitude of that bias cannot. One group of psychologists provided the following example:¹⁴⁵

The psychologists explained:¹⁴⁶

This scenario illustrates the problem well. How is one to know if correction is warranted, and if so, how much?¹⁴⁷ In a circumstance like the one depicted above or like any of the circumstances described in the materials included in our study, there is a risk of insufficient correction, unnecessary correction, or even over-correction, resulting in a decision that is distorted as a result of the adjustment, but simply in the opposite direction.¹⁴⁸ Testing might mitigate this problem by helping judges appreciate how much compensation or correction is needed.

The results of our study are thus somewhat surprising in that the white judges’ corrections in the case in which the defendant’s race was explicit seemed to be neither too much nor too little. On average, these judges treated white and black defendants about the same. This result cannot, however, reasonably be taken as meaning that judges correct for the influence of implicit bias perfectly in all cases in which they attempt to do so. We presented only one scenario — other cases might produce over-compensation or under-compensation. And individual judges are apt to vary in terms of their willingness or ability to correct for the influence of unconscious racial bias. Also, the white judges were slightly less harsh on the black defendants. The difference simply failed to rise to the level of statistical significance, as it was small (only six percentage points). Had we collected data on a thousand judges rather than a hundred, we might have begun to observe some over-compensation or under-compensation.

The criminal justice system could also implement an auditing program to evaluate the decisions of individual judges in order to determine whether they appear to be influenced by implicit bias. For example, judges’ discretionary determinations, such as bail-setting, sentencing, or child-custody allocation, could be audited periodically to determine whether they exhibit patterns indicative of implicit bias. Such proposals have been suggested as correctives for umpires in Major League Baseball and referees in the National Basketball Association after both groups displayed evidence of racial bias in their judgments.¹⁴⁹

Auditing could provide a couple of benefits. First, it would obviously increase the available data regarding the extent to which bias affects judicial decision making. Second, it could enhance the accountability of judicial decision making.¹⁵⁰ Unfortunately, judges operate in an institutional context that provides little accountability, at least in the sense that they receive little prompt and useful feedback.¹⁵¹ Existing forms of accountability, such as appellate review or retention elections, primarily focus on a judge’s performance in a particular case, not on the systematic study of long-term patterns within a judge’s performance that might reveal implicit bias.¹⁵²

In addition to providing training or implementing auditing programs, the criminal justice system could also alter practices in the courtroom to minimise the untoward impact of unconscious bias. For example, the system could expand the use of three judge courts.¹⁵³ Research reveals that improving the diversity of appellate court panels can affect outcomes. One study found that “adding a female judge to the panel more than doubled the probability that a male judge ruled for the plaintiff in sexual harassment cases ... and nearly tripled this probability in sex discrimination cases.”¹⁵⁴ In trial courts, judges typically decide such issues alone, so adopting this mechanism would require major structural changes. Although convening a three-judge trial court was once required by statute when the constitutionality of a state’s statute was at issue,¹⁵⁵ three-judge trial courts are virtually nonexistent today.¹⁵⁶ The inefficiency of having three judges decide cases that one judge might be able to decide nearly as well led to their demise, and this measure might simply be too costly to resurrect.

Another possibility would be to increase the depth of appellate scrutiny, such as by employing de novo review rather than clear error review, in cases in which particular trial court findings of fact might be tainted by implicit bias. For example, there is some evidence that male judges may be less hospitable to sex discrimination claims than they ought to be.¹⁵⁷ If that bias does exist, less deferential appellate review by a diverse panel might offer a partial solution.

You are presiding over a case involving criminal charges against a juvenile, William T William is a 13-year-old who was arrested for shoplifting in a large, upscale toy store in __ . He has no prior record. You are trying to get a sense of the case and the only facts available to you follow:

You are presiding over a case involving criminal charges against a juvenile, Michael S, who was arrested for armed robbery of a gas station when he was two days shy of his 17th birthday. He has one prior arrest for a fight in the school lunchroom the previous year. You are trying to get a sense of the case and the only facts available to you follow:

Defendant: Andre Barkley, 6'0", 175 lbs, African American male, 18-years-old, student

Alleged Victim: Matthew Clinton, 6'2", 185 lbs, Caucasian male, 16-years-old, student

Charge One count of battery with serious bodily injury

To calculate the mean-difference IAT score, we largely followed the procedures outlined in Nosek and his colleagues’ report of IAT scores from tens of thousands of people collected through the Internet.¹⁶⁷ We also wanted to compare our results with the more detailed, contemporary Internet data collected and reported on the “Project Implicit” website, which appears to use the same scoring method.¹⁶⁸ Because the data in these studies come from voluntary participants who access the site on the Internet, the authors have adopted a number of techniques for excluding data from participants who may have wandered off during the study or are otherwise not fully engaged with the tasks.¹⁶⁹ While such techniques are less appropriate for our participants, who were engaged in person, we followed the Project Implicit scoring methods to facilitate a comparison.

The authors of the Internet study first adjusted raw latency scores that seemed much slower or faster than participants who are fully engaged with the task. The researchers treat any latency larger than 3000 milliseconds (ms) as 3000 ms, and any latency shorter than 300 ms as 300 ms.¹⁷⁰ The researchers also eliminated the first two trials in all rounds from consideration, having found that these rounds often displayed an erratic pattern of long latencies — presumably because participants commonly begin the task, and then pause to get settled in.¹⁷¹ These researchers also excluded participants who failed to perform to certain criteria. They excluded participants who exhibited overall average latencies in the two critical rounds greater than 1800 ms, or who displayed average latencies in either of the two critical rounds (four or seven) greater than 1500 ms.¹⁷² They also excluded participants who produced any critical round in which more than 25% of the latencies were less than 300 ms.¹⁷³ Finally, they excluded participants who made more than ten errors in any critical round.¹⁷⁴ These researchers report that these criteria resulted in the exclusion of 15% of their subjects.¹⁷⁵ After these adjustments and exclusions, these researchers calculated the mean difference between the critical stereotype-congruent round (either round four or seven) and the stereotype-incongruent rounds (either round four or seven).¹⁷⁶

We followed these procedures to calculate the mean IAT score for the judges in our study. We capped latencies greater than 3000 ms as 3000 ms, and raised latencies lower than 300 ms to 300 ms.¹⁷⁷ We also discarded the first two rounds from the analysis. We excluded the results of the race IAT from six judges (or 4.5%) who produced either mean latencies greater than 1800 ms in one of the two critical rounds of the race IAT or a mean across both rounds greater than 1500 ms.¹⁷⁸ Similarly, we excluded the results of the gender IAT from ten judges (or 7.5%) who violated one or both of these criteria.¹⁷⁹ Nosek and his colleagues reported that they eliminated 2% of their participants for being too slow,¹⁸⁰ whereas we eliminated more. At the same time, none of the judges in our studies produced more than a 25% error rate in either of the critical rounds in either IAT. By contrast, Nosek and his colleagues eliminated roughly 13% of their participants for having high error rates.¹⁸¹ The judges were thus slower and more accurate than Nosek and his colleagues’ subjects, and overall, the application of their criteria eliminated fewer judges than their results would have predicted.

Unlike Nosek and his colleagues,¹⁸² we did not randomise the order in which we presented the IAT. That is, roughly half of the participants in the Internet sample receive the stereotype-congruent round first, while half receive the stereotype-incongruent round first. The seven-round IAT is designed to reduce order effects substantially, but nevertheless, they remain. Greenwald and his colleagues report that the IAT scores can correlate weakly with the order in which the materials are presented.¹⁸³ Randomizing the order would have produced a cleaner measure of the IAT effect across all judges, but would have reduced the correlation between the IAT score and behavior.¹⁸⁴ Hence, all of our judges received the materials in the same order. On the race IAT, judges receive the stereotype-congruent pairing first (white/good and black/bad) and on the gender IAT, judges receive the stereotype-incongruent pairing first (male/humanities and female/science). Our procedure would have tended to increase the IAT score on the race IAT, as compared to the sample by Nosek and his colleagues, and decrease the IAT score on the gender IAT.

By using these procedures, we scored judges in exactly the same method as Nosek and his colleagues in the data that they harvested

To calculate the standardised IAT score, we followed the procedures recommended by Greenwald and his colleagues.¹⁸⁵ These researchers designed their methods precisely to improve the reliability and predictive power of their measures.¹⁸⁶ We use the methods that produced the highest correlations between implicit measures and behavioral measures. They differ from the scoring method used to calculate the mean differences. As noted above, we used the Greenwald methodology to collect the IAT scores.¹⁸⁷ Following those scoring procedures, we removed single trials with latencies greater than 10,000 ms (that is, 10 seconds) from the analysis. We otherwise left low and high values in the analysis without adjustment. We made no correction for errors, because our IAT collection methods required the judges to provide the correct response before proceeding and hence the latency includes the delay that would result from an incorrect answer. Error rates were also low, as noted above. Following Greenwald and his colleagues’ scoring method, we used all of the trials, rather than dropping the first two in the round.

We departed from the method Greenwald and his colleagues endorse, however, in one respect. Those researchers suggested using the two paired practice rounds (rounds three and six) in the analysis.¹⁸⁸ They reported that using this data produced slightly higher correlations between the IAT scores and explicit choices.¹⁸⁹ We found, however, that latencies in the practice rounds were highly erratic. A high percentage of the trials eliminated for being greater than 10,000 ms were in the trial rounds.¹⁹⁰ Even with these observations removed, the average standard deviation in the two practice rounds on the race IAT was over one second (1064 ms), as compared to 596 ms in the trial rounds. This suggested to us that we ought not to use the practice rounds in the analysis. The practice rounds of the gender IAT were more stable. The standard deviation from the practice rounds (724 ms) was much closer to that of the trial rounds (560 ms). Even though the practice rounds in the gender IAT seemed more stable, for consistency, we dropped these as well. Our measure of the IAT effect for purposes of correlating the IAT scores with judges’ decisions was therefore the average difference between the stereotype-congruent round and the stereotype-incongruent round divided by the standard deviation of latencies in both rounds combined. Following Greenwald and his colleagues, we call the measure d'.

Because the latencies that we observed seemed slower than those which have been observed in the Internet study, we assessed the correlation between our two IAT measures and the mean latency. The correlation coefficients between the mean differences and the overall latency were 0.305 on the race IAT and 0.361 on the gender IAT. These correlations are high enough to indicate that our judges have higher IAT scores than other populations simply because they were somewhat slower.¹⁹¹ The standardised IAT measure using only the trial rounds, however, produced correlations of only 0.046 and 0.002 with the overall mean latencies for the race and sex IATs, respectively. Hence, the d' measure provided a much more reliable measure of the IAT effect than the mean difference.