James Grimmelmann and Daniel Westreich (Cornell Law School and Department of Epidemiology) have posted Incomprehensible Discrimination (7 California Law Review Online 164 (2017)) on SSRN. Here is the abstract:
Plaintiff Ruminant Defense Lague alleged that the defendant Zootopia Police Department's new algorithmic hiring process unfairly discriminates against herbivores in violation of Title VII.. The ZPD responded, and the Court of Appeals agreed, that because its algorithmic model does not explicitly consider species, it does not constitute disparate treatment, and that since the model is preedictive of job performance, any disparate impact on herbivores is justified as a business necessity. This case, in other words, squarely presents the problem of “proxies for proscribed criteria" identified by Solon Barocas and Andrew Selbst in the article, Big Data's Disparate Impact, 104 Calif. L. Rev. 101 (2016).
Held: Where the output of an algorithmic model is correlated both with job performance and with membership in a protected class, the employer bears the burden of demonstrating which of these correlations, if either, is causal. Pp. 170–77.
(a) It can be a permissible employment practice to use a properly validated predictive algorithmic model in personnel decisions, even if the model disproporionately favors some groups of applicants and employees over others. Pp. 170-71.
(b) Where such a model has a disparate impact on a protected group, the employer may rely on a defense of business necessity only if it shows that the factors the model relies on are not just correlated with job performance but causally connected to it. Here, it might be the case that the ZPD's model works not because it identifies the characteristics relevant to successful police work, but only because it identifies the herbivores in an applicant pool where the relevant characteristics are unequally distributed. Pp. 171-72.
(c) This allocation of the burden of proof will encourage employers to better understand the algorithmic models they use, and to share that knowledge with the public. Pp. 173-74.
(d) Other bodies of law, including the Fair Credit Reporting Act, already require explanations for why models behave the way they do. Pp. 174-76.
(e) This modification to the burden-shifting framework of disparate impact borrows from disparate treatment doctrine. Title VII does not permit an employer to do indirectly what it could not do directly. P. 176.
Our holding today is simple. Incomprehensible discrimination will not stand.