Contents

1. Main Title: Beyond the Black Box: Why Defendants Must Have the Right to Understand Algorithmic Risk Scores
2. Introduction: The shift toward algorithmic sentencing and the “black box” problem.
3. Key Concepts: Defining algorithmic risk assessments (COMPAS, PSA) and the legal friction regarding trade secrets vs. due process.
4. Step-by-Step Guide: How legal teams can challenge and demand transparency in algorithmic scoring.
5. Examples and Case Studies: *State v. Loomis* and the real-world implications of proprietary code.
6. Common Mistakes: Over-reliance on scores, failure to cross-examine software vendors, and lack of expert witness usage.
7. Advanced Tips: Pushing for “Explainable AI” (XAI) standards and local transparency ordinances.
8. Conclusion: The path forward for digital equity in the courtroom.

***

Beyond the Black Box: Why Defendants Must Have the Right to Understand Algorithmic Risk Scores

Introduction

For decades, the American criminal justice system has grappled with the challenge of human bias in sentencing and bail decisions. In response, jurisdictions across the country have turned to technology, adopting algorithmic risk assessment tools to provide “objective” data to judges. These software programs, often touted as mathematical panaceas, generate scores that predict a defendant’s likelihood of reoffending or failing to appear in court.

However, these tools frequently operate as “black boxes.” Proprietary software companies shield their algorithms from public and judicial scrutiny, claiming trade secret protection. When a defendant’s freedom hinges on a number they cannot explain, challenge, or even fully understand, the fundamental right to due process is undermined. This article explores why defendants must possess an affirmative legal right to access and understand the factors influencing these scores and how practitioners can effectively fight for that transparency.

Key Concepts

Algorithmic risk assessments are essentially statistical models that ingest data—such as age, employment history, prior criminal records, and zip codes—to assign a numeric risk level to a defendant. Tools like COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) or the Public Safety Assessment (PSA) are designed to help judges make informed release or sentencing decisions.

The central legal friction lies in the tension between proprietary rights and due process. Under the Fourteenth Amendment, defendants are entitled to notice and an opportunity to be heard regarding any evidence used against them. If a sentencing recommendation is based on a black-box algorithm, the defendant cannot effectively cross-examine the “witness”—the software—nor can they challenge the accuracy of the underlying data or the weight given to specific variables. Without understanding the causal logic of the score, the right to contest its validity is functionally non-existent.

Step-by-Step Guide: Challenging Algorithmic Evidence

If you are a legal professional or an advocate working to ensure transparency in a case involving risk assessment tools, follow this structured approach to challenge the use of non-transparent algorithms.

1. Discovery Request for Algorithmic Disclosure: File a formal discovery motion requesting not just the final score, but the specific weighting of variables and the underlying source code used by the vendor. Anticipate a “trade secret” objection and be prepared to argue that constitutional due process outweighs commercial confidentiality.
2. Secure an Expert Witness: Retain a data scientist or a statistician capable of reviewing the tool’s validation studies. You need someone who can translate the “math” into plain language for the court, identifying whether the data used to train the model is biased against protected classes.
3. Challenge Data Inputs: Scrutinize the input data for accuracy. Many systems rely on flawed arrest data rather than conviction data. If the defendant’s score is inflated by a false entry or a misunderstanding of their criminal history, that error must be exposed.
4. Request an Evidentiary Hearing: Seek a hearing specifically to challenge the reliability of the software. Frame the argument around the Daubert or Frye standards for scientific evidence. If the tool’s internal mechanics cannot be tested or peer-reviewed, argue that it fails the threshold for reliable expert evidence.
5. Demand “Explainability” Documentation: Ask the prosecution to produce documentation that outlines exactly how the software arrived at a specific risk level for your client. If the vendor cannot provide an “explanation” of the factors, argue that the output is arbitrary and legally insufficient.

Examples and Case Studies

The landmark case State v. Loomis (2016) serves as a critical example of the limitations of the current legal landscape. In this case, the Wisconsin Supreme Court ruled that the use of a COMPAS risk score at sentencing did not violate a defendant’s due process rights, provided the tool was not the sole basis for the sentence. The court reasoned that the judge still exercised discretion.

However, Loomis also acknowledged the danger of the “black box.” The court explicitly warned that because the tool’s proprietary nature prevents defense counsel from challenging its methodology, the scores should be used with extreme caution. This case highlights a major “real-world” challenge: while judges are warned to be cautious, they often defer to the “math” because it provides a convenient veneer of objectivity, effectively ignoring the underlying lack of transparency.

Common Mistakes

• Accepting the Score at Face Value: Never treat an algorithmic risk score as an immutable fact. It is a prediction based on historical data, which inherently reflects the biases of past policing practices.
• Failing to Cross-Examine the Vendor: In many cases, lawyers fail to subpoena representatives from the software company. If the state relies on the software, the state must be held responsible for providing an expert who can explain it.
• Overlooking Data Bias: Many algorithms use “zip code” or “neighborhood” as a proxy for race or socioeconomic status. Failing to highlight these proxies during sentencing is a critical oversight.
• Ignoring “Explainable AI” (XAI) Standards: Don’t settle for “we can’t show you the code.” In the modern era of computing, frameworks exist to explain model decisions (such as SHAP or LIME values). If you don’t ask for these, you are leaving your client’s defense incomplete.

Advanced Tips

To go beyond the basics, leverage the growing legal and academic consensus on algorithmic accountability. Many jurisdictions are beginning to pass local ordinances requiring government agencies to conduct “algorithmic impact assessments” before purchasing new software. Check if your jurisdiction has an AI transparency registry.

Furthermore, focus on the “feedback loop” argument. Explain to the court that these algorithms create a self-fulfilling prophecy: if an algorithm predicts an area is “high risk,” police are sent there more frequently, leading to more arrests, which in turn reinforces the algorithm’s initial, biased prediction. When you show the judge the circular nature of the system, you transition the argument from a technical dispute into a profound moral and constitutional question about systemic fairness.

Conclusion

The digitization of the courtroom should not come at the cost of the right to due process. When a defendant stands before a judge, they have a right to know the weight of the evidence being used against them. Allowing corporations to hide the inner workings of sentencing tools behind a wall of “trade secrets” is a dangerous deviation from our principles of open and transparent justice.

Advocates, defense attorneys, and policymakers must insist on total transparency. If an algorithm is too complex or too secretive to be understood by the defense, it is too dangerous to be used in the administration of justice. By demanding explainability, challenging the quality of input data, and centering the constitutional right to confront one’s accusers, we can ensure that human rights prevail over lines of proprietary code.