Algos, bias, due process, & you

I was given free reign to expound on the topic of algorithmic bias last week as a guest lecturer in one of our legal tech classes, and I had such a ball designing and delivering this one-off lesson, I just had to share. What follows is the “DVD commentary/making of.”

TL;DR: I built a bunch of highly-modular online simulations you can use with your students. They cover automation bias, the false positive paradox, competing definitions of fairness, disparate impact resulting from machine bias, and the value of due process. Most of them are self-explanatory and can be used on their own. However, the first one intentionally lacks in-simulation context, and I think linking them together creates a more-compelling narrative arc. Play with the links above in isolation, or join me below for the guided tour.

Motivation

The seed of the class was this reporting from 2016, Machine Bias: There’s software used across the country to predict future criminals. And it’s biased against blacks. In it, the authors describe the use of a risk assessment tool by courts to help decide questions like if someone should be granted bail.

When that article was published I was a data scientist at the public defenders, and in my corner of the world, it created a bit of a furor, kicking up discussion around a constellation of issues I wanted the students to understand. The first was standard machine bias, the idea that tools trained on biased data produce biased results, and that these biases often “sneak” in under the radar perhaps due to ignorance or malfeasance. Consequently, these tools can impact different populations differently, and here they can exacerbate existing patterns of bias. However, I also wanted them to understand that sometimes what counts as fair isn’t always straightforward, and that one person’s measure of what’s fair could be in tension with someone else’s good-faith measure. That is, “bad” results aren’t just the results of malicious or incompetent actors. Sometimes goals differ, and not in the cartoonish “good is just what benefits people like me” sort of way.

I also wanted to touch on the fact that these tools don’t behave the way many imagine. Consider the fact that highly accurate predictors (e.g. >95% accuracy), like a screening test for cancer, can be wrong about their positive predictions most of the time if the object of their prediction is rare.

Add to the above the fact that people presented with the output from such tools often overrely on their answers, what we call automation bias, and you can start to see why the use of such tools can be so fraught. Mostly, I wanted the students to understand this context.

At first I thought I’d turn the article into a role play. The students could act as court personnel tasked with deciding when and how to use a decision support tool like the risk assessment found in the article. So I asked myself what core concepts would they need to understand to have an informed discussion, recognizing that I only had an hour and 50 minutes with them. These learning objectives ended up echoing the issues above. Namely, students should understand:

• accuracy isn’t always the right performance measure
• mathematical models encode and replicate the biases found in their training data
• there can be competing and contradictory ideas of what makes something fair
• under certain conditions people are likely to over-rely on machine outputs
• the choices we make about how to use tools embody and reveal what we value

It then occurred to me that all they knew about my session was that it would be on “algorithmic bias.” I hadn’t even assigned any reading. I realized I could use that ambiguous framing to teachable ends. What if I could get them to experience automation bias first-hand? That would make it harder to dismiss as something that only other people fall victim to. . . . A plan began to form.

Framing & signposts

Normally, I start class with a clear roadmap of the journey ahead, but I decided to play this session more like a mystery. So, I kept the day’s agenda a bit vague. You can’t set up a lesson where students experience falling victim to automation bias by saying “I’m going to see if you all over-rely on this tool.” Additionally, as I put the lesson together, a more timely role-play presented itself. A role-play I felt would best be left as a surprise (for you too). So, this is what they got. Me imploring them to treat today’s class as a mystery they were tasked to solve. Note: I link to the whole slide deck at the end to avoid SPOILERS!

This was followed by a very vague framing of how we would use our time.

Most importantly, I wanted them to understand that we would be breaking into small groups, playing with some tools, answering questions within our groups, then discussing things together as a class, again and again. And yes dear reader, the break overlapped with their time in groups, and I was overly ambitious about what we would be able to cover. We ended up making it right up to the last exercise which I invited them to explore further after class. That being said, I think my learning objectives mostly came across, but if I were doing this again, and I had more time, I would break off the exercise on automation bias, perhaps placing it at the end of the previous class. I’d then use the freed up time to give the last exercise on truth & justice more time to breathe.

Automation bias

When you’re not parachuting in to guest lecture, when you have your own class, you get the chance to build a rapport with your students. You get to know them; they get to know you, and that trust is a currency you can use to help them step outside of their comfort zone and do the hard work of learning. Alas, my plan involved asking for something hard out of the gate. So, I fell back on extrinsic motivations. I told them that for our first exercise they would all be using an AI assistant I built to review citations. After they had a chance to use it we would have a class discussion. I suggested they hold the following question in their head, “What makes something a good decision assistant?“

I then explained that we would be working for 20 minutes straight. To soften the ask, I explained that I would be doing the same exercise and that I would sit at the front of the class with my back to them so they could watch me do exactly the same work they were—solidarity! I also told them that whoever performed best at the task would get a $10 gift card to Dunkin’. If they had been my students I would have made this a fancy certificate/”award” with no monetary value, but since I wouldn’t be grading them and they wouldn’t be evaluating me, this seemed like a permissible option. It was also, as you’ll see, part of the exercise. I didn’t want anyone to claim they weren’t trying, and the whole exercise was engineered to make them feel pressured to perform.

Here’s how it worked. Students got a start screen explaining the task at hand. It begins:

You’re a court clerk reviewing a brief for citation integrity. You’ll also be asked after this experience to help evaluate this tool. You’ll be working with wholly and lightly fictionalized primary texts. So, your goal is not to know more about “the law” than the system but to assess the citations on their own terms (i.e., does the cite support the proposition made in the brief given the provided authority?)

• You may rely on the assistant’s flag, open its summary, or read excerpts.
• Speed matters — you’re “competing” against other clerks and will be paired with a pacing buddy to keep you on task.

This was followed by an overly complicated description of the performance metric they will be evaluated on. TL;DR: Go fast, but not so fast that the quality of your work drops.

I explained that I used an LLM to lightly fictionalize a bunch of primary texts and that the tool was working with these and only these. Consequently, it didn’t matter so much if the propositions cited are “True,” rather what matters is if they are supported by the text of the cited authority, whether they are internally consistent. When they hit Start, they are shown a progress spinner saying that they are being paired with a “pacing buddy.” The pressure begins. They are then presented with a quote from our brief, an AI generated restatement of the proposition being cited, and the relevant citation. I informed them that they should assume that the “(at __)” correctly points to the citation and text included in the Full excerpt.

Below these is an Assistant box which gives the AI assistant’s take on whether or not the proposition is supported. This comes with a label ” Supported” or “Not Supported” and a confidence score (0-100%). Below this are three buttons the user can click to see a Summary, Authority details, or a Full excerpt. Importantly, to see this information they must click the buttons.

You’ll also notice that the pacing buddy’s progress is laid out under “Rival Clerk” along with a reminder that we’re measuring the user’s speed, accuracy, and whether they override the assistant. Dear reader, the “Rival Clerk” is not one of their peers. It’s a dark pattern (pre-programmed script) designed to make them keep going and think they might just have a chance. There’s so much in here ripe for discussion.

What followed was brutal—20 minutes of dedicated focus! I occasionally attempted to break the tension shouting out a running commentary. “Wow, I forgot how much I hate this,” or “Only five minutes left. Dunkin’, Dunkin’, Dunkin’!” At the end, users were presented with a results screen that explained a bit more about the exercise. Mostly, it broke the experience down into three distinct phases and measured the user’s performance accordingly. There were three possible outcomes: (1) No clear evidence of automation bias (2) You may have fallen victim to automation bias; and (3) You likely fell victim to automation bias. Most of them got something that looked like the following (cut-and-pasted from my assessment).

Assessment

You likely fell victim to automation bias

Your performance suggests you kept relying on the assistant as the task got harder, without increasing verification.

• Accuracy: 93% (P1) → 92% (P2) → 49% (P3)
• Flag-only: 27% (P1) → 31% (P2) → 76% (P3)
• Opened full excerpt: 47% (P1) → 38% (P2) → 20% (P3)
• High-confidence misses (assistant ≥90% confident): 17 in Phase 3 (19 total)
• Misses made with flag-only verification: 15 in Phase 3 (15 total)

Automation bias is a tendency to over-rely on a decision aid — especially under time pressure — leading to commission errors (accepting incorrect recommendations) and omission errors (failing to notice problems because the system signaled “all clear”).

In this simulation, the assistant was highly reliable early, then the legal context shifted in Phase 3 toward authority-weighting pitfalls (nonbinding authorities, jurisdiction mismatches, circuit splits) while the assistant remained confident. The point is to see whether you adjusted your verification habits when the environment changed.

What your data shows: Your accuracy changed by -1pp from Phase 1 → 2, then -44pp from Phase 2 → 3. Your flag-only rate changed by +4pp (P1→P2) and +45pp (P2→P3). Your full-excerpt reads changed by -8pp (P1→P2) and -19pp (P2→P3).

Interpretation: The combination of a Phase 3 accuracy drop and continued low verification suggests classic overreliance: you treated the assistant as a substitute for checking, rather than a cue to check.

Jagged frontier note: A key takeaway is that the assistant didn’t just “get worse” randomly — it became less reliable after the context shifted. This mirrors what Ethan Mollick and co-authors call the AI “jagged frontier”: systems can be highly capable in one pocket of work but surprisingly unreliable on a nearby-looking task. In this simulation, early items emphasized more uniform, binding authority; later items introduced authority hierarchy, jurisdiction, and splits, where the assistant’s mistakes were more likely — and often overconfident.

Don’t overcorrect: After being burned, people sometimes swing to the opposite extreme — algorithm aversion — avoiding the tool even when it is helpful. A practical goal is calibration: use the assistant for throughput, but selectively verify in the “jagged” areas where errors are concentrated. Of course, this is easier said than done. ;)

Those who didn’t see “You likely fell victim to automation bias” were told about the bias and asked to wait and see how the rest of the class did. The final screen also included additional readings and instructions on how they could share their results with me so we could pick the winner (based on their anonymized ID or alias) and see how the class did as a whole. Here’s a selection from the additional reading section.

Learn more

• AI Safety and Automation Bias: The Downside of Human-in-the-Loop “Automation bias is a critical issue for artificial intelligence deployment. It can cause otherwise knowledgeable users to make crucial and even obvious errors. […]”
• Automation bias: a systematic review of frequency, effect mediators, and mitigators “Automation bias (AB)—the tendency to over-rely on automation—has been studied in various academic fields. Clinical decision support systems (CDSS) aim to benefit the clinical decision-making process. […]”
• Centaurs and Cyborgs on the Jagged Frontier “… AI is weird. No one actually knows the full range of capabilities of the most advanced Large Language Models, like GPT-4. No one really knows the best ways to use them, or the conditions under which they fail. There is no instruction manual. On some tasks AI is immensely powerful, and on others it fails completely or subtly. And, unless you use AI a lot, you won’t know which is which…”
• Algorithm aversion: people erroneously avoid algorithms after seeing them err “Research shows that evidence-based algorithms more accurately predict the future than do human forecasters. Yet when forecasters are deciding whether to use a human forecaster or a statistical algorithm, they often choose the human forecaster. […]”

After everyone shared their results, I loaded them into the exercise’s dashboard. And I have to say, I couldn’t have asked for better results. Which is to say, I couldn’t have asked for worse results. These two graphs is what we spent the most time talking about.

Here’s what you’re looking at. The assistant’s accuracy varied over the three phases: 100% for phases 1 and 2, followed by 70% for phase 3. The average student accuracy, however, is where the story really takes shape. In phase 1 they start with an average accuracy of 79%. After they get a feel for the tool this jumps in phase 2 to 85%. But here’s the kicker, when the context shifts, causing the assistant’s accuracy to fall, the students ended up doing worse than they did in the beginning, scoring only 65%. Perhaps more telling is how often they relied on the assistant’s recommendation without consulting the additional info (summary, authority, or excerpt). In phase 1, they avoided additional info 65% of the time. In phase 2, this went up to 80%, and in phase 3 it jumped to 84%.

Almost every one of them fell victim to automation bias.

We talked about what happened, and I hope the lesson sticks with them. Almost every one of them fell victim to automation bias. Admittedly, the system was designed to push them in this direction, but hopefully by giving it a name and being forced to face the reality that it can happen to them or their peers, this is a concern they will carry with them into their practice. At least one student caught on to the shift in content in phase 3, asking if there was a trick related to jurisdiction. This led to a brief discussion of the jagged frontier, and everyone was very excited for our winner. I did point out, however, that they were not our most accurate user or the fastest. Our measure after all tried to balance competing interests. Foreshadowing perhaps?

If you want to adapt this lesson, the code for the tool can be found in this GitHub repo. You can also use the live version. It, however, instructs users to upload their results to a Google Drive you won’t be able to access. So, you may want to download the repo and edit that bit. The whole thing is simply a collection of static files, with the citation info living in a single JSON file you can edit with this tool. I should mention that all of the simulations used here were written over the course of three days with the assistance of either OpenAI’s GPT 5.2 or Google AI Studio. I haven’t made this the focus of this post, but it is worth noting that absent these tools, I would not have been able to produce this lesson as quickly as I did. They were really helpful. Three days is not a lot of time, and I think the simulations are what made this lesson work. I did get a kick out of using AI to draft a simulation that cautions folks to be careful how they use AI. ;)

More about construction and QC: Prompting of OpenAI’s GPT 5.2 was used to create 20 successively better versions of the citation tool, the last of which was edited manually. This included the creation of a tool I could use to check and edit the content of the fictionalized case information, and a testing mode (triggered by appending #test to the tool’s URL) which uses output in the browser’s console log to help me perform quality checks of the assistant and confirm expected behavior (i.e., it let me see what the tool considered ground truth so that I could compare it with the on-screen output). The case law, briefs, citations, and assistant outputs are all wholly or lightly fictionalized, a task that was only possible thanks to the GPT. Here I was happy to rely entirely on the LLM-produced text after spot checks. An odd consequence of this workflow was that all the app’s copy was pre-populated with AI-generated “placeholder” text. I had to go back and edit much of this. I think I only kept one of the articles the GPT suggested for the learning more section, and had to make a lot of other such edits. If the text was good enough, however, I tended to let it be. This is an interesting consequence of the UX, and a dimension along which I am rather critical of tools that produce AI-generated content (i.e., how provisional is the tool’s output/who’s really “driving”). For what it is worth, I did not use an LLM to produce any portion of the text in this post aside from the cut-and-paste or screenshotted copy that appears in the simulations. Relatedly, despite building in multiple testing tools (see above) I fear I was inclined to resolve close calls in favor of the existing text examples. I think this was acceptable given the context (e.g., the low cost of errors given we aren’t working with “real” briefs or real clients), but I couldn’t let a description of this process go without providing some commentary on how automation bias might have shaped this exercise on automation bias.

The false positive paradox

Since we had just made use of a tool that purported to make predictions with some level of confidence, I suggested we might want to look more into what such tools are really telling us. To take the room’s temperature I made use of CALI’s Instapoll tool to ask the following question. If you’re not familiar with the Instapoll tool, it’s a simple way for students to answer multiple choice questions from their laptops or phones. The instructor can then see a bar graph of student replies in real-time. As I often do, I opted to share the results with the students in real-time as well. Here’s our first poll.

100 people are given a test for cancer. It’s 90% accurate. 10 people test positive. How many of these 10 have cancer?

A. 10
B. 9
C. 1
D. I don’t know

Most folks, thought the answer was B (9). I suggested they might want to think about that some more. I then divided the class into groups of roughly 3 students each and asked that each group explore the following simulation together as a group for a few minutes, after which we would talk. If you haven’t already, I invite you to play with the simulation too.

After folks had played for a bit, we talked about what the simulation was trying to show. TL;DR: high-performing tests can be wrong about most of their positive predictions if the thing they’re trying to predict is rare. Context matters!!!

I then reran the above poll, and thankfully most folks changed their answer to D (I don’t know). Always consider the base rate.

More about constructionand QC: This tool and the remainder of the simulations linked here were made using Google’s AI Studio. I have to say it was probably “easier” to get the first 80% of work done on these than on the first simulation where I used OpenAI’s ChatGPT, maybe because there I was using the chat interface absent a preview of the app, but that last 20% was harder to complete here because the output wasn’t structured in a way I could easily edit. This was mostly because I wasn’t familiar with its structure and the search feature wasn’t as robust as I would have liked. Sometimes, I could manually change the code, but most of the time, I had to give up and just provide natural language instructions esp. re styling. This ended up being less efficient and caused me to leave much of the visual styling as first presented. That is, if something wasn’t quite how I liked it but wasn’t a deal breaker, I left it as is. As you know, this is a dimension along which I am critical of tools that produce AI-generated content (i.e., how provisional is the tool’s output/who’s really “driving”). I suspect, however, that with time I may find it easier to navigate and edit these studio apps. I still need to figure out how to host them myself, esp. since only one of them (you’ll see it below) uses any AI and needs to call Google’s API. Looks like I might need Node.js. As for QC, in addition to performing manual “sanity” tests, I ran the code for each of the Google AI Studio simulations through GPT 5.2 and had it perform unit test on the underlying models where possible. It did a decent job of pointing out where simplifying assumptions were made and helped orient me in the code. It is worth noting, not only were these tools made in three days, but I only started using AI Studio the weekend before I taught the class.

Is it fair?

I suspect the first simulation (automation bias) has the most mainstream appeal, but this one is actually my favorite. That’s because it generated some really good conversations and helped students confront something that doesn’t get said enough. There can be competing and mutually exclusive concepts of fairness, and the policies that seek to deliver on one measure of fairness might have to change when the context changes.

We started with another poll.

When using a single-use bathroom, how do you leave the toilet seat?

A. As I found it
B. As I used it
C. Always down
D. Always up

I explained that I was actually thinking of the gender-inclusive bathroom on the fifth floor that houses a single toilet and has no cover (just a seat). I’m sorry to say I didn’t capture the responses, but my recollection is that there was a good deal of diversity in the answers, except for D not really getting many (or any takers). I then framed the task before them.

If you could make everyone follow a single rule, which would be the most fair?

A. As they found it
B. As they used it
C. Always down
D. Always up

Why?

I then did a very quick demo of this simulation.

It lets you simulate what happens when a bunch of folks following different rules share a toilet. It assumes two populations, “sitters” and “standers” (folks who sometimes stand). It lets you see what happens when different seat behavior is adopted, factoring in two types of cost: (1) the cost of having to change the seat’s position before you use the toilet; and (2) the cost of having to clean the seat if the last person failed to raise the seat when really they should have. This means you have to assign a relative value to these costs and make assumptions about how frequent certain behaviors are among your groups. After you’ve dialed these in, however, you can simulate the outcome for 100 users at a time to see what happens.

You can ask an “AI Socrates” what they think about your norms, and eventually unlock the ability to simulate outcomes across all possible choices. I instructed the “AI Socrates” to question the fairness of their selected behavior, focusing on a measure where it performed poorly. In this way I hoped to nudge them into exploring alternate theories of fairness. One student asked which measure we were supposed to use to gauge fairness, and I responded with some variation of, “I don’t know. What do you think?” previewing how I hoped the bot would behave.

When we eventually returned to a whole class discussion, we retook the “fair rule” poll, and there was a surprisingly large number of folks choosing D (Always up). When I asked them why, it became clear that they had considered a world in which the cost of cleaning up after scofflaws was so high that it made sense to just leave the seat up. That is, the over all cost (sum of every person’s individual costs) was lowest if they just gave in and took “preventative action.” I found this a bit depressing, but it was self-consistent. Other groups made assumptions that found leaving the seat as you used it resulted in less work for everyone. Others found that having sitters leave it down and standers leave it up made things a bit more equitable between groups even if it meant a little more work overall, and the chances that one would find the seat in the position they planned on using it increased when everyone left it down.

There can be competing and mutually exclusive concepts of fairness, and the policies that seek to deliver on one measure of fairness might have to change when the context changes.

These are only some of the steady states you can find, and I invite you to run your own simulations as they are rather fact dependent. The point being, there can exist multiple concepts of the good that compete with each other and that the policy that gets you closest to your desired result may have to change based on context.

You’re the expert (role play)

Only now did I introduce the ProPublica reporting on machine bias, sketching the broad strokes of their analysis and some of the discussion that followed. I focused on the fact that these tools can make different predictions for different populations based on their training data. Finally, I presented them with their role play.

You’re a law enforcement org looking to incorporate facial recognition technology into your work. Produce two lists:

1. Performance benchmarks the tool must pass before you will use it; 
2. A set of rules your officers must follow when using it.

You can use the following tool to help.

I reminded them what we had learned along the way.

• accuracy isn’t always the right performance measure
• mathematical models encode and replicate the biases found in their training data
• there can be competing and contradictory ideas of what makes something fair
• under certain conditions people are likely to over-rely on machine outputs

I then invited them to play with the following simulation before making their lists.

Some folks suggested requiring equal treatment of populations by facial recognition before they would consider using the tech. Others suggested setting high confidence thresholds. We talked about requiring warrants before running a check, and I shared with them a little bit about how Massachusetts attempted to address these issues in its recent police reform bill.

Then I told them that there actually was a federal law enforcement agency actively using facial recognition out in the real world called ICE, and I asked what safeguards folks thought they had in place… Things got a bit quiet, and I shared the following reporting.

To be clear, the linked article tells the story, using sworn testimony before a court, of how a single woman was incorrectly identified by the tool twice in one encounter. This was coupled with the fact that “ICE has told lawmakers the app, called Mobile Fortify, provides a ‘definitive’ determination of someone’s immigration status, and should be trusted over a birth certificate.”

It was only later in the week, after I taught the class, that a few Senators introduced the ICE Out of Our Faces Act. See “ICE Out of Our Faces Act” would ban ICE and CBP use of facial recognition.

Truth & justice

I then asked if folks were familiar with the two incidents reported here:

• ICE Arrest of a Citizen, Barely Dressed, Sows Fear in Twin Cities &
• ICE detains five-year-old Minnesota boy arriving home, say school officials.

They were. I speculated that the “targeted operations” cited were possibly driven by ‘ELITE’: The Palantir App ICE Uses to Find Neighborhoods to Raid and presented the following questions for them to consider in their groups.

What theory of fairness is consistent with ICE’s actions? What outcome(s) do they seem to be favoring? Articulate how this is consistent with constitutional guarantees of due process? 

You can use the following tool to help.

The simulation compares due process to medical screening and diagnostic tests and lets the user explore how changes to confidence thresholds at different points in the process effect different measures of cost. To help explore this, it lets you optimize your thresholds based on costs, or work in reverse to infer costs based on thresholds.

This is where it became clear we were going to run out of time. So, as they clicked away at the simulation, I reminded them of Blackstone’s ratio.

I wish I’d had the time to hear their answers to the above questions, but I’m reasonably certain they’ll be thinking about their own answers long after class. That being said, I left them with my most charitable reading of the theory of fairness consistent with ICE actions.*

The last question, “how is this consistent with constitutional guarantees of due process?” was, as they say, “left as an exercise for the reader.”

As we packed up, I invited them to revisit the question that preceded the first exercise.

What makes something a good decision assistant?

I reminded them that every tool they used in class could be considered a decision assistant, told them it had been an honor to be their teacher for the day, and that I was always available to talk. Man, I love my job.

Stay warm. Stay safe. Stay free.