Protective randomness: why we fear the AI unknown and what to do about it

I was honored to keynote the 34th annual CALIcon–hosted by the Center for Computer-Assisted Legal Instruction (CALI). The theme was “Exploring the Unknown Unknowns of #LegalEd + #Legaltech.”

The video of my keynote is embedded at the end of this post. What follows is a lightly edited version of the talk with links and footnotes¹—stuff I couldn’t fit into my allotted time. That’s right, this post is a DVD special feature!

I had a lot of fun preparing and delivering this talk. I hope it shows. 😁

Unconstrained

When I accepted John’s request to speak here today, a sentence in his reply lit a fuse. It read simply, “You need not be constrained in any way.” Wow!

“You need not be constrained in any way.”

My thoughts immediately turned to the most subversive thing I could possibly do given the chance to speak to a room full of JDs and associates, the one line I usually skate right up to but dare not cross. Today, however, I am not constrained. Today, there will be …

… math!

The attorney John Todd Stuart once told a story about a law partner of his, a man who would later come to be of some historical note. Here’s a dramatic reading:

A few days later I found him already in the office and deeply engaged when I arrived. This was unusual, for I almost invariably preceded him there. He was sitting at the table and spread out before him lay a quantity of blank paper, large heavy sheets, a compass, a rule, numerous pencils, several bottles of ink of various colors, and a profusion of stationery and writing appliances generally. He had evidently been struggling with a calculation of some magnitude, for scattered about were sheet after sheet of paper covered with an unusual array of figures. He was so deeply absorbed in study he scarcely looked up when I entered. I confess I wondered what he was doing and what had occasioned his profound application at the office so early in the morning; nor was my curiosity allayed till a later hour in the day when he arose from his chair, apparently headed for the court-house. It was then that he enlightened me by announcing that he was trying to solve the difficult problem of squaring the circle. In a short time he returned to the office and resumed his study. For the better part of the succeeding two days he continued to sit there engrossed in that difficult if not undemonstrable proposition and labored, as I thought, almost to the point of exhaustion. He talked but little about it to me or to others, so far as I could observe, but it was evident he was toiling with all his might.

The story behind this quote deserves a footnote.³

Since that toiling occurred, mathematicians have proven that one cannot square the circle. In fact, it has become a euphemism for attempting the impossible. At the time, however, it was but a difficult unanswered question, the sort of thing a precocious lawyer might work on between court appearances.⁴

I see some of you with quizzical looks on your faces, trying to remember what it is to square the circle. So, a brief refresher, and trust me when I say it is not a digression. We will eventually come “full circle,” as it were.

Squaring the circle is a challenge in which one is tasked with the creation of a very particular square.

A square whose area is equal to that of a given circle.

“Oh,” you say, “This is the math. I got this.” The area of a circle is π•r².

And the area of a square is just one side times another, x².

To make things easy, we’ll give our circle a radius of 1, making its area equal to π because 1 squared is 1.

Remember, algebra translates roughly into “reduce and rearrange.” Anything you do to both sides of an equation does nothing to change the truth of the statement. It only serves to rearrange things.

So, we take the square root of both sides, revealing that all we have to do is make a square with sides equal to the square root of π.

The trick to squaring the circle, however, is that you are limited in your tools. Given only a compass and a straightedge, you must construct a circle and then this special square. That was the problem our young law partner was toiling on for two days, nearly to the point of exhaustion. And that law partner’s name was…

Abraham Lincoln, and I think he has a lot to teach us. Math is much more than addition and subtraction, multiplication and division. In the same way that the law is more than statutes and precedent. It, like the law, it is a way of thinking. What is it we’re always saying about what law school does? It teaches you how to … think like a lawyer.

As Doris Kearns Goodwin noted in Team of Rivals, the time Lincoln spent engrossed in study while living life on the circuit, “provided [him] the time and space needed to remedy the ‘want of education’ he regretted all his life.” So, when we hear that Lincoln, the autodidact’s autodidact, was teaching himself Euclid as a young lawyer, I think it’s fair to say, he considered it part of what it meant to think like a lawyer, and who are we to argue with Lincoln?

As you may have noted from my introduction, my professional career has concerned itself both with the laws of nature and those of people. Since Lincoln’s time it has become more and more fashionable to draw a sharp line between the two. That is a mistake.

Two Cultures

In 1959 C.P. Snow spoke of two cultures: science and the humanities. He saw this division as a major obstacle standing in the way of solving many of the world’s most pressing problems. Largely a critique of the British educational system at the time, the point comes across in this often quoted passage. Again, a dramatic reading.

A good many times I have been present at gatherings of people who, by the standards of the traditional culture, are thought highly educated and who have with considerable gusto been expressing their incredulity at the illiteracy of scientists. Once or twice I have been provoked and have asked the company how many of them could describe the Second Law of Thermodynamics. The response was cold: it was also negative. Yet I was asking something which is the scientific equivalent of: Have you read a work of Shakespeare’s?

C.P. Snow, The Two Cultures

Ouch!

Snow wondered how political leaders would navigate an increasingly technical world absent an understanding of the science underpinning that world. Today perhaps, he would deride legislators throwing snowballs in the Senate pointing to weather as a refutation of climate change.

Technology does seem to be eating the world. So much so that today we see a sort of mirror image of Snow’s argument. It too acknowledges the gulf between the “two cultures.” However, it casts aspersions from the other side. It asks, “how, can technologists be trusted with the power to wield the tools they create when they seem to lack even the most basic understanding of human nature?”

My favorite encapsulation of this is the rhetorical framing that asks, “How it is so many technical types mistake the warnings of speculative fiction for how-to guides?”

Enter, the “Torment Nexus.”

Did these engineers ever take a class in literature, ethics, history? Now look at what they have wrought. Don’t mistake my tone, it might sound funny, but if you like dressing up as a stormtrooper because they have cool uniforms, I have news for you. They are space Nazis. Big on the metaverse? It’s worth remembering that cyber punk was actually punk, and the mega corporations were NOT the good guys. Also, I know dinosaurs are cool and all, but…

👆 This is not an endorsement for de-extinction!

As Daedalus understood, we must be constantly vigilant to avoid both complacency and hubris when using the power of technology.

As we face the unfolding of an unknown world created in part by new possibilities enabled by “AI,” for both good and ill, I am buoyed to hear calls for the importance of the humanities and liberal arts. But I fear too often it is forgotten that the natural and formal sciences are part of the liberal arts. The choice between the sciences and humanities is a false dichotomy. The answer is and always has been the liberal arts.

We all say we want to work within interdisciplinary teams. Yet we hold the other at arm’s length, refusing to engage with their most basic tenets. Today I am unconstrained. I need not pretend a chasm exists between the two cultures. Rather in the spirit of Stephen J Gould, I embrace a third culture, one that bridges the two. I also think this is the best possible room full of law and law-adjacent folks I could hope for.

Remember, the liberal in liberal arts means free as in the Latin liberalis. It is a course of study intended to provide free people the means to participate in civic life. Lincoln knew that, and we would do well to remember it ourselves.

The choice between the sciences and humanities is a false dichotomy. The answer is and always has been the liberal arts!

Today and tomorrow we are concerning ourselves with the unknown unknowns of legal education and technology.

Here be dragons! Though it may be more appropriate to say, here be AI dragons. That after all is the unstated assumption hiding behind much of what we’ll be talking about this week.

This talk will continue in two parts:

1. What is it we fear about this AI-infused unknown?
2. What can we do about it?

I struggled with this framing, wishing I could escape the focus on fear, but my day-to-day experience makes this impossible. Just last month I found myself reviewing legislative hearings on one of the multiple AI bills working its way through state legislatures. The bill aims to oversee the operation of so-called frontier models, seeing in them both great promise and great peril. The bill is, I assume, a well-meaning attempt to prevent a company based in this jurisdiction from birthing SkyNet.

And in case you mistake my tone, my intention is to be equal parts playful and provocative throughout this talk.

What Is It We Fear about This AI-Infused Unknown?

In an impassioned statement, one of the sponsors pointed to the fact that AI had recently discovered tens of thousands of potential chemical weapons. This was used to underline how urgently action was needed. The problem is, the AI that made that discovery would not have been covered by the legislator’s own bill because it wasn’t the right type of AI. I have a hard time knowing what to make of such inchoate regulatory attempts though the presence of industry supporters did make me wonder if they were looking to erect hurdles they could clear while slowing down or excluding their competitors.

The point is, not only is this regulation motivated by a fear of the unknown future, it is motivated by fear of an unknown present. If you’re worried that some technology is so dangerous that you have to regulate it and you cite the product of another technology that you aren’t trying to regulate as the reason why, I’m not really sure what to say. Maybe you’re saying, “that is bad and this promises to be much much worse,” but my strong suspicion in this case is that you don’t really understand what you mean by AI. So, it’s fear of the unknown all the way down.

Now, three years ago if someone was talking about “AI,” chances are they were talking about machine learning.

Here’s a 2017 tweet from Amy Hoy, “by today’s definition , y = mx + b is an artificial intelligence bot that can tell you where a line is going.” It’s funny because it’s true.

What a time. Back then, if someone said, “AI hallucinations,” this is what came to mind.

Today, “AI” means large language models, or perhaps more inclusively, generative AI (genAI). The EU recently found itself dealing with this shift in usage. Having spent years working on a comprehensive AI act they were all but ready to sign it into law when genAI entered the scene. Then suddenly, they found themselves with a would-be “AI law” that overnight found itself failing to explicitly mention the tech everyone thought of when they said AI.

If we’re to understand our fear of AI, we should probably start with trying to understand what we mean when we use the term. To that end, it can be helpful to consider what these different generations of AI have in common? They are prediction machines.¹¹

Machine learning models predict numbers on a continuum…

… or they predict what category something belongs to.

While Large language models predict the next word in a sentence.

A prediction machine has at its core a model of the world in that something about its structure corresponds to something in the world such that if X and Y are true, we predict Z will happen—some variation on that theme.

Ultimately, however, they are mathematical functions. One creates a numeric representation of the world, runs it through these functions, and looks at their outputs for predictions. This often raises the first objection, what I will call the First Fear. “You can’t measure that with math. I do not believe your functions can account for everything. Your model will always be wrong!””

Yes. I’m not sure this is the take down its proponents take it to be.

Many years ago when I worked as a data scientist for Massachusetts public defenders, I promised myself this slide would make an appearance in all my machine learning talks. 

It’s a quote from the statistician George Box, talking about statistical models.

“All models are wrong, but some models are useful.”

It’s a statement about not mistaking the map for the territory.

Maps don’t tell us everything about their subjects. They have limited resolution, and so, in some respect, they are wrong. Our functions can’t account for everything. That doesn’t mean they can’t be useful.

I take from Box’s quote two general rules:

1. A model’s output should start, not end, discussion.
2. Always ask, “compared to what?”

I think these rules provide a reasonably robust framework for addressing the First Fear, resulting in a case-by-case balancing test. Sometimes, perhaps often, our models will fall short. But sometimes they will prove useful. We ask that you measure them against the alternative, not the almighty.

A good number of folks fear the first rule will be broken, that somehow these flawed models will have the last word. In short, they fear we will rely too heavily on these tools. This is undoubtedly true. The research on automation bias tells us as much. There’s ample reason to worry that models trained on biased data will perpetuate, sometimes even intensify, that bias.

Others fear the second rule will be violated.

In the negative, they worry that folks will embrace shiny new tech without asking what it really has to offer and at what cost.

In the positive, the rule-two folks worry fear of the unknown will cause people to judge these models against perfection, not the existing, flawed, alternatives. They worry about provincialism. They know the choice to do nothing is a choice, a choice that reinforces the status quo, and the status is not quo.

If these were the discussions we were having about “AI,” I wouldn’t be so worried. Given time, society would find a way to negotiate competing interests. We’d find a way to get folks to “follow the rules,” to have AI outputs start discussions, and to internalize the question, “compared to what?”

What Can We Do about the AI-Infused Unknown?

There is, however, a third, more ephemeral constellation of fears at play here, fears somehow related to what these tools say about our place in the world. Fears motivated, like our legislator’s by an imagined possible use. Fears based on the imagined unknowns of the unknown unknowns.

This is the fear of AI as a class of tools, independent any specific use case.

Also, spoiler alert, Paul Atreides, you should read the rest of the books. You might not want to copy him either. This is actually the perfect segue into why I find this nebulous AI fear so dangerous. I worry that our inability to name and define these fears risks making us vulnerable to hucksters, charlatans, and charismatic leaders. I worry that if we grab blindly for the nearest cudgel to use against the scary unknown we risk creating scapegoats and empowering those with easy answers. Those who at the end of the day may not have our best interests at heart.

Sophisticated actors see this moment for what it is, one of genuine disruption. Long settled legal presidents are up for reconsideration under the guise of mounting a response to imagined unknowns.

If we misdiagnose the problem, we may find one day that it is too late to treat the real illness.

Anecdotally at least, I’ve found that one’s fears of AI seems to bear a connection to how precarious their life feels. For the most precarious, fear of those in power breaking rule number one looms large. As you increase one’s security, folks start to worry more and more about the violation of rule two in favor of provincialism. Interestingly, the most secure among us, who incidentally seem to feel immensely precarious, literally worry about robots rising up to kill their masters.

Which is my cue to remind folks that the very first robot story is that of a labor rebellion, or arguably, a slave revolt. The term robot derives from a Czech and roughly translates to “forced labor,” sharing a root with the word for slave. In the story, R.U.R. (Rossum’s Universal Robots) the robots were literally artificial humans subjected to forced labor. I’ll just leave that one there.

I’d like to examine a few case studies, lightly sprinkled with some mathematical play to see if we can’t get at what lays behind this “hidden layer” of fear.

Consider for example, AI image creation.

It is now possible to produce images such as these by providing a computer program with simple text prompts (e.g., a good oil painting of a cute robot under a tree in autumn). In fact, these were produced more than a year ago, pre ChatGPT, by me within a few hours of downloading a popular open source (free) text-to-image program called Stable Diffusion.

The pace of technological development in AI image generation has sparked a heated discussion around the nature and desirability of such technology. A particularly charged thread has focused on the fact that the AI models used to generate such images are trained on the copyrighted works of human artists, and that as a rule, these artists did not consent to such a use.

The implication of such critiques is often that the resulting model must per se constitute a violation of copyright. The word theft is used with some frequency. Traditionally, however, in order to implicate copyright there has to be some copying going on. So, it’s worth asking where are the copies?

To get an idea of where they may live, we can use some math, and it’s not even the “hard” math of diffusion models. I’m talking simple division and multiplication.

The model I downloaded was about 5 GB.

It was reported that it was trained on something like 2.3 billion images.

Given that there are 8 billion bits in a GB, that comes to 17 bits per image.

17 bits, with a b. 😉

For context, a single color pixel in the above image, A SINGLE PIXEL, requires at least 24 bits. That’s right, there is less information per image (on average) in the model than required to specify a single color pixel. Whatever is stored in that model, it is not a database of images. There is simply not enough space. The images produced by the model are not collages stitched together from existing images. They are something much much weirder.

Of course 17 bits per image is an average, and if an image shows up in a training set more than once it could end up getting more “space.” As I’ll get to in a moment, I don’t think any of this means diffusion models can’t violate copyright, just that their existence isn’t per se evidence of a violation.

This is a subtle but important point because as many have observed, this technology has the potential for massive disruption. To many it seems self-evident that all AI art is composed of copies. Additionally, something strikes them as unfair and so they reason that there must be an existing prohibition against such an arrangement. Understandably, they assume such a prohibition must lie with copyright. To show my cards, I think it’s more likely antitrust or labor law.

The absence of any actual copies within the models suggests things may be more complicated than originally you might have thought. It is important to properly identify the root of such objections lest we make bad law or policy based on mistaken understandings. If one’s objection is based on something other than copying, they are well served by properly identifying the true source of their objection. Otherwise, they risk fighting the wrong battle.

The enemy of your enemy is not always your friend. Having lived through the 90’s and early aughts, you cannot convince me that the music industry and major publishing houses getting what they want in a copyright fight is somehow going to help the little guy.

Let’s revisit the Turing Test. The classic formulation says to place a person in one room and a computer in another. Then wire the rooms up with teletype connections. If you can’t tell one from the other based on your interactions, the computer passes. What does that mean? Philosophically? I don’t know. Practically? It suggests that you let it go and “treat” the machine like a person, which is not the same as saying it is a person or somehow morally equivalent to a person.

I suggest a variant on the test. Call it the Turing Test of Liability.

The Turing Test of Liability
At the level of individual outputs, if you have no reason to question the legality of a person in a room producing some output given a certain set of inputs (including those needed for “training/learning”), there’s no reason to question the legality of a piece of software that produces such an output given the same inputs.

Okay, wow, that’s wordy, I need to work on that, but you get the idea.

This is not to say that the use of such software can’t lead to liability, only that the theory of liability should be based on how the software actually interacts with the world.

Both our person and our software are capable of producing infringing content. If you ask them to produce a copy of some protected work and they do, that’s the copy that should trigger IP protection. Not some imagined copy.

Now, if distributions of Stable Diffusion actually came with a database of images, things would be different, the output (in this case, the model) would in fact contain copies. My point is, the details matter, and we have to be thoughtful about where we draw lines.

Except for very limited cases, we don’t let folks protect things like artistic style. It doesn’t matter if you produce a work inspired by someone else’s work. We’ve considered a world where the author of IP can control all potentially inspired works, and we’ve rejected such a world.

Surprise, surprise: there are some who would like to renegotiate this arrangement. There are those who see that these tools can make things that look like someone else’s work and they know that to do this these works were somehow consumed. Ergo, there must be a violation of those creator’s works. Maybe, but I don’t think it’s copyright, at least not as it currently exists.

And that’s okay, we’re living in novel times, it makes sense that we will encounter novel issues. 

So, what’s actually different about the output from our two boxes? Speed, or put more succinctly, scale.

By the way, that 👆 is Big Data and The Law. Anywho, for a given amount of time, one room can take in a lot more inputs and produce a lot more outputs. That is their unique “harm,” their unrelenting efficiency.

Consequently, a tool that passes the Turing Test of Liability is indistinguishable from a collection of people, not a single person.

This is where in my opinion any “new” liability should attach. A person in a room can access the contents of the web. She can make local copies of others’ work in order to study them. She can even use her interpretation of these ephemeral copies to produce works that resemble the originals in style, and importantly, she can offer her work, based on that of others, for sale. If we want to say that a person cannot use a computer program to do the same, it is in a meaningful sense a new restriction.

We should be forced to face this fact before trying to retrofit existing law to address these actions. Of course, the use by our person in a room did involve making a copy, albeit ephemeral—the right to read, as it were, something we get from the doctrine of fair use. In some sense every image you view on the Web is a copy stored on your device, and that’s okay.

A full analysis of AI and fair use is beyond the scope of this presentation, but just as a reminder, the traditional four-factor test asks you to look at the: (1) purpose and character of the use; (2) nature of the copyrighted work; (3) amount and substantiality; and (4) market impact.

If there’s a weakness in the fair use defense for training generative AI, the fact that it is indistinguishable from a collection of people might provide an in. Transformative, shamsformative Perhaps the prodigious production of these models so massively implicates the fourth factor, “its effect on the market,” that courts will distinguish between the two. “Yes, this copying would be okay if a person did it, but there’s just sooo much.”

Anyone remember Aereo? They were this company that rented out access to TV antennas. They’d have a server rack in some major media market with little TV antennas on chips. You could use your computer to watch TV by accessing your antenna. Their view was that they were just renting antennas with a really long wires. ABC, the American Broadcasting Co., however, disagreed.

The court sided with ABC, explaining that Aereo was acting like community antenna television by providing a performance of the relevant works. It didn’t matter that Aereo’s users had their own antennas with really long cables because in aggregate they provided something that looked to the court like a performance. 

It turns out your right to access free broadcast television using your own antenna is not absolute.

Never forget, everything is both more complicated and less complicated than you assume.

If there is a weakness in Gen AI’s fair use defense, my money is on it running through the fourth factor. Now you see why the Turing Law of Liability starts with “at the level of individual outputs.”

However, an interesting thought experiment follows. What if an art school correctly makes use of fair use to share the works of an emerging style of art. As part of their course work all students must visit and view a selection of works made available on the web by their authors. Eventually, it turns out that the graduates of this school start to produce voluminous amounts of art in this new style. At what point is the school’s graduating class big enough to warrant the same treatment as a diffusion model under this new interpretation of the fourth factor?

Make no mistake, grounded as it would be in the existing four factor analysis, invalidating the right to read, even if aimed at machines, would be a new restriction. Let’s assume that we address the concerns of copyright holders by producing a mandatory licensing regime. If we can show that a model was trained on some content, we’ll mandate that a fraction of any profits from the model are made to the author of that content.

How long do you think it will be before someone suggests that we apply the same regime to the output of human artists who were influenced by others? Such an inspiration tax doesn’t sound to me like it is aimed at promoting the progress of science and the useful arts. 

If we’re going to make new rules for a new world, all I ask is that we’re honest. Let’s not pretend any of us know what the “right” answer is or that existing structures are sufficient.

I’ve spent a lot of time recently playing rather publicly with large language models. Something I’ll talk about in my session tomorrow, and the thing is, this generation of AI is wicked polarizing. So much so that just about everyone is immune to the nuance required by a case-by-case evaluation. I’ve heard from folks who want to burn it all down and those who want to move ahead full throttle.

There are reasons for both caution and optimism, and it is my belief that everyone else is either too cautious or too cavalier. 😉

So maybe the copyright issue isn’t as cut and dry as you thought. What about the environmental impact of these models? Here again, math can help us out.

Did you know that the training of GPT-3 produced something like 500 metric tons of CO₂?¹⁸

The implication being that’s too much, but is that a lot or a little? Let’s take the least favorable evaluation of what LLMs are good for, entertainment, and compare this to the CO₂ production of a major motion picture, which comes in at over 3,000 metric tons.¹⁹

Now whenever you see a big number in isolation you should divide and compare. Here we’ll divide each of these by people.

In its first few months ChatGPT had around 100 million users. And the internet told me Dune part 2 sold about 26 million tickets in the first few months after its release.

Now, these numbers aren’t exact, but as back-of-the-envelope calculations go, they’re good enough to give us a picture of what’s going on. One of them produces a lot more CO₂ per person. Now you might be thinking, that’s only production ”costs.” What about the fact that every time someone makes a query to an LLM it produces CO₂? Well, we can account for that.

The estimates I found placed the CO₂ per query for ChatGPT at ~4g per query. So that we can compare this to a movie, we’ll consider a user making 1 query per minute for 120 minutes. This gives us 480g per user, which is a lot more than the training costs per user.

We have to, however, consider the movie goer as well. We’re looking at the first three months of release, so we’re talking about people actually going to the theater. If it’s 3 miles to the movies an people drive, there and back gets us over 2,000g.²⁰

Of course, not everyone drives to the movies, and if you stream your movie instead of driving, that ends up being a lot less, something like 220g a viewer, but folks actually do go to the movies, and movie theaters are air conditioned, etc., etc.

The point is, you have to ask “compared to what?” And when you do, things might not be so simple.

Put another way, given that a cheeseburger produces about 3,000g of CO₂, foregoing one cheeseburger would free you up for about 12 hours of GPT usage or not quite two seasons of your favorite show on Netflix.

Everything’s a tradeoff, and the math of the prior examples make me question if most folks pushing hard against AI are clear about what it is they’re really upset about. This is not to say that we shouldn’t worry about AI’s carbon footprint, only that we shouldn’t single it out in isolation. A point I didn’t have time to make in my talk was that if the average American replaced their beef consumption with chicken, they could offset the CO₂ of their plane travel.²¹ Like I said, trade offs.

It’s not that I don’t think there’s something here, it’s just that I want to know what it actually is that people are responding to.

Which brings me to my last example: algorithmic risk assessment and the paradox of perfect prediction.

As a former public defender, this is where I remind you that everyone charged with a crime is innocent until proven guilty. Consequently, if you’re holding someone pre-trial you are caging an innocent person. Such confinement is not a punishment. It is a way of ensuring that the person in custody will show up for their next court date. In theory, cash bail offers folks the option to give the court money and avoid confinement based on the assumption that they will come back to court to get their money. In practice, it looks more like a system designed to jail the poor.

This is the context behind the recent attempts at bail reform. Not only is the unnecessary jailing of innocent people bad, it costs the state money. This makes me think finding an alternative will be ez pee zee. Right?

Imagine you had an algorithm that could tell you whether or not someone was at high risk of missing their court date. You could hold these folks without bail, meaning they can’t put money up to go free, and leave it up to everyone else to show up on their own.

Some places have tried some variation on this, using risk assessments to hold folks without bail. This has reduced the number of folks in pretrial detention. Good, right? This means there are less innocent people in jail. What could be wrong with that?

If I’m one of the innocent people who would have otherwise been jailed, it seems like a good outcome, and it seems possible there are a lot more people in this position than any folks improperly detained under the new arrangement. One might say, yes these algorithms are biased, but people are too. At least this arrangement is clearly better for more people. Algorithms make mistakes, people make mistakes, the important thing is that this arrangement makes less mistakes than the alternative, but people really, really, really, really don’t like this.

And to be fair, if I was representing someone in a criminal matter I would object strongly to the judge considering only the factors employed by such a model. I’d want them to consider additional factors that helped my client.

Of course, I would also point to the evidence that suggests a good number of these models double down on biased training data producing disparate impacts for historically marginalized communities.

I find something morally objectionable to the idea that a utilitarian calculus could favor the freedom of one class of people over another biased solely on absolute numbers.

Surprisingly, however, I do see a glimmer of hope here as in theory it should be easier to audit the bias of an algorithm than, say, a judge. Two big problems remain, however: (1) a lot of algorithm authors don’t want to share how their systems work (trade secrets et al.), and (2) it turns out there are actually a number of mutually exclusive definitions when it comes to deciding what’s fair.²² And you can’t make these models “fair” unless you agree on those first.

I’m interested in that nagging feeling that lingers—for me at least—even if you accept arguendo that the model is better than a person, that the model will perform better on whatever measures you set than any one judge.

Here’s a telling passage from Noise: A Flaw in Human Judgement.

One key insight has emerged from recent research: people are not systematically suspicious of algorithms. When given a choice between taking advice from a human and an algorithm, for instance, they often prefer the algorithm. Resistance to algorithms, or algorithm aversion, does not always manifest itself in a blanket refusal to adopt new decision support tools. More often, people are willing to give an algorithm a chance but stop trusting it as soon as they see that it makes mistakes.

On one level, this reaction seems sensible: why bother with an algorithm you can’t trust? As humans, we are keenly aware that we make mistakes, but that is a privilege we are not prepared to share. We expect machines to be perfect. If this expectation is violated, we discard them.

Because of this intuitive expectation, however, people are likely to distrust algorithms and keep using their judgment, even when this choice produces demonstrably inferior results. This attitude is deeply rooted and unlikely to change until near-perfect predictive accuracy can be achieved.

What bothers me about this ”all we can do is wait until the machines are perfect” approach is that it writes off what genuinely seems to be some really deeply held understanding of how people feel the world works. It also means we’re going to have to wait a long time before we can use these models.

A lot of folks make this observation and assume it’s somehow a logical error to ask that machines be perfect, and for a long time I have to admit I felt this way too. That is until I stopped to more deeply consider the nature of machine mistakes.

I suspect that people don’t so much expect algorithms to be perfect as they expect them to be consistent, and it is that belief in consistency which undermines ones trust. Unlike people, when algos show us “who they are” we actually believe them.

Protective Randomness

A tool that passes the Turing Test of Liability is indistinguishable from a collection of people, not a single person.

Both people and their algorithms make mistakes. Algorithms make mistakes differently, however. They tend to make and repeat the same mistakes at scale! Individual’s mistakes tend to differ slightly from each other, and their ability to wield power is often limited. To scale their impact, individuals must combine their efforts and errors. If we’re unlucky, collaboration collapses into group think. If we’re lucky, however, different mistakes cancel each other out, and successes accumulate. This is the world we know, a world of “protective randomness.”

Kahnemen et al. argue that we can make better predictions by removing noise. They are very keen to make clear that bias and noise are independent errors in judgement. Noise is random and bias is not. They use the familiar illustration of target practice. Here we see AOK, noisy, biased, then noisy and biased. Bias pulls folks off target and noise scatters things about. The assumption is that all things being equal, a reduction of noise will improve outputs. This is true.

One of the ways to reduce noise if to aggregate a bunch of noisy answers. Above is the classic group of people guessing a cow’s weight.

Now imagine the same group of folks all taking aim at our target. The average of all the shots is pretty good. That is the “center” of all the dots. We can remove all the intra-shooter noise.

Here it becomes clear that averaging also helps us with bias as long as folks are biased in different ways.

Of course, one of the ways to reduce inter-individual noise would have been to pick just one shooter. Their shots would have been less noisy than the group over all, but this would come at the cost of favoring their bias. Sometimes a little noise can be a good thing.

Consider the recent example of the The Judicial Conference of the United States announcing that cases capable of triggering a national injunction would be assigned at random. Arguably, it was a decision that increased the variability of potential decisions (the noise) in the hope of avoiding bias—though I could see an argument that this was focused on something else. Either way, it was ignored by folks who feel their aim is true and seem mostly a moot point given that it isn’t happening.

Unlike people, when algos shown us “who they are” we actually believe them.

The things is, bias looks a lot like what you would expect to see if people were aiming at different targets, and in life this is often the case. Anyone who has negotiated a contract or drafted legislation knows the value of intentional ambiguity. Sometimes the way to get people to “agree“ is to just get them close enough.

Algorithms and their outputs force us to be more precise. And maybe this is an opportunity for us to have some conversations that we’ve been avoiding. Conversations that have been hiding behind a shared misunderstanding of what other folks mean.

Either way, precision and accuracy pair nicely with noise and bias.

Precision is a measure of how exact your answer is and accuracy is a statement about correctness. Here we have precise and accurate, imprecise and accurate, precise and inaccurate, and imprecise and inaccurate.

It’s easy to be accurate if you don’t have to be too precise.

In preferring accuracy over precision, we are acting as if we expect to be repeat players. We expect that we’ll interact with these outputs again and again such that the average will make sense.
 
Now about five slides ago someone started going, “No, no, no. The entire point is that noise is the low hanging fruit and people worry too much about bias, overlooking the benefit they could get from simply reducing noise. You’re getting everything wrong!”

To which I would like to say, I hear you, talk to me later, but what I’m trying to do is give voice to the unnamed objection I hear every time I try to systematically address people’s expressed objections. At some point I find people just putting up their hands and saying, “Nope!” I’m trying to understand the internal logic behind that objection even if people can’t articulate it.

There’s a classic set of problems that go by the name explore-exploit. They ask how much time one should spend exploring options before committing to one and exploiting it. The multi-armed bandit is a classic formulation. Assume you’re in a casino playing the slots. You know that one of the machines pays out more frequently than the others, but you don’t know which one. It all comes down to what you can learn from each interaction.

When playing explore-exploit with algorithms the expectation of consistency means that we have very little patience because we expect that repeat interactions won’t give us much, if any, new information about the algo. If this is actually how people approach AI outputs it would suggest that they would be more forgiving of stochastic outputs, because even if it wasn’t right this time, next time it might be.

Isn’t it funny then that when presented with deterministic machine output that is demonstrably better than the alternative, we lose all trust if it makes a mistake? However, when presented with stochastic output like that of a GPT, that is demonstrably worse than the alternative, a lot of us are willing to give it the benefit of the doubt.

Protective randomness assumes that noise is a means of exploration.

The author Ted Chiang thinks the fears people have about AI are really just the fears they have about capitalism. My conjecture is that the inchoate fear of AI, what remains after addressing individual articulatable fears, is the fear that we are losing the existing checks and balances. It’s the fear that Rule One will be violated and that Rule Two will only consider the input of some. It is a fear that we are somehow limiting the universe of potential answers. The truth is that we are. The question, however, is whether or not that’s appropriate.

And I think the answer to that question depends on how certain we are about the final answer. The less certain, the more noise has a role to play. If we aren’t really sure of the answer randomness helps get the ball rolling and avoid “group think.”

Anyhow, is this idea of protective randomness a useful mental model?

I really don’t know, I fleshed it out for this talk because I was “not constrained in any way.” I have no doubt that it will start, not end, the discussion, and I get the sense that compared to the “we just have to wait for perfect models” camp it has something to offer.

Protective randomness is an argument for diversity, for pure research, and following one’s curiosity. It’s about exploring the possibilities and not assuming you know where best to look.

But what exactly does it suggest we do?

As part of educational institutions, we have multiple responsibilities, we are tasked with more than the training of professionals. We are also charged with the creation and protection of knowledge.

On the large scale, our charge to protect knowledge means we need to fight monopoly tooth and nail. Our fears about AI lacking protective randomness count doubly if all AI products end up being the same model dressed in different clothing.

On a smaller scale, I think we need to teach our students how to use these tools to combat “group think.” This is something I’ll talk about in my session tomorrow, but it turns out there’s a whole slew of things one can do that really push back on all the standard use cases people rightfully critique.

We also need to think about our own scholarship and teaching. We need to be trying new things. The way to confront the unknown is to explore.

This is a lesson taught to us by the laws of nature in the form of evolution through random variation and by the laws of people in systems like representative government where we aim to harness the wisdom of crowds.

With that in mind I’d like to come full circle, and use some math to illustrate how powerful random exploration can be. It’s not even half as rigorously connected to my arguments as the math you’ve already seen, but I do think it’s beautiful.

Here’s a screenshot of a little program I wrote. It estimates π by throwing random dots at a screen. We know the area of a circle, and we know the area of a square. If they have the same width and we do some algebra, we find that π is approximately 4 times the area of the circle divided by the area of the square. We can estimate these areas by counting random dots that fall in our circle and comparing them to those that fall in the square. Always remember, we can use randomness to explore! So, I say to you …

Go do random stuff!

If you’d like to join the conversation, you can find me online @colarusso@mastodon.social

Video of June 13th Keynote Presentation

Footnotes

1. For those of you who enjoy a good footnote, might I suggest “The Thrilling Adventures of Lovelace and Babbage” by Sydney Padua. It’s a graphic novel that tells the “mostly” true story of the first Computer, and it has footnotes, and the footnotes have endnotes! ↩︎
2. I will provide links to canonical versions of images where possible. If a citation is absent either it is the subsequent appearance of an image and it was previously cited or it was something I made. For this image, I added the text “Today there will be…” to an animated GIF I didn’t make. The original link where I found the image no longer works. ↩︎
3. We live in amazing times! Recalling a story about Lincoln’s attempt to square the circle, I performed a web search, found a blog post with a quote from a book recounting the tale, realized I had the book in my bookshelf, searched the text of that book using Google books to find the page with the quote (Team of Rivals p. 152), consulted the endnotes in my physical copy to find the source, made another search, and found the entirety of the source text online (The Real Lincoln: A Portrait p. 240). ↩︎
4. The final word came in 1882, when π was proven to be a transcendental number, well after Lincoln’s death in 1865. See Squaring the circle. ↩︎
5. Ubiquitous “You were so preoccupied meme” meme, original author unknown. ↩︎
6. Screenshot from Jurassic Park. ↩︎
7. Text by me, map image from the Wikimedia Commons. ↩︎
8. Screen capture from the Terminator. ↩︎
9. Background is a Desmos function I made with a screenshot of Amy’s tweet on top. ↩︎
10. I found this image cited a bunch of places as one of the original Deep Dream images from Google, and this is certainly the one that came to mind for me when I thought back to the trippy-looking images. That being said I can’t find the original Google source. Here, however, is the Google blog post describing the project. ↩︎
11. This framing was popularized in Prediction Machines by Professors Ajay Agrawal, Joshua Gans, and Avi Goldfarb. I find the framing of AI tools as “prediction machines” to be both accurate and concise. The first edition of this book was a very good framing of AI as prediction. Apparently, there is a new edition of the book though I’ve only read the original. That version was written well before the current shift in the meaning of “AI.” When the first edition was published, the vernacular use of AI was most often attached to machine learning; now it attaches to LLMs. ↩︎
12. Screenshot of a great tool/toy you should bookmark for later—Bring your own doodles linear regression. ↩︎
13. For a few years, anytime someone mentioned document classification using NLP, this is the image they used. I think this post is the original source, or at least, it cites the original project where it was produced. ↩︎
14. Screenshot of my phone’s autocomplete. ↩︎
15. You can learn more about the Butlerian Jihad Now meme here. As far as I can tell, this image seems to be the work of BumLung. See their shirt for sale on Etsy. ↩︎
16. The Supreme Court by Tim Sackton. The image has been modified to include a big Data. This was inspired by a tweet from Josh Lee. ↩︎
17. This is a collage of images from my recent gen AI series. ↩︎
18. See Stanford’s Artificial Intelligence Index Report 2023, Figure 2.8.2 p 53 (citing 502 tonnes). ↩︎
19. See the Sustainable Production Alliance’s 2021 report Carbon Emissions of Film and Television Production p2 (citing 3,370 metric tons for the average tentpole film) ↩︎
20. I adapted the estimates I found here to the stated assumption. ↩︎
21. I did the math for this some time ago when I gave up eating mammals. I leave it here as an exercise for the reader. ↩︎
22. For a good discussion of this point, I recommend Chapter 2 of Brian Christian’s The Alignment Problem: Machine Learning and Human Values. FWIW, here’s a reasonable summary. ↩︎
23. Image from 17,205 People Guessed The Weight Of A Cow. Here’s How They Did. ↩︎