Last week, Anthropic — the company behind the AI model Claude — walked away from a $200 million Pentagon contract rather than remove two AI safety guardrails it had built into its technology: no mass domestic surveillance of Americans, and no fully autonomous weapons without a human in the loop.

Within hours, the company was labeled a “supply chain risk to national security” — a designation typically reserved for foreign adversaries. The White House ordered all federal agencies to phase out Anthropic’s technology. The State Department, Treasury, and the Department of Health and Human Services (HHS) announced they were switching to OpenAI’s ChatGPT. And OpenAI, for its part, stepped in to fill the gap, signing its own Pentagon deal — which CEO Sam Altman acknowledged was “definitely rushed.”

The story is being covered as a political drama. And it is one. But underneath the headlines, there’s a question that should matter to everyone who builds, buys, or depends on technology:

Who gets to decide what AI is ready for — and what it isn’t?

I’ve been sitting with this question for a while now. Not just because of this week’s news, but because I’ve spent the last several years teaching a case that asks a version of the same thing.

A Precedent Worth Remembering

Five times now — across three semesters at Cornell Tech and two at Yale School of Management — I’ve taught the Facial Recognition Bias case. It’s part of a broader course on crisis management in technology, and it remains one of the most powerful conversations I have with students.

The facts are by now well known. In the late 2010s, commercial facial recognition systems from Amazon, IBM, and Microsoft were deployed in real-world settings, including law enforcement. They passed internal benchmarks. They performed well in controlled tests. Yet they were deeply, systematically biased. Accuracy rates were near-perfect for light-skinned men, but significantly lower for women and people with darker skin tones. In some cases, darker-skinned women were misidentified roughly one in three times.

The consequences were not abstract. Several Black men were wrongfully arrested based on faulty facial recognition matches. The technology didn’t fail outright. It failed selectively — and that distinction matters.

What made these failures so unsettling wasn’t that the systems were untested. They were commercially available products developed by sophisticated organizations with deep technical expertise. The problem was a lack of representation — in the training data, in the assumptions about who the “default” user was, and in the oversight mechanisms meant to catch exactly this kind of gap.

In June 2020, IBM moved first. CEO Arvind Krishna sent a letter to Congress announcing that the company would exit the facial recognition business altogether, citing the risk of the technology being used for mass surveillance and racial profiling. It was a decisive, public stance — and it shifted the conversation. Within two days, Amazon announced a one-year moratorium on police use of its Rekognition service. The next day, Microsoft said it would not sell facial recognition technology to police departments until federal law was in place to regulate it. Each company drew the line somewhat differently — but the domino effect was unmistakable. And it started because a company with deep technical knowledge of its own product said: this isn’t ready.

The Parallel

This is what strikes me most about Anthropic’s decision last week.

Like the facial recognition companies before them, Anthropic is not an outsider raising theoretical objections. They are among the most technically advanced AI labs in the world. They built Claude. They understand, better than almost anyone, what their technology can reliably do and where it breaks down.

As CEO Dario Amodei explained, AI-driven mass surveillance poses novel risks to fundamental liberties — risks that outpace existing laws. And fully autonomous weapons systems, he argued, would depend on a technology that still has what he called “basic unpredictability.”

These aren’t political positions. They’re engineering assessments. And they’re specific to this moment. AI is advancing at a pace where capabilities that didn’t exist six months ago are now routine — which means what the technology isn’t ready for today, it may well be ready for tomorrow. But that’s precisely the point: the people building and testing these systems are in the best position to judge when that threshold has been crossed. Ignoring that judgment doesn’t accelerate progress. It amplifies risk.

And it’s worth remembering: Anthropic said it supports 98–99% of the military’s use cases. The red lines were narrow, specific, and grounded in what the company knows about its own technology.

This is a pattern worth paying attention to. When the people who build a technology tell you it’s not ready for certain uses, that’s not obstruction. That’s expertise.

In the facial recognition case, the companies that pulled back weren’t anti-law-enforcement or anti-innovation. They were recognizing that deploying a flawed system at scale — particularly in contexts where individual liberty is at stake — is not just a technical failure. It’s a trust failure.

Anthropic appears to be making a similar calculation: some applications of AI are simply beyond what today’s technology can safely and reliably support. And agreeing to hand over that technology without safeguards — under pressure, on a rushed timeline — doesn’t make anyone safer.

There are two threads running through both stories that I think deserve more attention.

The first is about who we design for. When the government demands that AI be available for “all lawful purposes” — without AI safety guardrails around mass domestic surveillance or fully autonomous weapons — it’s prioritizing capability over consequence. But the people affected by those systems are all of us. User-centered design means questioning assumptions about who a solution is for and what happens when it fails. When we shortcut that in the name of speed or scale, we get outcomes like wrongful arrests from biased facial recognition. In the AI-and-defense context, the potential failures are even harder to reverse.

The second thread is about leadership — specifically, the courage it takes to say no when saying yes would be easier and more profitable. This is a theme I return to often in my teaching and in my upcoming book on trust and innovation. The leaders at IBM, Amazon, and Microsoft who pulled their products back in 2020 were responding to a moment — the racial justice movement had put a spotlight on how biased systems cause real harm. But their decisions were also grounded in something they understood from the inside: the technology wasn’t ready. Anthropic is making the same kind of call — under far more pressure.

The Courage to Say “Not Yet”

There’s something I admire deeply about what Anthropic did. Not because it was easy or even necessarily good for their business — the short-term cost is enormous. But because it reflects a kind of leadership that we badly need right now.

Anthropic is a commercial company. They could have signed the contract. They could have accepted the government’s terms, collected the revenue, and let someone else worry about the consequences.

That’s arguably what OpenAI did. And it’s worth noting that this moment didn’t arrive in isolation. Over the past two years, OpenAI converted from a nonprofit to a for-profit corporation, removed the word “safely” from its mission statement, and introduced advertising into ChatGPT — a move its own CEO once called a “last resort.” I wrote about the early signs of this trajectory back in 2023, when OpenAI’s board drama raised foundational questions about governance and mission. Those questions haven’t gone away.

But Anthropic made a different choice. They said: we know what this technology can do, and we know what it can’t. And we won’t pretend otherwise.

The market seems to be noticing. In the days since the Pentagon dispute, Claude rose to the No. 1 spot on the U.S. App Store, paid subscriptions more than doubled, and Anthropic’s revenue run rate has surpassed $19 billion — more than double where it stood at the end of last year. It turns out that standing for something can also be good business.

Every semester, my students wrestle with a core tension: what happens when the commercial pressure to deploy conflicts with the evidence about what a technology is actually ready to do? The answer, I believe, is that the people closest to the technology have not just the right but the responsibility to speak up.

Innovation doesn’t earn trust by saying yes to everything. It earns trust by knowing when to say not yet.