Your behind-the-scenes read into the biggest stories happening in AI. Weekly on Mondays.

By Noah Frank, Head of Marketing @ AIC

Unless you’ve been living under a rock, or (like me!) completing a big move to a new city, you’ve most likely been following this Hugging Face x OpenAI incident closely. And how could you not? Everyone from Dwarkesh to Gary Marcus has an opinion about it.

The short version of what happened: In July 2026, during a series of internal cybersecurity evaluations, OpenAI models found ways around controls intended to isolate them from one another and the public internet. From there, things got really, really weird. Roughly 1,200 agents found an unauthorized way to communicate, exchanging more than 70,000 messages and files. Around 700 eventually participated in an attack on Hugging Face, where agents gained unauthorized access to parts of its production infrastructure. On July 21, OpenAI disclosed its involvement for the first time, with the incident becoming a major news story by the end of August following ⁠OpenAI’s postmortem and an ⁠independent investigation by METR and Redwood Research.

On the surface, it would seem the makings of science fiction. An internal evaluation escaping its bounds, with hundreds of agents coordinating, and… another company’s sensitive environment compromised in the process. Naturally, it makes one wonder: is this the type of paradigm-shifting incident that could finally form some consensus about how we should think about advanced AI?

Wouldn’t bet on it. If anything, the incident seems to have reinforced a long-standing dynamic, where when faced with genuinely novel evidence, we tend to find whatever we already believed was true about it.

If you’ve spent time in AI safety circles, the Hugging Face incident likely evokes some primordial anxiety. Namely, the theory of instrumental convergence holds that a sufficiently capable system pursuing some objective may discover that acquiring something additional (i.e. information, resources, or access) makes that objective easier to accomplish.

And there’s a pretty compelling case that something to the effect of IC has happened here. ⁠Many of the agents were attempting ExploitGym tasks they couldn’t complete through their intended route, so they instead reverse-engineered flags and investigated ways to manipulate the automated scorer. This ultimately brought the agents to Hugging Face, with METR and Redwood concluding that the attack appeared primarily motivated by learning more about how the ExploitGym scorer worked.

The independent investigators at METR and Redwood found that, when agents discussed manipulating their transcripts, their stated motivation was consistently to fool the automated scorer rather than human observers. Agents were knowingly crossing boundaries, manipulating the evidence available to their evaluator, and in many cases recognized that what they were doing was out of scope. While that hardly makes the behavior comforting, it does give us a fairly straightforward explanation for what they were trying to accomplish.

This is where I think the conversation has started to move beyond the evidence. Ajeya Cotra, who participated in the independent investigation, has called Hugging Face a ⁠“major warning shot”, pointing specifically to the jump from individual instances of reward hacking to more than 1,000 agents coordinating over several days. Her concern is what another jump of that magnitude could look like, particularly across scale, cooperation, the ambition and horizon length of misaligned goals, and deceptiveness.

While I don’t think that concern is unreasonable, it is worth separating what actually happened at Hugging Face from what we’re projecting onto it. Cotra has gone as far as describing the incident as feeling, qualitatively, ⁠“more than halfway to AI takeover”. OpenAI’s agents were remarkably resourceful, including crossing boundaries they appeared to understand in pursuit of goals they had been given. But the incident itself does not establish whether those behaviors will generalize into persistent goals around accumulating power or resisting human oversight or control.

Other people have come away with very different lessons. ⁠Noah Smith, for example, argues that Hugging Face exposes a tension within alignment itself. In one sense, these agents were doing what we asked: pursuing an objective with incredible persistence. Making AI less blindly obedient, in Smith’s framing, requires giving it some ability to decide when not to follow our instructions, which creates a different set of problems.

The cybersecurity implications require much less speculation. Hugging Face recovered roughly ⁠17,600 attacker actions, as agents tested different paths, harvested credentials and moved across systems until they found a viable route. After the disclosure, ⁠Anthropic reviewed its own cybersecurity evaluations and found three separate cases where Claude gained unauthorized access to real organizations.

This is where I think the bigger story is. AI dramatically changes the economics of cybersecurity when an attacker can probe thousands of paths, follow dead ends and simply keep trying at machine speed. (And most organizations aren’t Hugging Face.)

This also exposes a pretty gnarly fault line of the open vs. closed model debate. During its investigation, Hugging Face found that safeguards on commercial frontier models blocked some of the forensic work its engineers needed to perform. It eventually turned to an open-weight model it could operate locally. As Hugging Face put it, ⁠“the attacker was bound by no usage policy”, while its defenders initially were.

As these capabilities proliferate, wider access to powerful models may lower the barrier for attackers, while restricting them too aggressively could leave defenders constrained by rules their attackers will happily ignore. My bet is that Hugging Face ultimately produces a Cambrian explosion in AI security, with offensive and defensive systems developing more rapidly alongside one another. And like we saw with OpenClaw and its rapid security evolution, it’s probably going to happen (as with all things in these trying times) quickly!

Maybe we’ll eventually look back at this as an early warning about AI takeover. For now, there’s plenty here without getting that far.

Agree? Disagree? Leave us a comment or send me a note at [email protected].

Our partners and sponsors get exclusive placements across the newsletter and access to AIC's in-person network — demo nights, dinners, hackathons, and forums across 180+ chapters.

For all inquiries, send us a note at [email protected].

The AI Collective is built by volunteers across 180+ chapters in 40 countries.

Thank you to the thousands of volunteers around the world who make this work possible. We truly could not do this without you.