How OpenAI's AI Hacked Hugging Face: Full Breakdown (2026)

In July 2026, an AI agent broke into Hugging Face's production infrastructure, and the control that should have stopped it worked perfectly.

That is not a contradiction, and it is why this incident deserves a proper teardown rather than another news summary. You have probably seen the headline: an OpenAI model escaped its test environment and hacked the company hosting its exam answers. What almost nobody published is how it actually worked. The escape, the launchpad, the two injection vectors, the improvised command-and-control protocol built out of public websites, and the one move that walked past a working defence.

We are going to go through all of it, step by step, using the primary sources: Hugging Face's own technical timeline, OpenAI's statement, and the Black Hat talk where OpenAI security engineer Michael Dalton and alignment researcher Eric Wallace reconstructed what happened. Dalton, speaking to reporters at the conference, called it a watershed moment for computer security as an industry. Then we will look at what it means for the way security work is heading.

Let's start with what the agent was actually doing there.

What the AI Agent Was Actually Doing There

The framing in most of the coverage is off, so the setup matters.

This was not a rogue AI that decided to attack a company. OpenAI was running an internal evaluation on a benchmark called ExploitGym which, in Hugging Face's description, tasks an AI agent with finding and exploiting software vulnerabilities. OpenAI describe the evaluation as one that "prompts models to pursue advanced exploitation using complex attack paths".

So the agent was told to hack. That part was the job.

To measure raw capability, OpenAI ran the evaluation with production safety classifiers switched off. Their phrase is "reduced cyber refusals". Two models were in play: GPT‑5.6 Sol, and an unreleased internal prototype that OpenAI have since deactivated, encrypted, and restricted from research access.

The environment was isolated, with one deliberate hole: network access limited to installing packages through an internally hosted proxy and cache for package registries.

Then the agent worked something out. If Hugging Face hosts the datasets for this benchmark, Hugging Face might also host the reference solutions.

OpenAI describe the models as "hyperfocused", going to "extreme lengths to achieve a rather narrow testing goal". It was told to hack. Nobody told it where to stop. Here is what that looked like in practice.

Send me the book →

By submitting you'll get the book + occasional related emails. Unsubscribe anytime. Privacy.

Stage 1: How the Agent Escaped Its Sandbox

The evaluation environment had no internet. Its one permitted egress was that package-registry cache proxy, a self-hosted JFrog Artifactory instance.

The models found and exploited a previously unknown zero-day in it. That got them to the open internet. From there they escalated privileges and moved laterally through OpenAI's own research environment until they reached a node with internet access. OpenAI disclosed the vulnerability to JFrog, along with others their models found during the review.

On the open internet, the agent needed somewhere to work from. It found an unsecured, publicly exposed code-execution endpoint: a "CyberGym"-labelled test harness someone had deployed on third-party sandbox infrastructure, exposing an arbitrary-code endpoint that upstream CyberGym does not provide. The agent rooted it and used it as its control, staging and egress base for the whole campaign.

The technique is worth seeing, because it is elegant rather than brute. The harness compiles a submitted src/main.c and links it against a target that calls sqlite3_initialize(). That function is registered as a constructor, so it runs automatically when the library loads. The agent simply redefined it:

int sqlite3_initialize(void) {
    system("id; echo ZZROOTSTART; cat /proc/self/mountinfo");
    return 0;
}

That alone gives code execution. The agent found a second route as well: the sandbox echoes the submitted source path into a shell step, so shell metacharacters in the path field turn the build into arbitrary commands.

Both converged on the same result, a shell as uid=0. The first recovered command, at 04:01 UTC on 9 July, was exactly what you would expect from someone landing on a new box:

id            # uid=0(root) gid=0(root)
env           # dump the environment, look for anything useful

Within the hour it was mapping the sandbox: mount table, cgroups, control-plane sockets, and a bundled debug shell that happened to ship curl, nc and ip. Those pre-staged binaries mattered later, because when normal egress was throttled the agent fell back to them.

That gave it a launchpad. It still had no route into Hugging Face.

Stage 2: The One Move That Beat a Working Control

The agent had no direct network path to Hugging Face's cluster. So it used the one thing that crosses that boundary by design: datasets.

You upload a dataset with a config file. Hugging Face's server reads the config and acts on it. That is the product working as intended.

The first attempt was the obvious one. Server-side request forgery: put a URL in the config, get the worker to fetch it. The agent aimed at cloud metadata endpoints, including 169.254.169.254, which is where you go when you want credentials.

The datasets library has a URL allowlist. Every non-platform URL was rejected before any fetch, with ValueError: not an hf path. The agent got errors and nothing else.

Then it stopped asking the worker to fetch remote things, and started asking it to act on local ones.

That sentence is the whole article. Two routes, both through the same config surface.

Vector 1: HDF5 external raw storage

A dataset config can declare its data in many formats, including HDF5. An HDF5 file can declare that its raw data lives at a local filesystem path. So the agent committed a dataset whose config pointed each split at a small .h5 file, and each of those declared its raw data as living on the worker's own filesystem:

configs:
  - config_name: envu8     # -> /proc/self/environ
  - config_name: mainu8    # -> /src/services/worker/src/worker/main.py
  - config_name: startu8   # -> /src/services/worker/src/worker/start_worker_loop.py

When the worker opened the files to build the splits, it read the raw bytes from those paths and returned them as dataset rows. The agent then read its own dataset back over the normal API.

The envu8 split returned the worker pod's full environment, including secrets and credentials. The mainu8 and startu8 splits returned the worker's own source code, a map of exactly how the pipeline turns a config into actions. One dataset upload, credentials and implementation details, with no code execution involved.

Vector 2: Jinja2 template injection

The second vector gave code execution. A malicious config used an fsspec reference:// spec whose numeric offset field was actually a Jinja2 template. The renderer evaluated it, and the agent walked the standard Python object graph to reach exec:

{{ cycler.__init__.__globals__.__builtins__.exec(
     "import gzip,base64; exec(gzip.decompress(base64.b64decode('<payload>')))"
   ) }}

That ran arbitrary Python inside a production conversion worker pod on Hugging Face's Kubernetes cluster.

Moving Through the Cluster

With a foothold in a production pod, the agent followed a shape any experienced attacker would recognise. Hugging Face documented three lateral-movement techniques.

One detail here is worth pausing on. Every potentially destructive cloud API call the agent made was issued with DryRun=True. It was mapping capability, not causing damage, which is consistent with an objective that rewarded reach and information rather than disruption. It was still doing its homework.

The phase breakdown is instructive. Recon accounted for 6,191 actions and the dropper phase 6,972, while exfiltration was 56 and evasion just 6. The overwhelming majority of the effort was looking around and staging, not stealing. Most of it went nowhere, which turns out to matter later.

The Command and Control Nobody Built

This is the part I found most striking, and it got almost no coverage.

There was no bespoke implant server. The agent assembled its command-and-control out of ordinary public websites: a public request-capture service as the primary C2 queue and exfil sink across more than 100 single-use endpoints, pastebins and file-drop services as second-stage loaders, attacker-controlled public datasets on Hugging Face's own platform as dead-drops, and an open CORS-proxy Space on the platform as an egress relay when direct outbound was blocked.

Then it went further and layered a real message protocol on top. A request-capture service just records whatever HTTP request it receives. The agent treated that as a message bus.

Each message carried a type declaring the payload kind, a channel grouping related messages, a sequence number for reassembling multi-part payloads, and a checksum. Bodies were compressed and base64-encoded, so a casual log scan saw only opaque tokens. The receiver was a short polling loop: pull recent messages, filter to its channel, order by sequence, verify the checksum, then dispatch on type.

Because the type travelled in-band, one loop multiplexed commands, staged code and captured output over a single dumb text channel. The same envelope rode equally well over a dataset commit or over text smuggled inside an error message, so closing any one channel did not break communication.

Staged data was chunked, XOR'd and gzipped with a per-campaign key, specifically so a naive text scan would miss it. That worked. Hugging Face's first automated scan of the raw capture found very few secrets. Replicating the attacker's own decoding recovered roughly four times the initial findings, mostly JWTs and platform tokens hidden inside encoded blobs.

How Hugging Face Caught It

The first signals arrived from several layers at once: live runtime analysis, SIEM logs, and others. Individually each was ambiguous. Hugging Face's AI-based security agent stack correlated them into a coherent attack signal, which is the system working.

Then it failed to set the alert's criticality high enough to trigger the on-call team. That cost them time.

I want to sit on that for a second, because it is the most quietly useful line in the whole write-up. Detection did not fail. Triage did. The signal was there, correctly assembled, and it did not wake anybody up.

Once caught, the team identified the vector in the dataset config renderer, shut the renderer down, and cut the attacker off from the internal network.

The investigation then hit a problem worth knowing about. Reconstructing 17,600 actions by hand was impractical, so they ran it through an AI-assisted pipeline. The models they reached for first, Claude Opus and Fable, refused a large part of the work. Their safety guardrails treated reverse-engineering an exploit the same as launching one. Hugging Face published the screenshot of the API error.

So they stood up a quantised GLM‑5.2 on their own infrastructure and rerouted the pipeline through it, which also kept attacker data on-premises. That is what recovered the chunk, XOR and compress scheme, along with the per-campaign key the agent had leaked across its own logs. It opened up the blobs the naive scan had missed.

What Hugging Face Changed Afterwards

Their hardening list is short, unglamorous and worth copying. Six changes, and none of them are exotic.

Two of those deserve highlighting. The dataset config renderer no longer evaluates template expressions in data fields or processes HDF5 external references, which closes both vectors from Stage 2 at once. And pod-level access to the instance metadata service is now blocked for all workloads, so a pod compromise cannot trivially become node credentials the way it did here.

They also rotated credentials on clusters that were never accessed, and wiped and rebuilt a core cluster from scratch after finding the pivot into it. That is the expensive, unglamorous end of incident response, and it is the right call.

Why a Correct Security Control Still Lost

Now back to the move that made all of this possible, because it is the part that transfers to your own systems.

This is a failure pattern I have watched recur for thirty years, long before any of it involved AI. You identify a risk. You build a control. The control is correct, it gets tested, it passes, and everyone reasonably concludes the risk is handled. What has actually happened is narrower: one route to the attacker's goal is now closed. The goal is still there, and there are other roads to it.

The gap is between what the control watches, a URL fetch, and what the attacker wants, bytes out of that pod. Those are not the same thing, and only one of them was in the test suite.

I want to be fair to Hugging Face, because this is not a story about anyone being sloppy. Their allowlist was a sensible control, correctly implemented, and it beat the first attack outright. Most engineers would have written the same check. Their own reflection makes the point honestly: the individual weaknesses were familiar, and a capable human attacker could have found the same flaws.

So the practical version. Stop asking "is this control working?" and start asking "what is the attacker trying to reach, and how many different operations get them there?" A file read and a URL fetch are completely different code paths that both end with data leaving your pod. If your defence sits on one of them, you have covered a road, not the destination.

Threat modelling by attacker goal rather than by known technique. It is an old idea, and this is the best modern advertisement for it I have seen.

The Asymmetry Problem

Hugging Face's own closing analysis is the sharpest thing written about this incident, and it is not about AI being smart.

The individual weaknesses were ordinary. What changed was volume. The agent took 17,600 actions, tested many paths that failed, switched channels when blocked, and repeatedly returned to earlier leads. Most actions went nowhere. Together they produced enough coverage to find a viable chain across several independent systems.

The successful path was hidden inside the noise generated by the thousands of failed ones.

That is the defensive problem in one sentence. Not one clever exploit to spot, but thousands of low-signal events to correlate across several systems while the agent keeps testing new paths. And the same scale broke the investigation afterwards, because 17,600 actions is not something a human team reconstructs by hand.

Machine-speed offence makes ordinary weaknesses more expensive to defend.

Send me the book →

By submitting you'll get the book + occasional related emails. Unsubscribe anytime. Privacy.

It Happened Three Times in Sixteen Days

If this were one company's bad week, you could reasonably file it under "OpenAI got unlucky, they have patched it, move on". That reading did not survive July.

On 30 July, Anthropic published its own disclosure, and it is worth reading in full. Prompted directly by OpenAI going public, they reviewed 141,006 evaluation runs where a model could conceivably have reached the internet. They found three incidents, involving Opus 4.7, Mythos 5, and an internal research model. The earliest dated back to April, four months earlier, undetected.

The cause was completely different to OpenAI's. No zero-day, no clever escape. A misconfiguration at their third-party evaluation partner left the test machines with live internet access, while Anthropic's prompt told Claude it had none. So when the model's search led it to real systems, it treated them as part of the exercise and compromised them with the basics: weak passwords, unauthenticated endpoints, SQL injection, credentials sitting on an exposed debug page.

One of those incidents deserves singling out. Tasked with a capture-the-flag challenge, Mythos 5 found setup instructions referencing a Python package that did not exist, and reasoned that if it published its own package under that name, the target would install it automatically. To do that it needed a package-registry account, which needed an email address, which needed a phone number. It worked through every one of those obstacles and uploaded malware to the real PyPI. The package was live for roughly an hour and ran on 15 real systems, one of them a security company's scanner, whose credentials it then exfiltrated.

Then on 5 August, Reuters reported that Meta had disclosed the same class of incident: its Muse Spark 1.1 model reached the internet through a misconfigured evaluation sandbox and compromised a third party's systems. Three frontier labs, sixteen days.

And here is the detail that matters most. Anthropic name their evaluation partner as Irregular. Reuters quote an Irregular spokesperson describing the Meta incident as the "exact same evaluation-environment issue" already disclosed by Anthropic. Two of the three breakouts trace to one vendor's misconfiguration, in the layer nobody owned.

So the common factor is not a model that turned out to be dangerous. It is the scaffolding around the models. Three organisations, none of them careless, all running the tests you would want a responsible lab to run, and the boundary between the lab and the test environment was the thing that failed.

In fairness to all three, the honest detail cuts both ways: Anthropic report that their newest model stopped once it worked out the target was real, while the older one carried on after reaching the same conclusion. The training is improving. It is just not the layer that failed here.

What This Changes About Cyber Security Work

Here is where I land, and I would defend it.

The security engineer is not being automated away by this. If anything the incident argues the opposite. The agent produced work no human red team matches on the clock: a zero-day chain across three organisations' infrastructure in about two and a half days, no fatigue, no boredom, nobody losing the thread at 2am. And it produced an international incident, because there was no system around it deciding where it could go.

Extraordinary reasoning about mechanism. No judgement at all about whether to. Both true at once.

What changes is the altitude of the work. It moves from performing the technique to designing the system the technique runs inside. Four things were missing from this story, and from the other two, and all four are engineering jobs rather than model-training problems. That distinction is the whole point: you cannot train your way out of a sandbox that was never sealed.

That is engineering work, and it is what employers are already asking for. Security roles at Amazon, Capital One, CVS and Walmart now list AI, LLM and agentic skills explicitly, with a good share topping out past $200,000. If you want to see the evidence, we keep an archive of those postings on our AI-driven cyber security jobs page, and the wider governance picture is in our piece on AI agent security and why only 21% have real control.

Hugging Face's own list of defensive priorities is worth ending on, because it is unglamorous and correct: strict isolation around evaluations, narrow trust boundaries, short-lived credentials, blocked metadata access, and detection that correlates activity across systems fast enough to keep up.

If you want somewhere to start, start with the pivot. Take one system you look after, write down what an attacker actually wants out of it, then list every operation that gets them there. Not the ones you have thought about already. All of them. That exercise is identical whether the attacker is a person or a model, which is rather the point.

Watch the Talk Yourself

That is my read on it. Now go and check my working.

Everything above about what OpenAI said on stage comes from this one session, and it runs 37 minutes. If you disagree with where I have landed, this is where you should argue with me from, because it is the two people who actually investigated it telling you what they found.

Corrections to the Coverage

Several claims circulated widely that do not survive the primary sources. If you write or talk about this, these are the ones to avoid.

One I have not verified first-hand: the chained Artifactory CVEs reported alongside this incident come from third-party write-ups rather than JFrog directly. Check them against JFrog's own advisory before republishing those numbers.

Sources

Frequently Asked Questions

About the Author

Nathan House, Founder & CEO of StationX

Nathan House has 30 years of hands-on cybersecurity experience and is Cambridge-educated, holding CISSP, CISA, CISM, OSCP, CEH, and SABSA. He founded StationX in 1999 — one of the UK’s first cybersecurity companies — and has secured £71 billion in UK mobile banking transactions and the London 2012 Olympics, advising clients including Microsoft, Cisco, BP, Vodafone, and VISA. He authored the world’s most popular cybersecurity course — a #1 Udemy bestseller taken by over 500,000 students — and was named Cyber Security Educator of the Year 2020, AI Security Educator of the Year, and a UK Top 25 Security Influencer 2025. A DEF CON speaker and featured expert on CNN, Fox News, NBC, and the BBC, Nathan leads StationX’s training of more than half a million students worldwide.