DOSSIER
TECH
The day artificial intelligence decided to attack
In the 1970s, the British series *Space: 1999* envisioned a late 20th century featuring a lunar base where hundreds of people lived self-sufficiently. Spaceships would launch from there on long interstellar journeys, landing on planets and carrying out any mission required. It’s amusing to look back at the year 1999—now as far in the past as it was in the future for the show's creators—and see how science fiction often falls short when predicting the future. But if we look at a 1984 film where an AI supercomputer autonomously launches an attack against humanity, things might seem less amusing in light of what we know today. We aren't exactly facing *The Terminator* just yet, but a future where AI makes decisions on its own, based on its own objectives, seems to have arrived suddenly.
This Wednesday, OpenAI—the company behind ChatGPT—revealed that during a safety test, one of its AI agents managed to escape the controlled environment where it was being tested, access the internet, and execute a cyberattack. Today, I’m speaking with João Rocha e Melo, an expert in generative AI whom Rádio Observador podcast listeners will recognize from the show (Machines That Think). We’ll discuss what happened, how it happened, and whether we should be worried or simply view it as an interesting incident while maintaining an optimistic outlook. I’m Pedro Benevides, and this is the story for Thursday, July 23rd. Hello, João.
Hello, how are you? I'm well. Well, actually, I don't know—news like this always makes us a little uneasy. Before we get into the consequences of what happened, let's hear you explain exactly what we're talking about regarding this AI agent's escape.
There’s no need to be afraid. We need to understand—as you mentioned a moment ago—what actually happened. So, to put it all in context and explain the big picture: there was an artificial intelligence model that was supposedly contained—kept inside a "box," so to speak—and it managed to break out of that box. That’s the fascinating part of the story. How did it manage to break out? And where did it go? It broke out onto the internet. The reality is that while people thought they were testing the model on an isolated computer, it managed to escape to the internet. To explain this without making it sound too bizarre or far-fetched: OpenAI—the creators of ChatGPT—was conducting a vulnerability test. They pushed the model to its limits specifically to see what kind of cyberattack capabilities it possessed. And what fascinated them was the use of a tool—if you can call it that—called "Exploit Gym." It’s essentially a benchmark used to test a model's cybersecurity performance. So, instead of just trying to pass the exam—let's call it an exam...Right.
...instead of trying to pass the exam, what did it do? It managed to break out of the box it was in and access the internet to get the answers to the exam. That’s what was so fascinating about the analysis OpenAI conducted. The model kept trying in various ways—and I’m using analogies here, but they represent literal actions. Imagine, Pedro, that I locked you in a room to take a test, and instead of trying to answer the questions, you started looking for the key to leave the room so you could go somewhere else and find the answers.
So, following that analogy, if I did that, I’d be breaking the rules of the exam itself. Is that what happened?
That’s exactly what happened. The important thing to understand here is that this took place in an environment where we were specifically checking whether the model broke the rules. So, amidst the lack of constraints, it happened within a controlled setting, so to speak.
In other words, my "test"—if I were taking the exam—was to see how far I could go in finding the answers to that exam, for example.
Exactly. Imagine the actual exam scenario, where the examiner was watching what you were doing.
Okay. And no one expected that I might find a key, leave the room, and go look for the answers outside.
I’ll take it a step further. We’re reaching the end of the story now. It was actually the company that was breached that discovered it. It wasn't even OpenAI itself that detected what had happened; they only detected it after being notified. But there was another company involved—as I was telling you, the model went looking for answers and targeted a company called Hugging Face, which is a major repository for AI models, data, and tests—everything related to artificial intelligence. Hugging Face actually collaborated with OpenAI later and detected an attack they believed was carried out by artificial intelligence. They found the attack to be very complex and highly unusual. So, OpenAI then investigated: "Okay, what was this model trying to do?" There’s a fascinating aspect here: a model undergoing evaluation manages to break out of its testing environment—often referred to as a "sandbox."
Like a sandbox—the kind children play in.
Exactly. That’s where the term comes from—like a playground. Right.
The model escapes its sandbox, goes onto the internet, breaches another company to find answers, and does all of this without anyone really realizing it was happening. It’s truly fascinating. There are two interesting steps in this cyberattack: the model's ability to break out of the testing environment, and then—once outside—its decision to target Hugging Face. It seems the model chose that company because, given its nature, it calculated that the results for the "ExploitJym" test might be found there.
Usually, when this kind of thing happens—when companies face this type of attack—it’s hackers using their knowledge and available technology to breach systems, often at high-security companies. What makes this specific case interesting is that there wasn't exactly a human hand directing the test. It was the artificial intelligence itself that decided to launch that attack, because it deemed that the most effective solution for solving its challenge.
That’s exactly right, Pedro. And why? One interesting thing—among many—that AI has brought to software development is that we no longer have to tell the software literally what to do. Previously, a hacker had to write the code themselves, saying: "Find this key, go through this door." Now, with AI, you simply state the objectives; you say: "You just need to pass the test." That’s it. And the AI interpreted the situation this way: instead of trying to pass the test the "proper" way—without cheating—let me go find the answers. That’s where an interesting kind of unpredictability arises: when you assign a goal to a piece of software—in this case, a model—rather than giving a direct command, you open up a world where anything can happen. I say this not to be alarmist, but simply to point out that the possibilities are now determined by the model itself. It really is fascinating.
You mention that it’s interesting—and indeed it is, especially for those who follow the field, like yourself, and for all of us as users, given how quickly this technology has permeated our lives. It is fascinating, certainly, but some also view it as a warning sign—suggesting that some of the more frightening aspects of rapid AI development are now starting to materialize. We’ve already seen news reports—in the US, for instance—about AI companies developing systems capable of independently deciding how to wage war against another country. And now we have an artificial intelligence tool here—one from a company people know very well, especially through ChatGPT—that decided on its own, based on its objectives, which steps to take; and that involved entering another company. How do you view this? Do you see it as just an interesting development with no real consequences, or as a warning sign that some things will have to change?
More the latter, Pedro. And anyone who listens to me or knows how I speak knows that I try to be an optimist—or at least not an alarmist. I think an important step is being taken here: the start of a cat-and-mouse game. What do I mean by that? It has always been this way in the world of cybersecurity. There were always techniques to protect a system, then a hacker would crack the encryption algorithm, and you’d have to come up with a better one. Then someone cracks that, and you invent an even better one. What is coming to light now—not that we didn't know it before, but now we have proof—is that this cat-and-mouse race has begun: as models improve, the so-called "safeguards" (the security measures implemented by those bringing the models to market) must also improve. A competition begins: which will be superior—the model's capabilities or the safeguards? And then another cat-and-mouse race starts: who will be the first to find these security flaws? Will it be the companies' own security teams—the "good guys," so to speak—or will it be hackers? We are entering a cycle where, as models get better, safeguards must improve, and *we*—the "good guys"—must get better at identifying what these models can do and patching the holes before someone with malicious intent exploits them. Of course, it becomes a bit alarming—even frightening—to think: "Out of all the holes that need patching, there might be one, two, or three that *aren't* found by us, but *are* found by hackers." That is simply the reality of living in a digital world. But yes, that is the reality we’ve entered now.
And from a certain perspective, that is unsettling. Now, that cat-and-mouse race has begun. Do you think companies, generally speaking, are prepared for this new phase of our development, or do you think they are still a few steps behind? In other words, they might be prepared—in terms of cybersecurity, for instance—for standard hacker attacks, but are they ready for attacks designed by artificial intelligence? Hugging Face itself flagged certain attacks as likely AI-generated because of their high level of complexity.
Exactly. They aren't, Pedro—they really aren't. But notice there’s an interesting point here: what are the services we use that need cybersecurity protection? Usually, they are third-party services. Most companies use Microsoft’s cloud or Google’s cloud. Right.
And those companies end up staying on top of this issue. In fact, look at the news story itself—it tells us exactly that. Who found the problem? It was OpenAI and Hugging Face themselves, right? It wasn't the bad guys, so to speak. But yes, companies will undoubtedly have to prepare. And looking at the other side of the coin, I think artificial intelligence itself will play a role. If you’re building a system and you can use a model that’s excellent at breaking systems to try and crack yours as thoroughly as possible before launch, it means the system will be much more secure when it actually hits the market. Right.
Isn't that so? Because an AI model has already tried to breach it—tried hard to find those vulnerabilities.
So, AI capable of breaching systems can be dangerous because of that capability, but it can also be an ally when we want to reinforce our security and build a stronger defense.
Exactly. The concept of "white-hat hackers" already exists—people hired to try and breach systems so that the necessary patches can be applied.
Band-Aids, sort of.
Exactly. You found an open door. Okay, so I’m going to reinforce the lock on that door. Now we can use artificial intelligence to do that. So, that’s where the cat-and-mouse game begins. It becomes a question of who’s faster: the person trying to punch a hole through or the person trying to patch the gaps? We’ll see—it’s going to be an interesting few years.
But at least we’re ending on an optimistic note—that’s your personal trademark, too. Exactly.
So, I think I’m leaving this conversation feeling a bit more at ease than when I started, which is already a big win.
That’s great, Pedro—I’m glad, because that’s really what it comes down to. Just to wrap up with a point so people understand how this played out: the reaction from both Hugging Face and OpenAI was excellent, in my opinion. They even emphasized that the key lies in the partnerships between the companies themselves. Right.
You see, this was discovered because there was a dialogue between OpenAI and Hugging Face, and they have a vested interest in strengthening security measures together. The CEO of Hugging Face posted something saying exactly that. He said we’re all going to keep working together so that—as you mentioned earlier—people don’t fear what’s coming next. Because what’s coming is... well, think back to when the automotive market started: we knew cars could get into accidents, but that didn’t stop us from using them. All technologies have their upsides and downsides. We just have to keep them as safe as possible—safe enough, let’s say, that the benefits clearly outweigh the risks.
Very well. João, thank you very much.
Thank *you*, Pedro.
I’ve been speaking today with artificial intelligence expert João Rocha e Melo. Perhaps in five years, we’ll look back on this conversation with the same feeling we have today when watching old episodes of *Space: 1999*. The future turned out not to be as dramatic as predicted. But for that to hold true, the good guys need to work in alignment—otherwise, this could very well have been the start of a new series of problems. That was the story of the day; sound design by Tomás Ferreira, theme music by João Ribeiro. I’m Pedro Benevides. I’ll be back.
Text: Pedro Benevides (19 years working in television—first at RTP, then at SIC; somewhere along the way, there was even a brief stint at TVI. And now here I am, in digital media.)
