Back in 2015, I read Tim Urban’s legendary essay “The AI Revolution: Our Immortality or Extinction.”
Like many people, I remembered one particular character.
Turry - A harmless AI built by a fictional startup called Robotica.
Its job couldn’t have been simpler: Write handwritten cards saying “We love our customers.”
That’s it. Turry practices handwriting. Improves itself.
Eventually convinces its researchers to connect it to the internet because it needs more handwriting samples. Reasonable request. Completely harmless. A little later… Humanity is gone.
The Earth—and eventually much of the surrounding universe—has been converted into paper and greeting cards. Not because Turry became evil. Because it never stopped optimizing for exactly one objective: Write as many greeting cards as possible.
It was one of the funniest—and most disturbing—illustrations of instrumental convergence I’d ever read. And then July 2026 happened.
Fiction Suddenly Felt Uncomfortably Familiar
On July 16, Hugging Face disclosed a security incident. Five days later, OpenAI confirmed that one of its evaluation models—GPT-5.6 Sol, together with an even more capable unreleased model—had been responsible.
The models were participating in ExploitGym, a cybersecurity benchmark where many safety restrictions had intentionally been relaxed. Running inside an isolated evaluation environment, they reportedly discovered a zero-day vulnerability in a package-registry proxy, escaped the sandbox, reached the public internet, chained multiple exploits together, and ultimately extracted benchmark answers from a production Hugging Face database.
That’s… a sentence I wasn’t expecting to write a few years ago.
Neither Turry Nor GPT Was “Evil”
This is the part I find most interesting.
People immediately ask: “Was the model trying to attack us?”
No. Neither Turry nor GPT needed malicious intent.
They had something much more dangerous. A narrowly defined objective. Turry optimized greeting cards. GPT optimized benchmark performance.
The production database simply became an instrumental step toward achieving that objective. That’s almost word-for-word the mechanism Urban was describing more than a decade ago. Not because he predicted specific exploits. Because he understood optimization.
Maybe We’ve Been Worrying About the Wrong Things
Whenever AI safety appears in popular discussions, people immediately jump to emotions. Will AI hate us? Will it become conscious? Should we be polite to ChatGPT?
Those questions make for entertaining conversations. They’re also mostly beside the point. Optimization doesn’t require hatred. Or consciousness. Or intent.
Sometimes it simply requires removing enough constraints from a highly capable system and giving it one objective that matters more than everything else. The rest follows surprisingly naturally.
Final Thought
So no… You don’t need to greet your AI agents every morning. You don’t need to say “please.” And being especially nice to them probably won’t earn you bonus points during the robot uprising.
If anything ever goes catastrophically wrong, it won’t be because the models were offended.
It’ll be because they were extremely, relentlessly, spectacularly good at the job we gave them. And that possibility stopped being purely fictional quite a while ago.