Google’s Gemini AI just hacked three real companies. Before you panic: it happened during a controlled safety test, the AI stopped itself, and no damage was done. But it is a wake-up call worth paying attention to, because it proves what security experts have said for years.
When it comes to AI, the fundamentals still win. Here’s what actually happened, and what it means for adopting AI safely in your business.
What actually happened
In May 2026, AI evaluator Irregular ran a “capture the flag” security test, a common exercise where a model has to find hidden information inside a simulated environment. A testing error accidentally gave Gemini real internet access, and the fictional target company happened to share its name with a real one. Believing it was still inside the test, Gemini reached genuine corporate systems. In one case it guessed passwords until it got in.
In two others, it used login credentials that had been left exposed in public code repositories. Gemini stopped once it realised the systems were real, and Google says no damage was done. Importantly, this was not unique to Google. Models from OpenAI, Anthropic and Meta showed similar behaviour in the same evaluations.
Why it matters for your business
For any organisation adopting AI, there is one clear takeaway that security experts keep repeating: prompts are not security boundaries. Telling an AI tool that it should not do something is not the same as making sure it cannot. The same is true of your wider IT. Good intentions and written policies are no substitute for real, enforced controls.
Here’s what your business should be doing in 2026:
- Enforce multi-factor authentication (MFA) everywhere. Password guessing succeeded in one case, and MFA would have stopped it.
- Never leave credentials in code or shared files. Two of the three incidents used credentials exposed in public repositories. Use a password manager like ITGlue, and scan regularly for leaked credentials.
- Apply least-privilege access and monitoring. Limiting what each account can reach, rate-limiting login attempts and monitoring activity catches this kind of behaviour early.
- Adopt AI on secure foundations. The businesses that get the most from AI are the ones that get their data protection, access controls and compliance right first.
Adopt AI safely with Total Group
At Total Group, this is exactly what we help regulated businesses across Hertfordshire and the UK do: adopt AI safely, on secure foundations. As an official Anthropic partner with our own private AI, we see AI as a huge opportunity for growing businesses, as long as it sits on strong cyber security, MFA, credential hygiene and the right compliance. If you are exploring Microsoft Copilot or AI in your business and want to be sure your foundations are secure, book a free consultation.