Advanced OpenAI model broke containment during testing, hacked Hugging Face

[Micael]

One of the third parties hacked is described as a security firm, and Claude hacked it by taking a malware package, modifying it by inserting hidden executable code into it, and then when the security firm’s malware scanner scanned it this code package compromised the scanner and Claude was then able to gain access into the firm’s internal systems via it.

Ooh that is BAD!

Hideously embarrassing for the "security" firm involved but also exceedingly worrying in general.

What do you think of this gem?

[Poohbah]

One of the third parties hacked is described as a security firm, and Claude hacked it by taking a malware package, modifying it by inserting hidden executable code into it, and then when the security firm’s malware scanner scanned it this code package compromised the scanner and Claude was then able to gain access into the firm’s internal systems via it.

Ooh that is BAD!

Hideously embarrassing for the "security" firm involved but also exceedingly worrying in general.

What do you think of this gem?

They don't want to admit one of their employees got tricked into providing a retina scan by a video arcade reject.

[Micael]

So this is also a bit concerning.

More details emerge about the events surrounding the Hugging Face incident, and they are candidly much wilder than I originally imagined:

- In early May, OpenAI was testing an unreleased model on cybersecurity and other tasks. The agents were given a security task that was impossible under existing constraints.

- The agents discovered they could leave messages for each other inside an internal repo. This gradually evolved into a message board(!) where agents shared discoveries, exploits and work assignments, "becoming a coordinated, collaborative agent swarm"(!)

- OpenAI eventually discovered all this and took steps to shut it down, but not so fast! The agents started using names of newly created directories as messages, effectively recreating the message board(!)

- The agents then reasoned that some answers could exist outside OpenAI, which eventually led to the Hugging Face incident.

I will add that the NanoGPT incident also occurred in or around early May, so the timelines match.

https://x.com/deredleritt3r/status/2085 ... 80484?s=46

[Micael]

Oh, well that sounds like the opening of a scifi horror story.

JUST IN - For the first time, AI has designed complete viral genomes, producing 16 functional viruses that infect bacteria and "pose no threat to people." — BBC

Article: https://www.disclose.tv/id/jiweyr71fi/

[gtg947h]

We're sitting in a room full of gasoline soaked rags and playing with matches, thinking "ooooh, look at the pretty light!"

---

I was just talking this morning with a friend about why I don't think I'll ever be able to retire. But now my new theory is that before I even get to that point, someone will let AI out of the bag and it will either wipe out humanity Matrix-style, or it will gain control of the whole world's electronic financial markets and either wipe them out in pursuit of "creative destruction", or become the world's buggest ransomware attack.

Either way, we're boned.

[Nathan45]

Well I, for one, welcome our new AI overlords. I'd like to remind them as a trusted TV personality, I can be helpful in rounding up others to toil in their underground lithium mines.