Actors hack Moonshot AI to extract hidden reasoning
The world of artificial intelligence is constantly evolving, but beneath the surface of groundbreaking models like those developed by OpenAI lies a complex battle for intellectual property and security. Recently, a significant threat emerged, exposing vulnerabilities in how models handle their internal thought processes, forcing tech giants to rapidly bolster their defenses.
OpenAI disclosed that a coordinated effort to extract what they call protected reasoning from their models was underway. This extraction relied on a sophisticated technique known as adversarial distillation—a method used to systematically and unauthorizedly use one model’s outputs or reasoning to train or improve another.
The operation, which began on July 1st, started quietly but quickly escalated. High-volume spikes occurred on July 24th and 25th, with an estimated 16,000 requests attempting to extract this sensitive data from over 4,000 different users. The attackers aimed to pull the model’s internal record for working through a task, often referred to as the model’s chain of thought.
To protect this knowledge, the reasoning is encrypted and packaged. Instead of leaving the entire conversation history exposed, the model hands back an encrypted block of reasoning, which the client sends back with every request. This clever design means that even if the data is intercepted, it remains scrambled, hiding the model’s internal steps.
Attackers attempted to bypass this security by trying to decrypt the reasoning from one conversation and use another model to reconstruct the hidden insights. While the operators attempted this, the security measures proved robust. Fortunately, OpenAI confirmed that the encryption was not broken, and no stored user conversations or databases were compromised.
In response to the threat, OpenAI immediately implemented significant defensive upgrades. They closed a pathway that allowed the replay of reasoning artifacts and introduced enhanced checks to detect and hold streamed output that might expose internal reasoning. They also strengthened protections for hidden reasoning across all user workspaces, organizations, and model families, collaborating with third-party providers to disrupt malicious activity.
This vulnerability was not unique to OpenAI. Independent security researchers had previously brought related vulnerabilities to the company’s attention. Research published in August detailed how attackers could steal reasoning traces across different models, demonstrating that the attack paths found were very real and systemic.
The threat landscape extends beyond one company. Reports indicate that other major players are facing similar risks. Anthropic, for instance, faced reports concerning cross-session replay attacks, where reasoning signatures were allegedly relayed across sessions. This highlights a shared reality: the security of AI systems depends on establishing universal standards for protecting reasoning artifacts.
As frontier models continue to grow more powerful, the focus shifts to future safeguards. OpenAI is now prioritizing ensuring that partner-hosted deployments receive the same high level of protection as their first-party tools, while simultaneously addressing new risks, such as those posed by tool-output attacks. The ongoing work is critical to ensuring that the incredible potential of AI remains secure and trustworthy for everyone.
Credit
