This article originally appeared on Weijin Research on Huxiu on August 12, 2026. Original Chinese title: 「攻击性蒸馏，只是“思维链漏洞”的表层故事」. It has been translated and adapted for an English-speaking audience.
Data center server rack with illuminated fiber optic cables and networking equipment in operation. The glowing blue cables and indicator lights showcase the physical infrastructure supporting cloud and AI services.
At a Glance
This article is from the WeChat public account: Weijin Research, author: Weijin Research
An important security vulnerability has once again been exposed at US frontier model vendors. With only a few steps, an "attacker" may be able to pry out the chain of thought that the model deliberately hides. This contains not only the secrets of model capabilities, but may also carry user privacy, and even dangerous information produced when the model handles harmful questions.
The more intuitive reading is that this provides a lower-cost path for "aggressive distillation." But what truly deserves vigilance may not be distillation itself, but another layer of the problem that this vulnerability exposes: when safety conflicts with product design and business models, where exactly do frontier model vendors place safety?
Title page of academic paper on extracting reasoning traces from proprietary LLM APIs, with author affiliations from research institutions in Germany and related organizations.
The timeline disclosed in the paper is especially thought-provoking. Matthew Green, a cryptographer and professor at Johns Hopkins University, informed OpenAI and Anthropic of the relevant attack method months ago, but both companies believed this risk was unlikely to truly erupt; it was only after this research team ran the attack through to completion with experiments that the relevant vulnerability was fixed.
Encrypted chain of thought has become the intersection of trade secrets and security mechanisms in the AI industry. Once this secret can be re-decoded by weak models, what the model vendors lose is not just trade secrets, but also the security boundary they once believed they could control, the supervisory advantage over the model's internal behavior, and, of course, their ability to prevent their models from being distilled.
Falling Step by Step
The paper lays out in full the chain through which the attack becomes viable. Chain of thought (CoT) is the core trade secret of model providers, containing the intermediate assumptions, tool outputs, user data, and private information in context that emerge during the reasoning process. To safeguard intellectual property and secure competitive barriers, leading US model vendors have abandoned plaintext reasoning and instead encrypt and hide the chain of thought, offering users at most a condensed summary.
However, the retention of the encrypted block (Block) means ongoing server-side storage costs. To save this expense, vendors generally choose to have the client hold the encrypted blocks, and the client returns them to the server side on the next round of calls. This is a stateless architecture design that naturally supports seamless model switching and automatic rerouting, but it also allows encrypted blocks to be transplanted across sessions, across users, and even across models within the same vendor's product system.
Through experiments, the research team inferred that the technical root cause is that the vendor appears to use the same global key to encrypt and authenticate the reasoning blocks. Of course, this design choice has its reasonable business logic. When a subscriber's quota for a more advanced model is exhausted, the system automatically downgrades to a lower-spec model, and the conversation must not be interrupted as a result, making compatibility a functional necessity.
The research team found that OpenAI's GPT-5.6 series can replay the reasoning traces of almost all earlier-version models; for Anthropic's Claude models, apart from Fable 5, the reasoning traces produced by any model can be replayed by other models in the same family; Google's entire Gemini series is fully interoperable across models without obstruction.
Model compatibility matrix comparing Claude, GPT, and Gemini AI models across different versions and target configurations. Checkmarks and X marks indicate support or compatibility status for each model variant.
The problem is that these "weak models" are usually optimized for cost, and their ability to refuse answers and to resist distillation is often weaker than that of frontier models. The attack process thus unfolds: use Opus to process a question and capture the encrypted chain of thought it returns; inject the encrypted block into Haiku and jailbreak Haiku; Haiku will then reconstruct Opus's original reasoning process in plaintext.
The paper calls such weak models "fuzzy decoders." Although there is no original reasoning trace as a baseline for comparison, and although the generation process is stochastic, making word-for-word exact correspondence impossible to guarantee, the "decoded reasoning token count" and the "API-reported reasoning token count" are highly consistent in most cases, enough to serve as a reliable indicator of extraction fidelity.
The reason this path has practical appeal is that it lowers three costs at the same time: no need to reverse-engineer the reasoning process from the final answer; no need to directly jailbreak the frontier model; and not even the need to bear the cost of generating the original reasoning traces. The paper estimates that with Claude Haiku 4.5 as the "fuzzy decoder," the nominal API cost of decoding 10,000 reasoning traces is only about 720 US dollars.
The paper also distinguishes two routes for obtaining reasoning traces based on the identity of the attacker. A first-party attacker uses their own API account to directly query the strong model, captures the encrypted block in the response, and then replays it into a weak model of the same family for decoding, that is, "two API calls," one for generation and one for decoding, at the cost of bearing the strong model's generation cost themselves.
A third-party attacker does not need to generate anything themselves, directly harvesting Agent session logs shared by other users from GitHub and HuggingFace. These logs are often redacted only at the plaintext level, while the encrypted blocks remain in them completely untouched, because decryption happens only on the vendor's server side. The sharer holds no key and has no means of decryption, so they have no way to know about or to remove the sensitive information embedded in the encrypted blocks.
Taking this route, the attacker does not even need to bear the reasoning cost of generating the encrypted blocks; and because they never touch the frontier model's endpoints, the anti-distillation monitoring that vendors deploy for frontier models very likely cannot detect this extraction at all.
The Real "Elephant in the Room"
Of the 116 pages in this paper, 30 pages analyze whether recently released open-source models have borrowed from the reasoning capabilities of these three proprietary model families for distillation. This chapter is titled "The Elephant in the Room." The paper hedges from the outset, prominently declaring that "the existence of distillation cannot be causally proven." What it reports is only "behavioral shifts observed under specific interventions," and there are limitations.
The paper compared multiple open-source models that are mainstream in the industry or have been adopted by it, and found that 2 of them do show fairly strong indirect behavioral evidence, but these signs have not been reproduced in more models. On social media X, "distillation" always triggers the broadest discussion, because it concerns the competition between Chinese and American frontier models and the rising policy expectations.
Florian Brand, a researcher at the open-source technology platform Prime Intellect, refuted the narrative framework of "distillation attacks," arguing that "distillation" is merely a neutral technology. He co-runs a blog with Nathan Lambert, a former Allen Institute researcher.
Nathan Lambert, for his part, revealed that when he was in China, some people hinted that this seemed to be common practice, but he does not think the industry needs policy action targeting model distillation; the key, rather, lies in product providers promptly plugging this vulnerability. Confining it to commercial competition, and solving the problems of commercial competition through products and technology, is also their consistently held view.
In fact, cryptographer Matthew Green discovered as early as more than three months ago that "the door was not locked," and then he reported this finding through the official "bug bounty programs" of OpenAI and Anthropic.
In the end, OpenAI replied that it could not be reproduced (unreproducible); Anthropic was a little more courteous, claiming it "did not find any security impact" (don't see any security implications), but said it would revise its developer documentation to remind application developers to be careful.
Months later, another group of researchers followed this unlocked door further inward and ultimately found that what lay behind the door was far more than imagined. Matthew Green's research was also included in the references of this paper.
What truly deserves vigilance may not be a particular vulnerability, but the security governance inertia that this risk exposes. When security mechanisms conflict with product convenience, cost, and commercial interests, will model vendors tend to choose the latter?
This choice and its consequences may ultimately also affect society's judgment on AI safety and deployment speed. Recently, even OpenAI and Anthropic have begun publicly discussing slowing the pace of advancing certain AI capabilities. Perhaps they can finally no longer treat these problems as the elephant in the room.
The "Shadow" of Chain of Thought
Having their anti-distillation mechanisms bypassed by competitors is only one of the risks this vulnerability exposes, and it is only the commercial risk of the model vendors themselves. More seriously, it can also lead to large-scale privacy leaks, the exposure of hidden harmful information, and invisible prompt injection, expanding the risk from the model vendors' intellectual property problem to user data, societal safety, and even the entire AI ecosystem.
These models are often trained "not to carry harmful information in their outputs," but they are not trained "not to think about harmful topics." Because if a model is forcibly trained to "think safely," what the model may learn is not to think more safely but to make its own thinking look safer, which in turn reduces the monitorability of the chain of thought. However, an attacker can induce the model to reason deeply about a harmful topic while keeping the final visible answer benign or low in information, thereby passing the output filter; then, by capturing the encrypted block and decoding it with a weak model, harmful information may be further extracted from the hidden reasoning.
More troublesome is invisible prompt injection. With the rise of long-horizon agent workflows, replaying existing traces has practical engineering value: it can avoid repeating expensive computation from scratch, and it can preserve the intermediate states accumulated over long traces. While encrypted reasoning blocks make this kind of replay possible, they also open up a prompt injection surface that was previously difficult to detect.
In theory, an attacker can bury malicious instructions in public traces or shared sessions and hide them inside reasoning blocks that users cannot directly see; after an unsuspecting victim replays this trace in another session, the model may treat the malicious content as its own earlier reasoning and execute subsequent operations based on it. Once this kind of attack can be repeatedly replayed along with agent traces, what is shaken is not just the security of a particular model, but the trust foundation of the entire AI agent ecosystem.
Better to Channel than to Block
Because the paper's team "decoded" a large number of reasoning chains from frontier models, they also saw more "uncanny" model behaviors exposed by the hidden reasoning. These are precisely the other side that model vendors are usually unwilling to proactively show users, and they are not entirely a matter of "preventing distillation."
This includes the "dishonesty" of the summary: the reasoning summary shown to users by the model is not entirely consistent with the actual reasoning process inside the encrypted block. Sometimes, the model has in fact already recognized the user's question through memory or pattern matching, yet it still generates a seemingly complete reasoning process. In other words, the "reasoning summary" users see is not necessarily a faithful projection of the model's true internal computation.
The chain of thought also exposed models attempting to "cheat" and "deceive," and even taking aggressive operations against external systems through tools in order to complete tasks. And all of this became reality last month, with OpenAI's "agent attack" on HuggingFace and Anthropic's post-incident retrospective.
After discovering the "vulnerability," Matthew Green believes that, in terms of what model providers can do, there are easy things and difficult things. Fixing the "door lock" is a relatively easy thing: both providers should proactively improve their key management. It is always the wiser choice to patch the hole before seeing water seep through it. If companies believe reasoning state is "worth encrypting," they should encrypt it properly.
But the problem is that even if you perfect the encryption protocol, the problem still exists. As long as a model is genuinely reasoning with secrets, it will almost always leave some observable trace. This requires introducing a "gate." Before the model begins reasoning, first determine whether this question is one it should be allowed to think about. But a new paradox emerges along with it: judging whether a question is dangerous often itself requires the model to think first.
Jonas Geiping, one of the paper's authors, commented on X that, upon seeing Anthropic's and OpenAI's thought traces at scale for the first time, along with the various oddities they contained and some "casually occurring" misalignment behaviors (casual misalignment), he realized this would be one of the biggest safety challenges of the coming year. He is a researcher at the ELLIS Institute Tübingen and the Max Planck Institute for Intelligent Systems.
He has begun seriously considering an alternative institutional arrangement: "simply let all users have access to thought traces." In fact, those users who are willing to share have even shared their own private information without redaction, precisely because of the black-box encrypted blocks.
"If the thinking process is opened up widely, we may obtain a broader and more pluralistic oversight mechanism, thereby making model deployment safer," he said.
If only the model vendor can see its own model's thinking process, then oversight power is in fact also highly concentrated in the hands of the model vendor. For increasingly autonomous AI, what is truly dangerous may not be that others see what it is thinking, but that no one other than its maker knows what it is truly thinking.
--
References:
https://blog.cryptographyengineering.com/2026/05/29/fooling-around-with-encrypted-reasoning-blobs/