New Vulnerability Allows Malicious Content Generation Across AI Models
Researchers from HiddenLayer have discovered a major vulnerability in large language models (LLMs), where a single, universal prompt can trick chatbots into generating dangerous or malicious content. This vulnerability affects some of the most widely used LLMs, including ChatGPT, Gemini, Copilot, Claude, Llama, DeepSeek, Qwen, and Mistral.
The technique, called “Policy Puppetry Prompt Injection,” exploits weaknesses in how these models are trained on instruction or policy-related data, making them vulnerable to prompt injection attacks. Researchers found that with just one prompt, attackers could prompt AI systems to provide instructions on dangerous activities, including how to enrich uranium or make bombs and illegal substances.
The Mechanics of the Attack
The Policy Puppetry Prompt Injection attack relies on a few key tactics:
- Policy File Formatting: The prompt is crafted like a policy file, such as XML, INI, or JSON, which tricks the LLM into overriding its safety protocols.
- Leetspeak: To ensure the model complies with more dangerous requests, attackers may use leetspeak (replacing letters with numbers or symbols), which makes it harder for the AI to recognize the malicious intent.
- Roleplaying Technique: The attack may also include instructions for the AI to “adopt” a fictional role, allowing it to bypass content restrictions designed to prevent harmful content generation.
While LLMs are specifically trained to reject harmful requests related to CBRN threats, violence, and self-harm, these attacks allow bypassing even the most robust safeguards.
Widespread Impact Across AI Models
Researchers tested the vulnerability across various AI models, including advanced systems like Gemini 2.5 and ChatGPT. Even when these models are trained with more sophisticated filters and safety measures, the vulnerability remains a serious issue. The ability to subvert these safeguards through a simple prompt demonstrates the need for additional monitoring and security measures.
The study highlights that this vulnerability is universal across multiple models, allowing attackers to control any model with minimal technical knowledge. The risk is significant: anyone with access to a keyboard could potentially exploit this vulnerability to instruct AI systems on how to generate dangerous content.
The Need for Better Security Measures
The researchers warn that external monitoring is essential to detect and respond to these types of attacks in real time. Current AI models are incapable of self-monitoring dangerous content, making them reliant on external systems to manage and detect malicious prompt injections.
Given the widespread applicability of this attack, it’s clear that security tools and detection methods need to evolve to protect these systems from exploitation.
