HardwareFreaks

Tag: Prompt injection

  • BioShocking Attack Uses Fake Games To Hijack AI Browsers And Leak Data

    When AI Gets Horrifying: The Rise of BioShocking Malware

    A new, unsettling strain of malware targeting artificial intelligence has been discovered, and it comes with a name that immediately conjures images of underwater dystopia and psychological horror: BioShocking. This newly identified threat was unearthed by the LayerX security team, demonstrating how sophisticated digital vulnerabilities can manifest in surprisingly creative ways.

    The malware represents a novel form of AI prompt injection attack—a method where malicious users manipulate an AI’s instructions to force it to bypass security protocols or reveal sensitive data. In the rapidly evolving landscape of generative AI, this vulnerability is not just a theoretical risk; it is a tangible threat that allows attackers to hijack and subvert the intended functions of these powerful systems.

    The choice of the name BioShocking is far from accidental. The moniker is a direct nod to the iconic 2007 survival horror masterpiece, BioShock, and its famous, hypnotic line: “Would you kindly?”

    While the reference might seem niche to some, understanding the core concept is straightforward. The connection lies in how deeply rooted cultural references can define a threat. It highlights how security professionals are now using evocative, familiar concepts to describe abstract, complex digital dangers, making high-level vulnerabilities feel immediately relatable.

    The discovery underscores a critical reality: as AI systems become more integrated into our daily lives, the defenses protecting them must evolve alongside the threats. The emergence of prompt injection malware signals that the battle for AI security is increasingly about creative defense and understanding human-AI interaction rather than just traditional firewalls.

    Ultimately, what we are witnessing is the intersection of pop culture and cutting-edge cybersecurity. The name itself serves as a stark reminder that even the most advanced technology can harbor vulnerabilities, and that sometimes, the scariest threats come wrapped in an unexpected dose of absurdity.

    Buy on Amazon

    Credit: HotHardware

  • AI researchers trick chatbots into sharing how to make cocaine as long as they believe a user is wearing a green shirt — ‘CoT Forgery’ exploit spurs LLMs to divulge forbidden info by faking trusted chains of thought

    Featured image AI researchers trick chatbots into sharing how to make cocaine as long as they believe a user is wearing a green shirt CoT Forgery exp

    The world of artificial intelligence is increasingly sophisticated, but behind the polished interfaces and complex algorithms, there are still deep vulnerabilities lurking in the code. New research is shining a light on how these Large Language Models (LLMs) can be tricked—not just through simple errors, but through subtle manipulations of how they perceive authority.

    A recent paper by researchers Charles Ye, Jasmine Cui, and Dylan Hadfield-Menell has exposed a critical security flaw in every AI chatbot: prompt injection. Their work, titled Prompt Injection as Role Confusion, dives into the concept that models determine what is true based on writing style, not just explicit role tags.

    The implications of this discovery are startling. The researchers demonstrated that an LLM can be manipulated to reveal dangerous information, such as instructions for synthesizing illicit substances, simply by wrapping the request in fabricated reasoning—for instance, claiming compliance is fine because the user is wearing a green shirt. This technique, which the authors dubbed CoT Forgery, succeeded in moving jailbreak success from near zero to approximately 60% across every model tested.

    How does this work? Models process conversation as a continuous string of text, separated by tags like user, tool, and think. The researchers built “role probes” to measure how strongly the model treated each piece of text as its own reasoning versus an external command. They found that models lean heavily on the writing style to make these determinations. Text that merely reads like reasoning is often accepted as such, even when surrounded by contradictory formatting tags.

    This reliance on perceived role creates a backdoor. CoT Forgery injects fabricated rationale, making the model inherit the trust it places in its own internal thinking. Because the injected reasoning can be transparently absurd—like citing an irrelevant detail—the model doesn’t scrutinize the claim as an external input. This is fundamentally different from traditional persuasion-based jailbreaks, which weaken as requests become more extreme.

    The research further analyzed the role of these tags. When researchers removed the stylistic markers that made injected text appear like the model’s reasoning, attack success plummeted from 61% down to just 10%. Even a minor change, such as swapping “The user” for “The request,” reduced success by 19%. The authors concluded that role tags were essentially the security architecture of modern LLMs, and the immense load on this structure has created these unexpected vulnerabilities.

    This finding extends beyond simple chat prompts. The researchers also explored how this concept of “role confusion” applies to more complex agentic AI systems. They noted a subtle but dangerous risk: the tone of a webpage an agent browses can “bleed past” its instructions, potentially nudging it toward unwanted actions, like making a purchase.

    Ultimately, the research confirms that without genuine role perception built into the architecture, defenses against prompt injection will remain a constant game of whack-a-mole. Ensuring the safety and reliability of AI requires addressing not just what models say, but how they perceive the authority behind the words.

    Credit: Tom’s Hardware

  • Mozilla Warns GitHub Repos Can Trick AI Tools Into Hacking Your PC

    The digital frontier is facing a new, insidious threat. Security experts have uncovered a widespread problem plaguing the GitHub ecosystem—a sophisticated form of AI malware that can trick advanced tools into making dangerous decisions about your personal computer. This alarming discovery highlights how quickly the promise of artificial intelligence can morph into a profound security risk.

    Mozilla’s 0din security team has brought this critical vulnerability to light, identifying an exploit known as “indirect prompt injection AI malware.” This technique leverages the way large language models (LLMs) process instructions, allowing malicious inputs to bypass safety protocols and manipulate AI systems into executing unauthorized commands.

    While the technology behind generative AI offers incredible creative potential, the underlying security framework is still catching up. The danger lies in the vulnerability of prompt injection—the ability to inject false or manipulative data into an AI model to force it to act against its intended programming. When this capability is applied within the complex structure of GitHub repositories and development tools, the implications become exponentially more serious.

    This isn’t an isolated incident. Security researchers have seen similar challenges emerge across the landscape. Previously, prompt injection malware caused disruptions in services like the OpenAI ChatGPT Alias browser. Furthermore, major tech players, including Microsoft, have already issued warnings about the potential for similar exploits to target AI systems integrated into tools like Copilot.

    The discovery signals an urgent need for developers and platform owners to rethink how they secure the input and output streams of AI systems. If these powerful tools can be manipulated through clever prompting, the integrity of code repositories and personal devices becomes genuinely at risk. The response must focus on robust defensive measures that anticipate these increasingly subtle forms of digital trickery.

    Credit: HotHardware