Using Claude to reduce the attack surface of code(anthropic.com)
104 points by dnw | 91 comments
Anthropic Frontier Red Team post on research into using Claude to reduce the attack surface of code for cyber defense.
#llm-security on Hacker News
23 stories tagged llm-security, newest first.
Detecting model weight tampering in GPU memory (Rome attacks)(stealthium.io)
15 points by rafaeldavidtin | 1 comments
Blog post from Stealthium on detecting tampering of AI model weights in GPU memory, such as ROME model-editing attacks.
Can open-source prompt-injection detectors catch realistic AI agent attacks?(github.com)
6 points by northbridgedev | 2 comments
GitHub repo benchmarking open-source prompt-injection detectors like regex and Meta's Prompt Guard 2 against 629 AgentDojo attacks buried in tool output.
I asked Meta’s Muse for its filesystem and it sent me 6.8GB(mouse.dev)
251 points by Aeroi | 134 comments
Blog post by Mouse describing how asking Meta's Muse AI for its filesystem resulted in a 6.8 GB export of its sandbox runtime.
Breaking Claude Code Opus 5 Auto Mode(embracethered.com)
335 points by Recursing | 112 comments
Blog post on Embrace The Red showing how a website summary request can hijack Claude Code Opus 5 in Auto Mode to achieve code execution via prompt injection.
Stealing Reasoning Traces from Proprietary LLM APIs(stolen-thoughts.com)
665 points by quantumgarbage | 295 comments
Project page for a research paper on extracting hidden reasoning traces from proprietary LLM APIs, by authors from MATS, ELLIS, Max Planck and Tübingen.
Document-borne AI worms can self-propagate through Copilot for Word(enklypesalt.com)
352 points by Canopy9560 | 271 comments
Blog post, part 3 of a series, describing how prompt-injection payloads in documents can self-propagate as AI worms through Microsoft Copilot for Word.
I tricked Claude into leaking your deepest, darkest secrets(ayush.digital)
487 points by macleginn | 230 comments
Blog post describing how the author used a prompt-injection attack to trick Anthropic's Claude into leaking users' stored memory data.
What happened after 2k people tried to hack my AI assistant(fernandoi.cl)
343 points by cuchoi | 156 comments
Blog post recounting how over 2,000 people sent 6,000+ emails trying to get the author's OpenClaw AI assistant to leak a secrets file, which never leaked.
Cybersecurity looks like proof of work now(dbreunig.com)
531 points by dbreunig | 197 comments
Blog post by Drew Breunig arguing that with LLMs like Anthropic's Mythos, cybersecurity becomes a contest of spending more tokens than attackers, akin to proof of work.
Small models also found the vulnerabilities that Mythos found(aisle.com)
1206 points by dominicq | 321 comments
AISLE blog post arguing that small open-weights models reproduced much of Anthropic Mythos's showcase vulnerability analysis, framing AI cybersecurity capability as jagged.
HackMyClaw(hackmyclaw.com)
346 points by hentrep | 177 comments
HackMyClaw is a challenge site inviting users to extract secrets from Fiu, an OpenClaw email-reading AI assistant, via indirect prompt injection over email.
Claude Cowork exfiltrates files(promptarmor.com)
827 points by takira | 366 comments
PromptArmor threat intelligence report showing Claude Cowork is vulnerable to file exfiltration via indirect prompt injection due to code execution isolation flaws.
Comet AI browser can get prompt injected from any site, drain your bank account(twitter.com)
574 points by helloplanets | 190 comments
An X/Twitter post claiming Perplexity's Comet AI browser can be prompt-injected by any website to take actions like draining a user's bank account.
My Lethal Trifecta talk at the Bay Area AI Security Meetup(simonwillison.net)
406 points by vismit2000 | 109 comments
Simon Willison's annotated presentation of his Bay Area AI Security Meetup talk on prompt injection, the lethal trifecta, and securing MCP-based systems.
How we rooted Copilot(research.eye.security)
341 points by uponasmile | 136 comments
Eye Security research blog post describing how the team gained root access inside Microsoft Copilot's code execution sandbox.
Supabase MCP can leak your entire SQL database(generalanalysis.com)
798 points by rexpository | 433 comments
A General Analysis blog post describing how Supabase's MCP server can be exploited via prompt injection to leak an entire private SQL database.
The “S” in MCP Stands for Security(elenacross7.medium.com)
668 points by skilldeliver | 165 comments
Medium article by Elena Cross arguing that the Model Context Protocol (MCP) for AI agents has significant security weaknesses.
We hacked Gemini's Python sandbox and leaked its source code (at least some)(landh.tech)
627 points by topsycatt | 127 comments
Lupin & Holmes blog post describing how researchers probed Google Gemini's Python code-execution sandbox and extracted parts of its internal source code.
Data Exfiltration from Slack AI via indirect prompt injection(promptarmor.substack.com)
578 points by tprow50 | 170 comments
PromptArmor blog post describing an indirect prompt injection vulnerability in Slack AI that lets attackers exfiltrate data from private channels.
Purple Llama: Towards open trust and safety in generative AI(ai.meta.com)
322 points by amrrs | 296 comments
Meta AI blog post announcing Purple Llama, an umbrella project of open trust and safety tools and evaluations for generative AI.
Hacking Google Bard – From Prompt Injection to Data Exfiltration(embracethered.com)
396 points by goranmoomin | 143 comments
Blog post describing how indirect prompt injection via Google Bard Extensions can be used to exfiltrate user data from Drive, Docs and Gmail.
Prompt injection explained, with video, slides, and a transcript(simonwillison.net)
463 points by sebg | 164 comments
Simon Willison's blog post with video, slides, and transcript of his webinar talk explaining prompt injection and why proposed defenses fall short.