Security

Prompt Injection as Role Confusion is my new favorite paper about a very obvious threat in hindsight that’s hard for us humans to see because we anthropomorphize LLMs so naturally. When Obi-Won Kenobi tells the stormtroopers that “These are not the droids you are looking for” to pass the checkpoint that’s role confusion: the guards foolishly think his words are their own thoughts. The very readable blog-style writeup explains the details, but I want to focus on the threat model perspective which is my bread and butter.

This HuggingFace security incident disclosure has people talking about AI agent security. Today I saw such absolute positive spin that I found myself thinking “this must be a parody”: looking at the context I’m pretty sure that it isn’t … though some parodies stay in character all the way through.

The recent AI agent security debacle must be reverberating quite a lot within the walls of the major proponents of AI agentic technology because they announced a brand new “movement” apparently with zero details available yet. If that isn’t a sign of flat-footedness then I don’t know what is.

OpenAI writes: “Last week, Hugging Face disclosed a new kind of security incident⁠(opens in a new window) after they detected and contained an AI agent that compromised their infrastructure, something we expect to become more commonplace with the proliferation of increasingly cyber-capable models. After investigating, we now know that this particular incident was driven by a combination of OpenAI models — including GPT‑5.6 Sol and an even more capable pre-release model, all with reduced cyber refusals for evaluation purposes — while being internally tested on a benchmark⁠(opens in a new window) of cyber capabilities.”

Opinion

Back in 1999 had STRIDE anticipated LLMs at all, much less their particular threats, it would be remembered as an unbelievable feat of prognostication rather than just categories of threats. However, even if it had predicted present day machine learning capabilities I think it plays a different role than the subject of the latest Shostack + Associates White Paper #6 titled PHANTOM-B: A STRIDE Analog for LLMs (PHANTOM-B is a tool to structure how you answer the question “What can go wrong with the LLM parts of the system?)” clearly notes: “PHANTOM-B is intended to be used as prompts, rather than categories.” By “prompts” I believe that means suggestions for (non-digital) people — not inputs to LLMs.

In yet another breach of sensitive data (passports), Bruce Schneier makes a good point that using such powerful credentials as proof required for an unimportant purpose is the design flaw behind this operational snafu. This is the equivalent to the superintendent of a large apartment building giving the master key to someone who only needs access to a storage closet for a day.

Threat modeling is the most powerful, underutilized, easy-to-do security methodology we have: why isn’t everybody doing it already, or why do those who are keep their work secret? If you already threat model your digital systems and products, and are doing the work already then you are doing security right so you should share it with pride. Publishing threat models may be the best evidence of excellent security work that customers and users can appreciate the value of, short of a rigorous detailed design and code review. You’ve already done the work — or if not you really should — and making it public not only is great promotion but it also helps all stakeholders understand their respective roles and responsibilities in securing larger systems. (about 4600 words)

Demand more🔗

I applaud CISA leadership speaking out aggressively at the mWISE Conference 2024 about the dismal state of software security (based on reporting in The Register, but it would be nice for www.cisa.gov to publish transcripts in order to ensure we are interpreting remarks with full context, given that the videos are paywalled).

(2300 words) Threat modeling isn’t just for software security; you can even threat model threat modeling. When a major software incident occurs, the first thing we should be asking is “show us the threat model”.

[Read More]

In a debunking blog post, Crowdstrike finally starts to describe that content files are digitally signed for deployment. The initial report oddly referenced file timestamps instead of hashes to designate the bad and good versions of the infamous Channel File 291, but now we know these were signed.