PHANTOM-B: threat modeling systems using LLMs

#security #threatmodeling

Opinion

Back in 1999 had STRIDE anticipated LLMs at all, much less their particular threats, it would be remembered as an unbelievable feat of prognostication rather than just categories of threats. However, even if it had predicted present day machine learning capabilities I think it plays a different role than the subject of the latest Shostack + Associates White Paper #6 titled PHANTOM-B: A STRIDE Analog for LLMs (PHANTOM-B is a tool to structure how you answer the question “What can go wrong with the LLM parts of the system?)” clearly notes: “PHANTOM-B is intended to be used as prompts, rather than categories.” By “prompts” I believe that means suggestions for (non-digital) people — not inputs to LLMs.

This is a great topic for discussion and a wide open greenfield, so for the sake of discussion what follows is first reactions to the paper. Beyond right and wrong, I hope it’s instructive even if totally wrong.

If I follow, the new acronym that serves as the centerpiece of the paper enumerates not what rather than how things go wrong using LLMs. Here are the LLM pitfalls covered:

Some comments that again I stress are personal opinions where I’d like to understand better:

More threats #

An additional threat unmentioned concerns the disposition of data sent to the LLM which could be a component running on the same box as the app in question, or a remote service: what could possibly happen with that data? Not only could there be an external leak, but also could there be influence between independent prompt invocations? Is prompt data recorded or not? Will LLM usage be used for future training purposes? And if the LLM is provided by a third party there are supply chain risks to consider like any other software.

LLMs are just code #

Perhaps this is obvious: LLMs are code like any other software so prone to all the same issues. Specifically, inference engines are table-driven algorithms with the unusual tweak of being stochastic: they introduce entropy into their input which is not something you often would see back in the day.

I mention these because whenever special treatment is afforded to LLMs it’s easy to focus on these issues and forget all the rest that we always need to consider.

Two LLM threat scenarios #

Wisely, to maintain sharp focus, the paper does not “include any defenses, controls or mitigations” which would make it much longer. However I see two fundamentally different kinds of LLM applications worth separating out for this discussion (of course it isn’t for me to decide the scope of this work, so consider this my own extension). At times the paper mentions this distinction which I consider applicable across the board.

The difference between these amounts to a distinction in data flow and tainting: is the data simply passed through or can LLM output control or influence running code — thereby opening the door to usurping control and possibly escalating (such as with prompt injection). All other cases potentially let the LLM influence code execution so the risk is substantially higher.

Avoiding specific mitigations, I think that one mitigation universally applies to all LLM uses because fundamentally they cannot be fully trusted. Model parameters are far beyond conceivable means of analysis, plus with stochastic variation added in testing because effectively an infinite number of cases. While it is true that no complex software can be “fully trusted” with LLM inference it’s unclear that any real metric for degree of trust is possible. With classic software we have methods such as test coverage that provide a reasonable proxy for the extent of testing.

Inherently Risky? #

The paper concludes: “In contrast to more traditional software, LLM issues seem to be inherent, and the best available defenses are probabilistic, rather than reliable. The dangers that result from LLM being token-prediction machines seem inseparable from the LLMs”. If this is true and the extent of risk cannot reliably be restricted to some ceiling on blast radius how can these be safely used for serious purposes?

At least in theory I would say there is a decent answer, not one many LLM advocates will like. As with any software component that cannot be fully trusted, LLMs should be sandboxed if security is a serious concern. Depending on the application this can be relatively straightforward, since LLMs are regular code tried and true secure coding techniques work great providing that the security requirements are well defined.

Every application is different, but here’s a sketch for a generic application that invokes an LLM. Suppose, for example, a JSON with a prescribed schema prefixed by a standard prompt is passed to the LLM, and the response should be a JSON (different schema). Sandboxing by checking both JSONs in and out with standard input validation techniques goes a long way to preventing trouble and if it happens detecting and rejecting it before bad things happen.

With sandboxing the data transformer use cases are relatively tame. If it’s a poem composition app then worst case it writes lousy poems, no STRIDE worries at all. Once LLM output can influence code execution then the attendant risk derives from a combination of how much influence — from minimal up to arbitrary code execution — and the scale of the blast radius of its potential privileges.

Viewed as sandboxed code, LLM security takes on the look and feel of any other code.