Opinion
Back in 1999 had STRIDE anticipated LLMs at all, much less their particular threats, it would be remembered as an unbelievable feat of prognostication rather than just categories of threats. However, even if it had predicted present day machine learning capabilities I think it plays a different role than the subject of the latest Shostack + Associates White Paper #6 titled PHANTOM-B: A STRIDE Analog for LLMs (PHANTOM-B is a tool to structure how you answer the question “What can go wrong with the LLM parts of the system?)” clearly notes: “PHANTOM-B is intended to be used as prompts, rather than categories.” By “prompts” I believe that means suggestions for (non-digital) people — not inputs to LLMs.
This is a great topic for discussion and a wide open greenfield, so for the sake of discussion what follows is first reactions to the paper. Beyond right and wrong, I hope it’s instructive even if totally wrong.
If I follow, the new acronym that serves as the centerpiece of the paper enumerates not what rather than how things go wrong using LLMs. Here are the LLM pitfalls covered:
- Prompt injection
- Hallucination
- Anthropomorphization
- Non-explainability
- Training issues (including data quality or “poison”)
- Over-reliance on the LLM
- Missing security engineering
- Biases
Some comments that again I stress are personal opinions where I’d like to understand better:
- Given the probabilistic nature of LLM algorithms I think of prompt injection as a continuum rather than a binary (this is prompt injection, that is not). All prompt tokens including choice of words that humans consider equivalent must tweak the output to some degree — and depending on the topic and sensitivity minor influence might bleed into more significant effects on outcomes.
- For large models it seems inconceivable that so-called guardrails can possibly be 100% — and there is credible claim of a proof of that. If prompt injection is unavoidable (undetectably and to an unknown extent) how can you possibly model the residual threat? (see Mitigation, below)
- Hallucination seems to be a catch-all for any incorrect response (that’s not my usage but no point bickering over definitions). However, I will opine that the stated definition is infeasible for LLMs (“LLM hallucinations are outputs that are at odds with facts, math, or common sense.”) My point is that an LLM’s Umwelt is determined by the training data it ingests: it’s clearly impossible for it to independently fact check what it learns by testing inputs in the real world; that is, a “perfect” LLM must still 100% believe its training.
- I must admit that I could not follow the Anthropomorphization discussion for two reasons. First, our tendency to infer human-like inner state is deeply embedded from ancient times (primitive gods as humans with super powers) or even projecting human thinking and intention onto almost any living organism (“stubborn weeds”) or inanimate objects (“my car didn’t want to start for me this morning”). Also, very related, is the fact that our language doesn’t have pronouns that fit LLMs so repurposing the existing ones is not only reasonable but without inventing words from whole cloth what is the alternative?
- Non-explainable or inexplicable answers seem an old problem with classic (not LLM) software. The infamous COMPAS system used for correctional rehabilitation and judicial sentencing is one case in point. A numerical algorithm cannot be an explanation for such a subjective decision because it allows no exceptions, not to mention data bias (another PHANTOM-B factor). However, every experienced programmer has run into issues where one must shake their head and admit, “I have no idea why that is happening”, so I don’t follow the LLM-specific point here.
- I would add a third training data threat: insufficient data. Due to common sycophancy (are there any exceptions?) LLMs will conjure up plausible responses outside any available data.
- Over-reliance on a model sounds like “no human in the loop” but that’s infeasible for almost any application so I don’t know what to make of this threat (see below for ideas).
- Missing security engineering is hardly particular to LLMs. I’d like to see an example of how “an LLM magnifies”.
- Biases in data also seem endemic in classic software that processes any kind of data produced by humans who are vulnerable to countless biases and thinking errors.
More threats
An additional threat unmentioned concerns the disposition of data sent to the LLM which could be a component running on the same box as the app in question, or a remote service: what could possibly happen with that data? Not only could there be an external leak, but also could there be influence between independent prompt invocations? Is prompt data recorded or not? Will LLM usage be used for future training purposes? And if the LLM is provided by a third party there are supply chain risks to consider like any other software.
LLMs are just code
Perhaps this is obvious: LLMs are code like any other software so prone to all the same issues. Specifically, inference engines are table-driven algorithms with the unusual tweak of being stochastic: they introduce entropy into their input which is not something you often would see back in the day.
I mention these because whenever special treatment is afforded to LLMs it’s easy to focus on these issues and forget all the rest that we always need to consider.
Two LLM threat scenarios
Wisely, to maintain sharp focus, the paper does not “include any defenses, controls or mitigations” which would make it much longer. However I see two fundamentally different kinds of LLM applications worth separating out for this discussion (of course it isn’t for me to decide the scope of this work, so consider this my own extension). At times the paper mentions this distinction which I consider applicable across the board.
- LLM as data transformer that morphs inputs to output (unable to influence any action)
- LLM as agent with privileges that does or can possibly do things
The difference between these amounts to a distinction in data flow and tainting: is the data simply passed through or can LLM output control or influence running code — thereby opening the door to usurping control and possibly escalating (such as with prompt injection). All other cases potentially let the LLM influence code execution so the risk is substantially higher.
Avoiding specific mitigations, I think that one mitigation universally applies to all LLM uses because fundamentally they cannot be fully trusted. Model parameters are far beyond conceivable means of analysis, plus with stochastic variation added in testing because effectively an infinite number of cases. While it is true that no complex software can be “fully trusted” with LLM inference it’s unclear that any real metric for degree of trust is possible. With classic software we have methods such as test coverage that provide a reasonable proxy for the extent of testing.
Inherently Risky?
The paper concludes: “In contrast to more traditional software, LLM issues seem to be inherent, and the best available defenses are probabilistic, rather than reliable. The dangers that result from LLM being token-prediction machines seem inseparable from the LLMs”. If this is true and the extent of risk cannot reliably be restricted to some ceiling on blast radius how can these be safely used for serious purposes?
At least in theory I would say there is a decent answer, not one many LLM advocates will like. As with any software component that cannot be fully trusted, LLMs should be sandboxed if security is a serious concern. Depending on the application this can be relatively straightforward, since LLMs are regular code tried and true secure coding techniques work great providing that the security requirements are well defined.
Every application is different, but here’s a sketch for a generic application that invokes an LLM. Suppose, for example, a JSON with a prescribed schema prefixed by a standard prompt is passed to the LLM, and the response should be a JSON (different schema). Sandboxing by checking both JSONs in and out with standard input validation techniques goes a long way to preventing trouble and if it happens detecting and rejecting it before bad things happen.
With sandboxing the data transformer use cases are relatively tame. If it’s a poem composition app then worst case it writes lousy poems, no STRIDE worries at all. Once LLM output can influence code execution then the attendant risk derives from a combination of how much influence — from minimal up to arbitrary code execution — and the scale of the blast radius of its potential privileges.
Viewed as sandboxed code, LLM security takes on the look and feel of any other code.