Currently all the talk is about how frontier models (e.g. Mythos) can be used offensively — and not talked about in polite company, probably most, if not virtually all, enterprise and home systems likely are sitting ducks. In addition to that threat (call it number 1) of offensive LLM vulnerability finding, there is also (2) attacks on the model surface area by poisoning training data and prompt injection (etc.); and also (3) self-inflicted harm (these are not attacks) by the LLM acting in unanticipated ways (so-called “hallucination” etc.). Incidentally, it’s unclear that the third of these types is well understood as we keep hearing reports of exactly such trouble. This is about threat modeling an application when it invokes an LLM and specifically the third scenario.
When using LLM in an application, never forget that it’s stochastic and to a large degree the model parameters are a black box, so if you don’t sandbox it then literally it might do anything. Bear in mind that a Remote Code Execution vulnerability allows an adversary to run code in your system that might do anything. The odds may be small but I don’t think we know how to estimate that, and in my view, it’s a matter of when not if. Since the model has billions of parameters you certainly cannot thoroughly test it or know exactly how it works.
On top of all that, beyond stochastic, LLMs are non-determinant (hardware does not guarantee operations are done in the same order even with identical inputs). Furthermore, writing great prompts is not an easy thing and it’s easy to implicitly or explicitly direct the LLM to do something other than you intended (the so-called “genie effect”).
If you do sandbox your LLM then your threat model will need to analyze that: in other words, assume all possible inputs and app possible outputs and maximal use of its privileges and consider what could go wrong. Unless the sandbox is extremely restrictive that’s a lot of work to do well.