Recent Writings — Loren Kohnfelder

As we continue to see all the “AI goes rogue” in testing reports, I keep wondering if this isn’t the aptly named “genie effect” (hat tip: Bruce Schneier), and if so if anti-requirements might help. Generally speaking I’d say they are necessary but not sufficient: if the humans are surprised when an LLM does a thing that should be covered in the prompt.

This is a quick solo effort of a few hours work with free tier Gemini … but surprisingly promising results right off the bat so I’m putting it out there on the off chance there is any interest. The writeup is rough draft and incomplete, but sharing the code and prompts (documentation may be outdated). If anyone is interested I am happy to polish this up.

Currently all the talk is about how frontier models (e.g. Mythos) can be used offensively — and not talked about in polite company, probably most, if not virtually all, enterprise and home systems likely are sitting ducks. In addition to that threat (call it number 1) of offensive LLM vulnerability finding, there is also (2) attacks on the model surface area by poisoning training data and prompt injection (etc.); and also (3) self-inflicted harm (these are not attacks) by the LLM acting in unanticipated ways (so-called “hallucination” etc.). Incidentally, it’s unclear that the third of these types is well understood as we keep hearing reports of exactly such trouble. This is about threat modeling an application when it invokes an LLM and specifically the third scenario.

Racing to beat an unknowable deadline ahead of quantum computing breaking legacy cryptography, achieving the defenses we would wish to be in place in time, or to meet announced stated goals varying by country has a long way to go. However, what exactly are we wishing for and what minimum level of defense would be acceptable assuming we will not get all that we wish for? Threat modeling would answer this clearly and more importantly explain the thinking behind the goals, but I can’t find one.

AI continues to dominate the news, largely involving security, and not in a good way.

• Role confusion: one more reason we can’t trust LLMs
• AI agent parody?
• Threat modeling Duress code data wiping
• More on OpenAI/HuggingFace
• Normalizing cybersecurity facepalms
• PHANTOM-B: threat modeling systems using LLMs
• Toward Better AI Legislation

From around the web:

Prompt Injection as Role Confusion is my new favorite paper about a very obvious threat in hindsight that’s hard for us humans to see because we anthropomorphize LLMs so naturally. When Obi-Won Kenobi tells the stormtroopers that “These are not the droids you are looking for” to pass the checkpoint that’s role confusion: the guards foolishly think his words are their own thoughts. The very readable blog-style writeup explains the details, but I want to focus on the threat model perspective which is my bread and butter.

This HuggingFace security incident disclosure has people talking about AI agent security. Today I saw such absolute positive spin that I found myself thinking “this must be a parody”: looking at the context I’m pretty sure that it isn’t … though some parodies stay in character all the way through.

The GrapheneOS data wiping function is in the news due to a customer now facing serious legal jeopardy for using it at a border crossing. I don’t know the details of the case not exactly how the duress feature works, but from public knowledge we certainly can talk about learning from this.

The recent AI agent security debacle must be reverberating quite a lot within the walls of the major proponents of AI agentic technology because they announced a brand new “movement” apparently with zero details available yet. If that isn’t a sign of flat-footedness then I don’t know what is.

OpenAI writes: “Last week, Hugging Face disclosed a new kind of security incident⁠(opens in a new window) after they detected and contained an AI agent that compromised their infrastructure, something we expect to become more commonplace with the proliferation of increasingly cyber-capable models. After investigating, we now know that this particular incident was driven by a combination of OpenAI models — including GPT‑5.6 Sol and an even more capable pre-release model, all with reduced cyber refusals for evaluation purposes — while being internally tested on a benchmark⁠(opens in a new window) of cyber capabilities.”