Currently all the talk is about how frontier models (e.g. Mythos) can be used offensively — and not talked about in polite company, probably most, if not virtually all, enterprise and home systems likely are sitting ducks. In addition to that threat (call it number 1) of offensive LLM vulnerability finding, there is also (2) attacks on the model surface area by poisoning training data and prompt injection (etc.); and also (3) self-inflicted harm (these are not attacks) by the LLM acting in unanticipated ways (so-called “hallucination” etc.). Incidentally, it’s unclear that the third of these types is well understood as we keep hearing reports of exactly such trouble. This is about threat modeling an application when it invokes an LLM and specifically the third scenario.
[Read More]See latest writings about software security and a little miscellania.
Kohnfelder, Loren. Designing Secure Software: A Guide for Developers. No Starch Press, 2021.
Designing Secure Software consolidates more than twenty years of experience into a concise, elegant guide to improving the security of technology products. Written for a wide range of software professionals, it emphasizes building security into software design early and involving the entire team in the process.
The book begins with a discussion of core concepts, covering trust, threats, mitigation, secure design patterns, and cryptography. The second part, perhaps this book’s most unique and important contribution to the field, covers the process of designing and reviewing a software design with security considerations in mind. The final section details the most common coding flaws that create vulnerabilities, making copious use of code snippets written in C and Python to illustrate implementation vulnerabilities.
You’ll learn how to:
- Identify important assets, the attack surface, and the trust boundaries in a system
- Evaluate the effectiveness of various threat mitigation candidates
- Work with well-known mitigations and secure design patterns
- Understand and prevent vulnerabilities like XSS and CSRF, memory flaws, and more
- Use security testing to proactively identify vulnerabilities introduced into code
- Review a software design for security flaws effectively and without judgment
“The writing in this book is very clear and easy reading, and the examples used are both captivating and easy to understand. Kohnfelder does a great job of making a point that is easy to understand, and most of the chapters could stand alone for developers just working in that one particular area.” (read the full review)
LLM Cyber defense
This is a quick solo effort of a few hours work with free tier Gemini … but surprisingly promising results right off the bat so I’m putting it out there on the off chance there is any interest. The writeup is rough draft and incomplete, but sharing the code and prompts (documentation may be outdated). If anyone is interested I am happy to polish this up.
[Read More]Anti-requirements in prompts
As we continue to see all the “AI goes rogue” in testing reports, I keep wondering if this isn’t the aptly named “genie effect” (hat tip: Bruce Schneier), and if so if anti-requirements might help. Generally speaking I’d say they are necessary but not sufficient: if the humans are surprised when an LLM does a thing that should be covered in the prompt.
[Read More]Threat modeling post-quantum strategy
Racing to beat an unknowable deadline ahead of quantum computing breaking legacy cryptography, achieving the defenses we would wish to be in place in time, or to meet announced stated goals varying by country has a long way to go. However, what exactly are we wishing for and what minimum level of defense would be acceptable assuming we will not get all that we wish for? Threat modeling would answer this clearly and more importantly explain the thinking behind the goals, but I can’t find one.
[Read More]July 2026
AI continues to dominate the news, largely involving security, and not in a good way.
- Role confusion: one more reason we can’t trust LLMs
- AI agent parody?
- Threat modeling Duress code data wiping
- More on OpenAI/HuggingFace
- Normalizing cybersecurity facepalms
- PHANTOM-B: threat modeling systems using LLMs
- Toward Better AI Legislation
From around the web:
[Read More]Role confusion: one more reason we can’t trust LLMs
Prompt Injection as Role Confusion is my new favorite paper about a very obvious threat in hindsight that’s hard for us humans to see because we anthropomorphize LLMs so naturally. When Obi-Won Kenobi tells the stormtroopers that “These are not the droids you are looking for” to pass the checkpoint that’s role confusion: the guards foolishly think his words are their own thoughts. The very readable blog-style writeup explains the details, but I want to focus on the threat model perspective which is my bread and butter.
[Read More]Threat modeling Duress code data wiping
The GrapheneOS data wiping function is in the news due to a customer now facing serious legal jeopardy for using it at a border crossing. I don’t know the details of the case not exactly how the duress feature works, but from public knowledge we certainly can talk about learning from this.
[Read More]AI agent parody?
This HuggingFace security incident disclosure has people talking about AI agent security. Today I saw such absolute positive spin that I found myself thinking “this must be a parody”: looking at the context I’m pretty sure that it isn’t … though some parodies stay in character all the way through.
[Read More]More on OpenAI/HuggingFace
The recent AI agent security debacle must be reverberating quite a lot within the walls of the major proponents of AI agentic technology because they announced a brand new “movement” apparently with zero details available yet. If that isn’t a sign of flat-footedness then I don’t know what is.
[Read More]Normalizing cybersecurity facepalms
OpenAI writes: “Last week, Hugging Face disclosed a new kind of security incident(opens in a new window) after they detected and contained an AI agent that compromised their infrastructure, something we expect to become more commonplace with the proliferation of increasingly cyber-capable models. After investigating, we now know that this particular incident was driven by a combination of OpenAI models — including GPT‑5.6 Sol and an even more capable pre-release model, all with reduced cyber refusals for evaluation purposes — while being internally tested on a benchmark(opens in a new window) of cyber capabilities.”
[Read More]