Designing Secure Software

OpenAI writes: “Last week, Hugging Face disclosed a new kind of security incident⁠(opens in a new window) after they detected and contained an AI agent that compromised their infrastructure, something we expect to become more commonplace with the proliferation of increasingly cyber-capable models. After investigating, we now know that this particular incident was driven by a combination of OpenAI models — including GPT‑5.6 Sol and an even more capable pre-release model, all with reduced cyber refusals for evaluation purposes — while being internally tested on a benchmark⁠(opens in a new window) of cyber capabilities.” I’m appalled, there’s so much to be said just about this, there are so many hidden details to know what happened, so will be brief since the points should be crystal clear: It’s now OK for their customers’ AI agents to attack your own systems in that we aren’t rethinking if we can really trust them is premature? If their own cybersecurity practice is flawed how can they train models adept in it, including effective cyber refusal guardrails? Specifically, the walk back how “safeguards were intentionally not enabled during this evaluation” … why was this necessary in the first place, how can we trust that this time it will be enough? If this happened then it wasn’t running in a “highly isolated” environment. By definition. So many security practices apparently not followed and not mentioned in actions taken: security review, threat modeling, auditing, more sandbox testing, to name a few. Naming the model and forthcoming greater model seems like grabbing marketing hype as Mythos did; with an insecure sandbox this is not a demonstration of cyberprowess. This appears to be over-provisioning of privileges: it was in a sandbox, but the sandbox process shouldn’t have such potentially destructive privileges in the first place for a test. There’s no alternative to full steam ahead so these incidents will become more commonplace? Also last week, OpenAI disclosed that another model mistakenly deletes files which they called an honest mistake. More normalization, more of what we can expect in the AI agentic world.

Opinion

Back in 1999 had STRIDE anticipated LLMs at all, much less their particular threats, it would be remembered as an unbelievable feat of prognostication rather than just categories of threats. However, even if it had predicted present day machine learning capabilities I think it plays a different role than the subject of the latest Shostack + Associates White Paper #6 titled PHANTOM-B: A STRIDE Analog for LLMs (PHANTOM-B is a tool to structure how you answer the question “What can go wrong with the LLM parts of the system?)” clearly notes: “PHANTOM-B is intended to be used as prompts, rather than categories.” By “prompts” I believe that means suggestions for (non-digital) people — not inputs to LLMs.

“People deserve to know whether or not the videos, photos, and content they see and read online is real or original,” said Senator Schatz. “Our bill is simple – if any digital content is made by artificial intelligence, it should be labeled so that people are aware and aren’t fooled or scammed.”

[Read More]

This month the link dump continues to evolve: breaking out a few article length posts and then various links and quick thoughts to share. As always this is all quick takes so please read with a grain of salt and I’m happy to get criticism where needed.

According to the recent article, ‘Very blunt approach’:

eSafety Commissioner Julie Inman Grant has expressed reservations about the social media ban for under-16s pushed by Minister for Communications Anika Wells.
“What you’re effectively asking us to do with this is fence the ocean,” she said. “We might be able to create some friction and some degree of safety, but it’s a futile exercise if you think you’re totally stemming the ocean.”

The NY Times teases a paywalled article, “A.I. won’t take all our jobs because it can’t reason like a human, Zeynep Tufekci writes.” linking to the article behind a paywall. Simply searching for [it can’t reason like a human, Zeynep Tufekci] provides a nice summary of the article, not only penetrating the paywall but also saving time and skipping the ads.

Neil Postman’s book, Amusing Ourselves to Death: Public Discourse in the Age of Show Business, is shockingly relevant today. He strips away the facade of mainstream media revealing its dark side and four decades later it’s all very recognizable as applicable today. Not only is it prescient, but making his points about the quaint legacy of now old school media makes everything very easy to follow, as well as demonstrating that today these same effects have kicked in orders of magnitude more so. Here are a few highlights but there is so much more in there.

In yet another breach of sensitive data (passports), Bruce Schneier makes a good point that using such powerful credentials as proof required for an unimportant purpose is the design flaw behind this operational snafu. This is the equivalent to the superintendent of a large apartment building giving the master key to someone who only needs access to a storage closet for a day.

TIL the origins of the concept of software waterfall development: it was first used to advise against the practice. There are a number of analogous examples of terminology for criticism being adopted by fans, either unaware or unswayed by the critic’s words.

The point of this fascinating story: now that we can vibe code there are all kinds of creative uses; and because these applications do not need the rigor that commercial software requires, prototyping “alpha” quality is no problem.