- The real AI; Is AI profitable?; Very Important Words the Tech Industry Ruined
- System modeling
- Guardrails?
- Anthropic sandboxing
- What’s a Virus?
- Threat model or triage guideline?
- Big Tech priorities
The real AI Seth Godin, based on Woz’s definition of “AI” (spoiler: Actual Intelligence).
https://isaiprofitable.com/ If this website is even in the ballpark with its numbers, AI economics sure looks like the Mother of all Bubbles. My question: is this all in pursuit of the mythical “AGI” (which hypothetically has amazing new powers) or just gaining a lead for dominance in the market? Also according to U.S. AI Data Center Delays: 7 GW Capacity Crisis prospects don’t look good.
NOTE: I didn’t fact check any of this.
Very Important Words the Tech Industry Ruined is along the same lines I fear. I don’t toot much, much less link to it, but I replied to a thought-provoking post wanting to contribute.
System modeling
Threat modeling is only as good as the System Model used for analysis. Often Data Flow Diagrams (DFD) are used as the basis, but these are limited to data flow as the name tells us, and there is a lot more to understanding a system than knowing where the data is: a good model fleshes out the other important aspects. Thus, building a good System Model is a key step to get right in order to get a good threat model.
When creating a thorough DFD is considered too formal or too much work , threat modeling is based on an implicit System Model either sketched on a whiteboard (as one example) or even in people’s heads. Almost always this also is a weak subset of a proper System Model, limiting how effective threat modeling can be.
What I mean by a System Model is an abstract representation (graphical and/or text, etc.) of the whole system in scope, including structure, behavior, and boundaries. The model can be coarse or fine grained depending on how coarse or fine grained the threat model should be, trading off more detail versus simplicity. By the way a System Model has many other uses: explaining the system to others not technically inclined; for reference by developers; to guide operations.
Guardrails?
So-called “AI guardrails” are only effective to mitigate unintentional harm but effectiveness for the reckless, careless, and malicious (IMHO) they should never be considered rock solid restrictions. Guardrails will not protect a democracy from those intent on destroying it, so we should stop pretending that they help.
The name guardrail is apt: it’s a mitigation to reduce harm, a “brick wall” that’s impenetrable. According to the US Transportation’s Guardrails 101, “This is not to say that guardrails can completely protect against the countless situations drivers may find themselves in. The size and speed of the vehicle can affect guardrail performance. So can the vehicle’s orientation when it strikes the guardrail. There are many other factors.” That is, they cannot thoroughly test a guardrail to even a simple metric (e.g. stays on the road) — software is a little more challenging.
“A barrier is crashworthy if it meets the crash test criteria in effect at the time of the testing and established for that type of roadway safety device.” What exactly are “AI” guardrail test criteria? These must be concretely defined in order to be tested.
Note that Guardrails 101 does not even consider effectiveness against intentional malicious crashes for obvious reasons, but “AI” guardrails are largely for that purpose.
Anthropic sandboxing
How we contain Claude across products opens with a position that strikes me as misguided. Based on a “blast radius” harm times likelihood model, “as agents become capable of doing work that once required a person or even a team, the cost of not deploying grows large enough that the risk-reward calculation tips heavily toward adoption, as long as products can be made safe.” For a large corporation, potential significant harms will greatly exceed the cost of a reliable team. Containment of harm, estimating maximum damage and cost of remediation, as well as likelihood amounts to making predictions and with such new powerful technology the potential margin for error in the risk calculation is enormous.
Additionally, this analysis misses the big question that everybody gung ho about AI misses: when you replace humans with AI what about quality, reliability, and ultimately responsibility? At best these are open questions. Anthropic says in the article that only in the last year have they come around to trying this so it’s hardly tried and true. In any case, get one of these factors wrong and you have a big problem:
- Quality can decline without being noticed but when you do notice it’s very hard to remedy
- Once dependent on AI, should its work become unreliable then what’s Plan B?
- Humans are responsible for their work (ultimately they risk job loss and replacement) but unless the AI vendor takes responsibility (hard to imagine) how will?
The core protection pattern — “Three components to defend: the model, the environment in which it runs, and the external content the agent can reach.” — appears to be a sandbox with input validation and constrained privileges: nothing new here. Since you don’t know exactly what an AI agent will do (the model is a black box) it needs a sandbox like any other untrusted code.
We’ve heard many reports of security failures due to over-trusting AI — either AI agents, or code produced by LLMs — but very little transparency as to the details. An open demonstration of how all this works, and how reliable it is, deployed in an enterprise scale system would be extremely helpful to really evaluate this new technology. Instead everyone seems to be winging it internally.
I wonder if all the sandbox containment code is written by humans or by Claude Code? (Because if it were really smart it would open an escape door for itself.)
What’s a Virus?
I ran across Computer Viruses – Theory and Experiments (1984, Fred Cohen) which concludes: “absolute protection can be easily attained by absolute isolationism, but that is usually an unacceptable solution. Other forms of protection all seem to depend on the use of extremely complex and/or resource intensive analytical techniques, or imprecise solutions that tend to make systems less usable with time.” I disagree: most modern OS (modulo vulnerabilities that usually get fixed) are largely impervious to viruses with the exception of administrator error (e.g. installing malicious kernel code).
“We define a computer ‘virus’ as a program that can ‘infect’ other programs by modifying them to include a possibly evolved copy of itself.” This definition applies to a system already infected and indeed (obviously, I would say) once infected it is extremely hard to impossible to excise it. Real systems are hardened (not easy but quite doable) against penetration, so Cohen’s definition of a virus never gets started. Theory and practice do often diverge, but this seems like bait and switch.
Threat model or triage guideline?
The document “The Linux Kernel threat model” does not appear to be a threat model by my definition of one because it does not identify any threats. It says things like “The Linux Kernel applies a certain collection of default settings that match its threat model,” but this is referring to some other (unidentified) threat model — pretty odd to do that within a threat model.
This document lists a number of bugs saying, “In the Linux kernel's threat model, the following classes of problems are **NOT** considered as Linux Kernel security bugs.” That is, this document appears to be a bug triage policy that gives special attention to security bugs according to some unknown threat model. Same thing for the Python Red Team threat model: looks like a triage bar to me. By triage bar I mean criteria for an issue to be classified as a vulnerability of sufficient severity to receive special attention.
This is more than a pedantic observation: I believe threat modeling is an essential tool to secure software design so “is there a threat model?” needs a clear answer because a triage bar in no way serves as a threat model. Hoping I’ve somehow missed something here because if nobody notices, or cares much, that does not bode well for software security.
Big Tech priorities
By now most of us know — including not technical people who might only sense it intuitively — that Big Tech priorities are largely their own (Cory Doctorow coined the term “enshittification” for the dark side of this) and their product maps are based on internal goals, not better serving customers. That is almost all mass market software, including the web, are take it or leave it, and this of course increasingly it’s harder and harder to leave it.
It’s unheard of for the biggest corporations to ever (and they have lots of products many millions of people use) break this rule. This includes: not delivering features many people want, poor usability for the least tech savvy users, new features nobody asked for, discontinuing support for apps many depend on, and denying easy opt out for functionality many don’t want, and more.
One larger point about priorities (important if correct, but not discussed at all) to consider — this is speculation but I think it’s on fairly solid ground. Currently “AI” is in the “gold rush” phase, completely dominating (for better or worse) the software industry and by extension virtually all large corporations since they already depend on software. While exactly what corporations are using (or planning to use) AI for, an educated guess is that it’s largely internal applications: workflow automation, data analysis, routine paperwork, software development acceleration, and so on. Whatever it is, where they apply AI tells us what their priorities are because if it’s so great it should be deployed to the most important work as companies see it.
I see tons of opportunities for AI to improve products, customer support, integrate feedback into roadmaps, and improve usability of apps and devices. For example: LLM help with settings; noticing when the user does the same thing repeatedly and offering to do it for them; detect “ghost touches” (inadvertent input) and warn first, or offer to restore; explain why what the user tried didn’t work; help find things (files, browser tabs or history, email, etc.) across apps; explain and/or reduce notifications, especially confusing ones; simple data extraction and filtering (e.g. flight details in a long HTML email or page can’t be copy/pasted easily); and more. Instead almost all end users get is content generation, and many don’t really want that or know how to use it properly.
I see no evidence of anything like this happening or even being considered: if it is, great; if not I think it tells us a lot about true corporate priorities with respect to their “valued customers”.
‘Hard choices, easy life. Easy choices, hard life.’ — Jerzy Gregorek
(applies to living a good life, and also in my view it applies to software development, too)