We are at an important inflection point with at least two fronts on the ongoing challenges of software security that AI is impacting at unprecedented speed.
- the rush to offload more and more software engineering work onto LLMs, at best without any reliable solution to keep up responsible reviews without hampering the speed-up and cost savings which are the point (and at worst without protections against seriously compromising what security and reliability we have already);
- the threat of bad actors leveraging frontier models to exploit systems;
Right now the latter is getting all the attention so I am focusing on that here. Specifically a new open letter signed by many of the big players: A call for collective action on cyber defense: An open letter for a global surge in cyber defense.
Before getting to the topic, I want to put a pin in #1 above for another day: it’s very important, and the solution push here for #2 is (obviously) more of #1.
This is written blog-style as I started looking into this and trying to interpret what this open letter means to say, and what might be behind it left unsaid.
To get warmed up, my first question is about the absence in the signatories of a few of the biggest of the big software companies: Apple, Meta, Salesforce. Why aren’t they supporting a call for such an urgent collective action? Without them on board our collective action is hampered from the get-go: did they disagree with some premises or the strategy? Without naming names, some acknowledgment of the behind-the-scenes debate (I can’t imagine they did not participate or consider signing) we cannot help but wonder what the problem could possibly be.
Quotes below like this are all from the letter.
Caveat: I’m not saying that I know better, but I do have questions. This is a quick take for now, if anyone is interested I’m glad to elaborate.
We have a limited window to strengthen cyber defenses.
This is both a vague (years, months, weeks?) and ominous warning that I would say deserves at least one verified, non-classified incident before calling for “industry and government to bring the full weight of their technology, resources, and expertise to this effort”.
Is there any evidence that August 2026 frontier models are that much superior for this purpose than they were, say, two months ago? Supposedly little technical expertise is required with the big models so if the feared attacks are possible it should have started I would think. In my view, the signatories could have provided more evidence to back the extraordinary claims so others could understand the motivation for this broad and unprecedented initiative impactin countless systems and software components.
In the coming months, AI-enabled cyber attacks will become far more widespread and sophisticated as models around the world become increasingly capable.
There have been musings that AI will take off in capabilities suddenly for years, but this time we’re sure? Perhaps they will plateau, or perhaps we aren’t leveraging them well yet and they are better than we know. Can anyone say for sure?
Today’s AI advances are already giving defenders new ways to fix weaknesses …
Mythos preview (April 2026) only offered that we should “Think beyond vulnerability finding”, not start fixing. They suggest that “models can also accelerate defensive work in many other ways”, including, issue reporting and triage, write repros and reports, aid review, etc. That is, help humans who are doing the real work. Translation: this will all go at human speed for the time being.
With LLM driven attacks at inference speed, human defenders even greatly aided are not going to be a match if this threat is for real, considering that:
- very few software engineers have experience doing this work (and they all have day jobs);
- the number of vulnerabilities is vast (and nobody even knows its scale).
Unless the “new ways” are beyond what the April report lists, then if the premise of all the coming attacks is true I’d be very worried.
Recognize that status quo security won’t be enough.
This has been the case in general for two or three decades in my view.
Even if the “historically under-resourced” systems are given AI tools, without expertise (given that fixing is gated by human participation) this will be a flat-footed effort.
I’m not saying this should all be worked out – clearly it can’t be – but my point is we need a more detailed view of the current facts on the ground and realistic assessment of the problem to better envision how this might all work. Without transparency, collective action is far more difficult.
It’s a hard problem, but I think it’s safe to say that it’s way more than a technology problem and that the highly competitive software is not exactly known for collective action without market or legal pressure.
- Should “Every organization” independently do all that security work, and given that they haven’t for a long time we need to understand why not.
- Haven’t the cybersecurity companies and partners supposed to have doing all this stuff for many years against conventional attacks?
- Haven’t governments been trying to coordinate defense and collect incident reports for many years with little impact?
- AI companies granting access is great but defenders need experienced people too (and there can’t possibly be enough of them).
Additionally, I would suggest AI companies also consider backing research toward LLMs fixing vulnerabilities (not just finding and helping fix), making pull requests easier for humans to assess for both blocking exploitability as well as lowering the risk of introducing a new bug.
In my view, fixing vulnerabilities may not be that hard).
On top of this, it’s very well known that patching complex systems is risky and there’s no mention of balancing that against a call for urgent action at break neck speed.
Personal opinion: I think is very important to ask for each system, how much security improvement is sufficient? (This is a notoriously hard, yet important question.) It’s great when new tools are applied where the need is great, but we have no idea how much change will be required to bring secure defenses up to snuff – or how we would even know when it’s good enough. Pointing advanced tools at older systems (which I think is a fair assumption for the general case, more so for the under-resourced) could produce an effectively unbounded spew of potentially serious issues. Then what?
One last question for now: why are many of our infrastructure systems “historically under-resourced” in the first place? Software technology aside, securing our utilities, healthcare, and other critical systems is a glaringly obvious priority that it does not take cybersecurity expertise to recognize.
This open letter presents a strategy aimed at the technical challenge, but in the context where infrastructure security has not been a high priority, without first understanding those root causes first that approach is unlikely to be sufficient in my view.
Returning to the letter’s opening line, what exactly does “limited window” mean? To me this is a veiled warning that if you act too late the game is lost. Do they mean an adversary can take down parts of infrastructure at will, or possibly irrevocerable infrastructure destruction or take over? It’s horrendous to consider, but are there no physical overrides with manual operation in the worst case, or do we throw up our hands in the event of a remote software attack?
Threat modeling is how I make vauge threats more concrete, and how I regularly advise anyone every chance I get. In this context that means beginning with each infrastructure system making a basic threat model that would answer questions such as above.
Finally I wonder if there might not be a lot we can do without a massive AI call to action. Maybe we don’t need a huge infusion of new (bleeding edge?) technology that requires experts to oversee and carefully hone patches for safe merges. Just to toss out a few obvious things that in general make systems vulnerable:
- Perhaps simply installing backlogged patches makes it much more secure (note that if this is difficult then all the best AI patches in the world are not easily deployed either).
- Perhaps an audit finds that a subsystem thought to be air gapped isn’t and it can safely be disconnected or a compatible replacement found that doesn’t require an internet connection
- … depending on the system there are many other common vulnerabilities like this that self-review can identify that are safely mitigated
Unless the resourcing and expertise gap is somehow quickly filled, I would urge these system owners to consider these sort of steps (which are all very doable right now and not nearly as much work) now until proven AI assistance becomes available to address remaining weaknesses.
I have no inside access to any of this, using public information, so all this is just one opinion. Nonetheless, I do have all these questions which I believe are quite relevant if not actually important if nothing but to make the underpinnings of the open letter more apparent and easy to see the logic of.
Despite all this, I really do because Gemini critiqued this, and it actually came up with a great summary in closing (to which I would only add transparency): technological acceleration without root-cause analysis and operational readiness is a recipe for churn, not security
In closing I’m going to bury the lede because I was puzzled by one phrase about agentic identities, but I think I figured it out. Here’s what the letter calls on AI companies to do as the major technical response to the coming threat that they foresee:
Build observability and security tools, ensure agentic identities are traceable and accountable, and share best practices in continuous monitoring.
All the way through I was assuming that the direly needed defensive AI response was LLMs working alongside the development team to carry as much of the load as possible, e.g. triage, coding, reviews, testing, and more mentioned above. However, for that work you don’t need to worry about “agentic identities” since software engineers are invoking agents to work along side them, and they can always just ignore sloppy pull requests or other bad input from the LLMs. Then the light went on, it was at once so obvious and deeply unsettling: if you have AI agents in production then “traceable and accountable” are extremely important when something goes wrong.
So I’ll close with still more (to me) unfathomable questions:
- Could it be that the risk they are talking about here be unsafe deployment of AI agents in live systems? I dearly hope not: but what else could it mean for defense against AI? Certainly the attackers are not going to cooperate and hand us the identities of their agents!
- Can anyone please connect the dots for a different interpretation of how that call for action relates to the goal of the letter in another way?
- Is this the imminent “limited window to strengthen cyber defenses”, that the signatories refer to, in part at least, defending against our self-deployed AI agents running amok?