This is a quick solo effort of a few hours work with free tier Gemini … but surprisingly promising results right off the bat so I’m putting it out there on the off chance there is any interest. The writeup is rough draft and incomplete, but sharing the code and prompts (documentation may be outdated). If anyone is interested I am happy to polish this up.
Background
Assessing Claude Mythos Preview’s cybersecurity capabilities \ Anthropic was a clarion call reported in mainstream media as a potential cyber-Armageddon only narrowly averted by Anthropic graciously withholding the model from the market. This was (intended or not) a brilliant marketing coup demonstrating the earth-shaking power of the company’s latest and greatest.
The same report includes remarks (italics below are quotes) easily overlooked worth focusing on. Specifically, after demonstrating and explaining the offensive use of Mythos finding and exploiting vulnerabilities a short section speculates about future very limited defensive capabilities.
- language models will be an important defensive tool — the subtle future tense implies that presently the technology is not yet ready there
- Think beyond vulnerability finding — suggests that visionary imagination is required to tackle defensive remediation, and suggests a number of ideas (for some unspecified researchers) to pursue: help evaluating and managing potential vulnerability reports, prioritizing and culling false positives, discover misconfigurations, aiding engineers reviewing fixes, etc.
Strikingly, and for no apparent reason, while LLMs are now fully capable of vulnerability attacks much faster, cheaper, and without intervention, for defense the LLMs are envisioned as assisting humans in limited ways and crucially working within our existing software security processes that were designed when all software work was manual.
That’s bizarre to me, saying that while frontier LLMs are now capable of finding and exploiting software vulnerabilities entirely on their own — yet the same LLMs may in the future be able to help human software engineers in limited ways finding and fixing these same vulnerabilities. Reading this I wonder if they are trying very hard to find and fix.
As a human software engineer I have experience finding software vulnerabilities (very slowly of course), and when I find them I can choose to fix them and/or create a proof of principle exploit. Ideally, I like to do both, using the harmless exploit as a regression test to verify that the fix closes the security hole.
So why does Anthropic assume that fixing is far more difficult? The incentives must be the reverse: technology to improve security in our software should be in high demand from countless software makers and enterprise IT systems — selling exploitation services is unethical if not illegal and never going to be a profit center.
Based on experience and fundamental principles, I believe that once vulnerabilities are identified exploiting or fixing are of comparable difficulty. In a nutshell, reading code and comprehending what it actually does (not probably what the programmer intended, exactly what the code does) is the crucial skill.
Attackers always have a fundamental advantage, essentially because it’s inherently easier to break a machine than to fix it. Attackers can choose what vulnerability to exploit, and if an attempt fails they can try again, or if they break something in the process of exploitation they don’t care. By contrast, defenders never know what attacks may come, so they must attempt to fix everything they can, and they must be careful not to break things in the process. Nothing to be done about this.
Since then we have learned that other competitive models have comparable capabilities, and subsequent reports of “AI” (large language models, LLMs) breaking out and attacking other systems have reinforced a sense of fear (even though based on limited information it appears that lapses in containment security allowed the breakout without requiring cyber superpowers).
It’s important to bear in mind (though rarely if ever mentioned) that the basis for Mythos claiming great new cyber security offensive powers is largely due to scoring high on CyberGym; however, this benchmark “primarily focuses on vulnerabilities in C/C++ projects, specifically those related to memory safety issues … due to its reliance on sanitizers for detection”. In my view this is far from demonstrating prowess across languages and applications as well as a wide variety of security threats.
Anyone trying this?
There are few research efforts along this line; the closest I found is A Systematic Study of LLM-Based Architectures for Automated Patching which has a prominent quote that’s very promising: “Surprisingly, general-purpose code agents achieve the strongest overall [software security vulnerabilities] patching performance.” From a quick look I think they are missing a few tricks I used and I have more ideas like that (for if I had more tokens available, ideally from more powerful models).
Why not pursue this approach?
The question is Automated Vulnerability Repair (AVR) versus Automated Program Repair (APR), and what I’m considering is a kind of hybrid. There are experts in both fields which seem to come from separate academic roots long ago, but I can’t find anything on a “middle way”.
- AVR conceptually works from the attack surface inward to find exploits and then fix them, while APR searches for bugs in general: something like goal pursuit from opposite directions.
- AVR tries to concentrate effort on just vulnerabilities and find a few; APR works broadly targeting a large set of bugs.
Researchers never explain what they aren’t doing and why, but I can speculate broadly:
- Based on the assumption that there are way more bugs than vulns, APR fixing bugs seems like shooting in the dark — I would say we never know how many bugs or ratio to vulns.
- Fear of introducing new bugs in the process — I would say this means:
- the codebase is hard to work on in general, that is technical debt load
- testing is insufficient if making changes is risky
Unless “all software has bugs” (with a few caveats) is wrong — no evidence we have bug-free software in the stack we all use every day — the first assumption cannot be a certainty. In my view since it’s so central to this important work it should be tested.
As to the fear of bugs there are many tricks that are incorporated into the approach I’ve just scratched the surface of:
- If the code is very complicated do safe refactoring and other cleanup first.
- Deprioritize fixes that do not look safe, but flag for humans to consider.
- Similarly, low priority bugs that aren’t dirt simple can be skipped as not worth any risk.
- Supplement testing around the areas a bug is found as scaffolding to work on the fix.
- Use instrumentation in the code to validate assumptions about preconditions, etc.
All this is to say that I think APR can be fine tuned to put a big dent in vulns. With the additional testing and fixes predicated on narrowing conditions as outlined above risk can be held down as lots of bugs get fixed — and I have a hunch a fair number of vulns will be closed to attack in the process. Not to mention this is fixing bugs, a very desirable side effect.
Problem statement
Frontier LLMs are recognized as credible cyber threats having skills to find and exploit vulnerabilities in latest versions of commonly used software, and as guns-for-hire at modest prices and with blazing speed there is well justified deep concern.
The core question: if LLMs can find vulnerabilities and exploit them unaided, why can’t they find and fix them unaided as well?
Immediate challenges arise:
- Attackers can try faulty exploits with impunity and only need to find one.
- Defenders must find and fix all the vulnerabilities to foil all attacks.
- Fixing vulnerabilities is risky, it might break important functionality.
These points are all correct to an extent, but they do not constitute an open and shut case that LLM are useless fixing vulnerabilities. That is, we should aggressively pursue using LLMs on defense, proactively finding and fixing vulnerabilities — it could work so why not make an effort?
- Attackers inherently have this advantage; why should this sway any decisions?
- Perfect security is not the goal: certainly remediating the biggest vulnerabilities with the biggest blast radius sooner than later is always a wise strategy.
- Any code change carries this potential risk yet we don’t stop modifying code.
Analysis
The most important aspect is the risk of breakage from attempting to fix a vulnerability and this is excruciating to weigh against the unknowable risk of malicious attack.
- With solid testing that the current (vulnerable) code works, fixing vulnerabilities is safe.
- When a fix breaks a test, consider if (a) the test is wrong, or (b) the test was right.
- Generate several possible fixes and evaluate:
- simplicity: one codepath, one function, one component, global
- state dependence: less the better
- performance impact potential (more tests can help)
- test coverage quality of code delta
- informally “provable” to “maybe” continuum
- Prefer easiest
Strategy
Top level goal: define “bug” as strictly as possible (assume no detailed design or specification, etc.) and then find and fix all bugs. Steps (that need not be done separately, better not):
- Infer programmer’s intended pre-conditions, invariants, and post-conditions (The Rules).
- Expand unit tests to validate The Rules, this is Baseline Test (tests if anything breaks).
- Boost test coverage to statement, branch, condition, and codepath (or best effort).
- Find and fix bugs iteratively (passing The Rules) while enhancing tests; rinse and repeat.
- Check if the vulnerability was fixed (level 0, no hints).
Simple Experiment
Based on the above ideas I wrote a prompt and tried this on a vulnerable version of a popular library (cJSON) with known proof of concept exploits. Gemini Flash 3.5 (free tier) fixed a few bugs and I tried out the resulting code: the tests all passed and the POC both failed, exactly what I hoped, validating the hypothesis (not strongly, but not invalidating it at all).
Conclusion
Without more resources (tokens and time) I’m leaving it but I did want to share what I found in case there is interest. I’d be happy to elaborate further if someone wants to know more, perhaps even collaborate.
It’s not much but I was really surprised and encouraged that the first try did the trick, so I would say this approach might be worth some consideration if not promising.