“AI” is already disrupting many institutions simply by extending the reach of digital technology faster and further than ever before. Of course, it is always going further, and the process dates back to ancient times allowing for the reach of mechanical technologies.
Systems evolution in a changing world
Our systems are almost rarely explicitly designed (and if they are unintended consequences seem to be inevitable) and morph as they evolve to adapt to external changes that impact their operation. New technologies that transform how we interface to our systems must disrupt them and conservative institutions will struggle to respond and restore balance.
The key to the kind of re-envisioning I’m suggesting is to unearth the original purpose of the existing system (often long forgotten when practices have become entrenched as “the way we do this”). For example, crossing paths with another hiker we wave our hand as a friendly gesture, yet rarely does anyone recall that this originated as how to demonstrate not holding a weapon in hand.
The latest disruption
AI disruption examples are many, including (just a few):
- Education: students using LLMs to do homework;
- Business: customers using LLMs to file complaints;
- Government: citizens using LLMs to comment on legislation;
- Internet: spammers and scammers using LLMs to customize ads;
- Software: bots submitting pull requests (PR); . . .
Let’s consider that last one as representative of the “friction” dependence. Until recently, people had to spend time documenting, creating, and testing submissions; this meant the incoming rate could be handled by the people on the project evaluating them, and arguably raised quality because it shows motivation, effort, and commitment of our most valuable resource time. Friction is just one of many aspects of how technology impacts or systems, and I assert that precisely when there seems to be no good response to save the legacy system is precisely when there is a great opportunity to reexamine the purpose of our existing processes and deliver more value while adapting to thrive within the new reality.
With the advent of cheap and powerful LLMs, accelerated by coding harnesses, making a PR is perhaps a couple of orders of magnitude faster. Not only does this accelerate the incoming rate proportionally but with the LLM doing the setup and coding and testing and documenting, people with minimal coding skill can no step up to the plate, further increasing the number of submissions.
While the way toward solutions will vary with each instance of this kind of disruption, there’s a common pattern across the different fields listed above:
- AI speeds up creation of incoming to the system;
- AI enhances human capabilities doing much of the job, answering questions;
- AI can easily create software tools that further accelerate all of this;
All of our institutions have established processes with predictable resource allocations to handle the ambient flow of incoming submissions. When all of this gets accelerated and incoming rate ramps up substantially, the legacy process must change when it cannot keep up. All of this background for the discussion following.
The recipient’s dilemma
What is called “friction” here is made even worse by a fundamental asymmetry between submitters and those handling the incoming within the system (the “acceptors”):
- Submitters invest effort (which keeps falling) but can relax quality control: the acceptors will judge the work anyway, and the worst that can happen is the work will be rejected.
- Acceptors do care very much about quality, but it takes effort to properly assess each submission, and as the pace ramps up they end up recreating the famous “I Love Lucy” chocolate factory episode.
The imbalance is multi-faceted: cost in time spent, standards of quality, and there are no checks and balances when submitters can send in all they want as often as they want.
Notably this is much like the attacker’s advantage in cybersecurity. Attackers can keep trying repeatedly until an exploit works, they don’t care if what they do breaks the system in unintended ways, and they only have to find one vulnerability. On the other hand, defenders need to protect all the vulnerabilities, have limited resources, and they definitely must avoid breaking anything in the process of improving security.
Threat modeling perspective
Let me apply what I call “pinhole threat modeling” to the software project incoming PR challenge. The term pinhole means just a small piece of particular interest of what would be a full threat model, which in this case does not exist.
Here we consider the inbox for PRs before the AI onslaught:
- Scope: incoming submissions for the acceptors to triage them into bins “Accept”, “Needs work”, and “Reject”; then the project team reviews the PRs and merges
- Threat: more incoming than the acceptors can handle
- Remedy: submitters need to do more work than acceptors, and this factor ensured the incoming rate is modest and stable
- Assessment: it worked well, and if occasionally the queue backed up some delay is acceptable
NOTE that this breakdown corresponds to Shostack’s four question framework: what are we working on (Scope); what can go wrong (Threat); what can we do about it (Remedy); how did we do (Assessment)?
This threat is in the Denial of Service category, in information security terms it reduces the Availability of developer bandwidth to work on the project.
That is, while it wasn’t a conscious decision, natural friction served to balance inbox flow – and the functioning system became heavily dependent on this mitigation that in hindsight was just a simple expedient. When the world changed and AI burst on the scene, the Assessment suddenly went from “working well” to “melting down” because the assumption of balanced effort collapsed.
Threat modeling lessons
In hindsight it’s clear that the “friction of effort” balance mitigation (which most likely wasn’t even thought about) was more of a stop gap than solution.
Lesson 1: Identify all the threats, even if not threatening today.
Threat modeling current threats is where the big payoffs are, but ideally potential threats not yet problematic should be on the radar. For years projects operated with very modest levels of incoming PRs so the possibilities of rapid submission was hard to imagine. Yet if it takes, say, an hour to handle one and there are only so many hours without slowing down progress in general, it’s a clear future threat.
Lesson 2: Don’t take incidental mitigations for granted.
In this example, the balanced inbox flow just worked out fine, so the threat of being swamped by submissions was hard to even see. The reason this is important because if an unseen mitigation starts failing you are easily caught flat footed.
Without predicting the advent of modern AI obsoleting the cost of effort the advance of technology and continuous efficiency improvement should be our operating assumption – we certainly should never assume these factors stay the same for long. Add to this the asymmetric submitter’s advantage (basically inherent so far as I can see) and we could have seen this coming years ago.
Lesson 3: Consider the implicit assumptions behind the mitigations.
Specifically, the possibility that submissions get easier and easier to create was wrongly assumed. By the way, a lot of speed up was possible without LLMs with the advance of code scanners, instrumentation, better test tooling, automated PR submission tools, and much more.
Lesson 4: React to change by re-examining the threat model.
In this case, when the rate of incoming PRs started to rise significantly reviewing the threat model (or doing a pinhole when none exists) is the first step. This amounts to considering the lessons above:
- Was this looming threat identified and anticipated?
- How are the mitigations working?
- Additional mitigation options not previously considered?
- Are all the assumptions behind the mitigations holding up?
- What else have changed in the threat landscape that impacts this?
These are a few generic questions to ask in a threat model review, and each application will have many more particular to it.
Nonetheless, with many reports of teams suffering under heavy loads of submissions (software projects, but also publishers, solicitations for public comments, and so on) too often it seems that the threat is growing rapidly and nobody can think of any mitigations sufficient to the challenge.
What else?
- get more acceptors – volunteers have limits, effort spend on the inbox detracts for other work, and necessary know-how is limited
- work faster – significant risk of quality drop with bad consequences
- lottery system and so forth – untried and does not look promising
Now what?
“Just use AI”?
The obvious but difficult first idea is to fight fire with fire and the acceptors use AI agents to handle the new incoming flow. This is a big change, lots of work, and fraught with risk due to the submitter’s advantage forcing them to hold PRs to a higher bar than the incoming can be expected to routinely meet.
This gets into the gnarly question of how much human involvement and decision making it important and how much of the workload can LLMs pick up? Surely ready to go product offerings will appear and improve with time, volunteer projects can’t be expected to pay for services, and how and where humans need to be in the loop depends on many specific factors. Even large organizations will surely struggle with these questions for many reasons; one big non-technical challenge is if you depend on LLMs and things go sideways, who is responsible for recovering, and if the LLM isn’t up to the job at that point how do you re-insert humans to mop up? LLM vendors would take responsibility and they can’t fix every customer’s system in any case; the executive ordering the AI will say it was botched if it doesn’t work out; the staff handling deployment will say they were following instructions; …
Retreat and regroup
If AI won’t “magically” solve the dilemma what else is left?
Threat modeling tells us that when we identify a threat that we don’t see how to mitigate and is unacceptable that something has to give. In this case we need to redesign the inbox submissions system, potentially both the incoming side for submitters and inbox the acceptors work from. That is to say that to get out of the cul de sac something has to change.
We are stuck on no good alternatives for handling the inbox process, so that’s the first place to start:
- What is the inbox for? External contributions to improve the product.
- Why accept external contributions at all? Free work, outside perspective, encourage involvement in the project.
- If this worked previously what changed? Volume exceeded capacity to accept and quality dropped (some AI slop).
This simple dialogue reveals some insights about the problem: External contributions on the whole just aren’t up to snuff; Acceptance process is too laborious for the value of effort spent; Contributed PRs may not be aligned with the project direction and goals; and so on.
Each project will have it’s priorities, there are no generic answers here. Perhaps that’s why when the problem is tackled in the abstract it feels insurmountable.
Lesson 5: Threat models (pinhole or full blown or anything in between) can shine a spotlight on known problems to product structured analysis (starting with the four questions + threat categories) revealing underlying assumptions and priorities.
Since assumptions and priorities evolve over time, it’s important to revisit threat models, and a stubborn problem is the perfect reminder that this is overdue.
Move forward
With a fresh look at the submitions inbox process and how it’s going, now we have options and some clarity for choosing a promising path forward.
Real acceptors in a real project would have the context to make decisions, but here an example of some new ideas that might help can be offered to illustrate how this would work in practice.
- Overall have PR submissions produced the expected result or not?
- Is the problem separating the wheat from the chaff, or are there few worthwhile PRs?
- Would tighter submission requirements help raise average quality?
- Can more work for acceptance be shifted to the submitter? (e.g. PRs must include enhanced testing for better code coverage)
- Can some of the acceptance process be automated to lighten the load?
- Prioritize high reputation submitted PRs setting expectation that first timers will wait.
- If most submissions are AI generated, the team can do that themselves, so conditions have changed and there’s no need for contributions anymore.
- Perhaps accept proposed concise prompts as submissions and let the team make the PR from there: doing it yourself is a great way to understand.
These ideas are fairly obvious and not especially great. However, the point is the process that serves up the ingredients to generate useful ideas.
The question of quality
The question of quality hangs over all of this discussion: are the submissions increasing in volume while maintaining average quality, or is it dropping off as automation ratchets up? This is – given the astonishing rate of investment – the trillion dollar question (no exaggeration). If LLMs deliver more bang for the buck even faster, that would be remarkable (setting aside the political and social impacts for now).
Not only is it hard to measure quality in anything but a purely subjective way, the faster everything moves the less time there is to follow along and even think about quality. But it’s crucial to the long-term health of our digital ecosystem: Technical Debt is essentially “accumulation of low quality code”.
However we choose to steer the ship, I’d say that quality is staying the course.
Conclusion
Whether the PRs are from the team or contributions, human or AI crafted, quality work is valuable and should be used in my view. If AI can help or do the heavy lifting and it produces quality code should that be rejected based on principles? If external PR submissions are too difficult to process, even after filtering and applying automation, or add too little value, it may be best to give up.
I don’t know the answers, but I do think that threat modeling helps to see clearly how to deal with challenges like these.
- Identifying threats provides early warning of what might go wrong, as well as seeing clearly after it goes wrong.
- Mitigations need re-evaluation, and often new problems are a sign for just that.
- Assumptions behind the threat model can be outdated or need reassessment.
- Big problems are always a reminder to revist that threat model for new insights.
- Renewing the threat model (even as a pinhole) is the best way to see the system anew.
Bonus: nostalgic history of digital disruption
This pattern of disruption is older than digital technology, but an example from living memory of how the speed of computerization changes the threat model is informative. In this case, when information published in paper form was first made available as data.
We used to have paper phonebooks – the yellow pages for businesses and the white pages for home phone numbers. If you ordered a phone line then they published it in the phonebook, with no consideration of “personally identifiable information” (a completely as yet unknown and unneeded concept). At the time this was considered a service so friends could find your address and phone number to stay in touch: a kind of primitive social networking. Incidentally, as a later service they allowed subscribers to opt out of being listed (that carried a small additional monthly charge for unclear reasons).
Back then authorities such as police departments were issued special “reverse” phone books the cross-indexed phone numbers to names, or addresses to names, for use in investigations. These allowed calling everyone in one block, or tracing the number of a suspicious call to who placed it – possible but totally infeasible to do with paper.
When CD-ROMs for PCs were becoming common, of course they published the phonebook in digital form. But very quickly one of those unintended consequences reared its head: suddenly it was easy to scrape (to use the modern term) and index the phonebook. Now there was no way to put the phonebook in digital form without effectively giving everyone the “reverse” index as well because it was so easily derived. I suspect this was a major factor in phonebooks (any media) going away for home subscribers, and yellow pages are now a sad remnant of the past.
One of the quaint practices seen in the yellow pages could be seen as a precusor to search engine optimization. In an effort to be found first when people were looking for a service, business names such as AA Towing competed to be listed first; this led to escalation with more A’s, going up to seven or eight.
In college when I was working on public key cryptography deployment designs that led to Digital certificates, the first idea that the inventors of RSA had was to publish a phone book with public keys listed instead of phone numbers. At least keys for practical use were much smaller then so less typing required.