Ai on Designing Secure Software

Questions about Collective action on cybersecurity

Sat, 29 Aug 2026 00:00:00 +0000

We are at an important inflection point with at least two fronts on the ongoing challenges of software security that AI is impacting at unprecedented speed.

the rush to offload more and more software engineering work onto LLMs, at best without any reliable solution to keep up responsible reviews without hampering the speed-up and cost savings which are the point (and at worst without protections against seriously compromising what security and reliability we have already);
the threat of bad actors leveraging frontier models to exploit systems;

Right now the latter is getting all the attention so I am focusing on that here. Specifically a new open letter signed by many of the big players: A call for collective action on cyber defense: An open letter for a global surge in cyber defense.

Before getting to the topic, I want to put a pin in #1 above for another day: it’s very important, and the solution push here for #2 is (obviously) more of #1.

This is written blog-style as I started looking into this and trying to interpret what this open letter means to say, and what might be behind it left unsaid.

To get warmed up, my first question is about the absence in the signatories of a few of the biggest of the big software companies: Apple, Meta, Salesforce. Why aren’t they supporting a call for such an urgent collective action? Without them on board our collective action is hampered from the get-go: did they disagree with some premises or the strategy? Without naming names, some acknowledgment of the behind-the-scenes debate (I can’t imagine they did not participate or consider signing) we cannot help but wonder what the problem could possibly be.

Quotes below like this are all from the letter.

Caveat: I’m not saying that I know better, but I do have questions. This is a quick take for now, if anyone is interested I’m glad to elaborate.

We have a limited window to strengthen cyber defenses.

This is both a vague (years, months, weeks?) and ominous warning that I would say deserves at least one verified, non-classified incident before calling for “industry and government to bring the full weight of their technology, resources, and expertise to this effort”.

Is there any evidence that August 2026 frontier models are that much superior for this purpose than they were, say, two months ago? Supposedly little technical expertise is required with the big models so if the feared attacks are possible it should have started I would think. In my view, the signatories could have provided more evidence to back the extraordinary claims so others could understand the motivation for this broad and unprecedented initiative impactin countless systems and software components.

In the coming months, AI-enabled cyber attacks will become far more widespread and sophisticated as models around the world become increasingly capable.

There have been musings that AI will take off in capabilities suddenly for years, but this time we’re sure? Perhaps they will plateau, or perhaps we aren’t leveraging them well yet and they are better than we know. Can anyone say for sure?

Today’s AI advances are already giving defenders new ways to fix weaknesses …

Mythos preview (April 2026) only offered that we should “Think beyond vulnerability finding”, not start fixing. They suggest that “models can also accelerate defensive work in many other ways”, including, issue reporting and triage, write repros and reports, aid review, etc. That is, help humans who are doing the real work. Translation: this will all go at human speed for the time being.

With LLM driven attacks at inference speed, human defenders even greatly aided are not going to be a match if this threat is for real, considering that:

very few software engineers have experience doing this work (and they all have day jobs);
the number of vulnerabilities is vast (and nobody even knows its scale).

Unless the “new ways” are beyond what the April report lists, then if the premise of all the coming attacks is true I’d be very worried.

Recognize that status quo security won’t be enough.

This has been the case in general for two or three decades in my view.

Even if the “historically under-resourced” systems are given AI tools, without expertise (given that fixing is gated by human participation) this will be a flat-footed effort.

I’m not saying this should all be worked out – clearly it can’t be – but my point is we need a more detailed view of the current facts on the ground and realistic assessment of the problem to better envision how this might all work. Without transparency, collective action is far more difficult.

It’s a hard problem, but I think it’s safe to say that it’s way more than a technology problem and that the highly competitive software is not exactly known for collective action without market or legal pressure.

Should “Every organization” independently do all that security work, and given that they haven’t for a long time we need to understand why not.
Haven’t the cybersecurity companies and partners supposed to have doing all this stuff for many years against conventional attacks?
Haven’t governments been trying to coordinate defense and collect incident reports for many years with little impact?
AI companies granting access is great but defenders need experienced people too (and there can’t possibly be enough of them).

Additionally, I would suggest AI companies also consider backing research toward LLMs fixing vulnerabilities (not just finding and helping fix), making pull requests easier for humans to assess for both blocking exploitability as well as lowering the risk of introducing a new bug.

In my view, fixing vulnerabilities may not be that hard).

On top of this, it’s very well known that patching complex systems is risky and there’s no mention of balancing that against a call for urgent action at break neck speed.

Personal opinion: I think is very important to ask for each system, how much security improvement is sufficient? (This is a notoriously hard, yet important question.) It’s great when new tools are applied where the need is great, but we have no idea how much change will be required to bring secure defenses up to snuff – or how we would even know when it’s good enough. Pointing advanced tools at older systems (which I think is a fair assumption for the general case, more so for the under-resourced) could produce an effectively unbounded spew of potentially serious issues. Then what?

One last question for now: why are many of our infrastructure systems “historically under-resourced” in the first place? Software technology aside, securing our utilities, healthcare, and other critical systems is a glaringly obvious priority that it does not take cybersecurity expertise to recognize.

This open letter presents a strategy aimed at the technical challenge, but in the context where infrastructure security has not been a high priority, without first understanding those root causes first that approach is unlikely to be sufficient in my view.

Returning to the letter’s opening line, what exactly does “limited window” mean? To me this is a veiled warning that if you act too late the game is lost. Do they mean an adversary can take down parts of infrastructure at will, or possibly irrevocerable infrastructure destruction or take over? It’s horrendous to consider, but are there no physical overrides with manual operation in the worst case, or do we throw up our hands in the event of a remote software attack?

Threat modeling is how I make vauge threats more concrete, and how I regularly advise anyone every chance I get. In this context that means beginning with each infrastructure system making a basic threat model that would answer questions such as above.

Finally I wonder if there might not be a lot we can do without a massive AI call to action. Maybe we don’t need a huge infusion of new (bleeding edge?) technology that requires experts to oversee and carefully hone patches for safe merges. Just to toss out a few obvious things that in general make systems vulnerable:

Perhaps simply installing backlogged patches makes it much more secure (note that if this is difficult then all the best AI patches in the world are not easily deployed either).
Perhaps an audit finds that a subsystem thought to be air gapped isn’t and it can safely be disconnected or a compatible replacement found that doesn’t require an internet connection
… depending on the system there are many other common vulnerabilities like this that self-review can identify that are safely mitigated

Unless the resourcing and expertise gap is somehow quickly filled, I would urge these system owners to consider these sort of steps (which are all very doable right now and not nearly as much work) now until proven AI assistance becomes available to address remaining weaknesses.

I have no inside access to any of this, using public information, so all this is just one opinion. Nonetheless, I do have all these questions which I believe are quite relevant if not actually important if nothing but to make the underpinnings of the open letter more apparent and easy to see the logic of.

Despite all this, I really do because Gemini critiqued this, and it actually came up with a great summary in closing (to which I would only add transparency): technological acceleration without root-cause analysis and operational readiness is a recipe for churn, not security

In closing I’m going to bury the lede because I was puzzled by one phrase about agentic identities, but I think I figured it out. Here’s what the letter calls on AI companies to do as the major technical response to the coming threat that they foresee:

Build observability and security tools, ensure agentic identities are traceable and accountable, and share best practices in continuous monitoring.

All the way through I was assuming that the direly needed defensive AI response was LLMs working alongside the development team to carry as much of the load as possible, e.g. triage, coding, reviews, testing, and more mentioned above. However, for that work you don’t need to worry about “agentic identities” since software engineers are invoking agents to work along side them, and they can always just ignore sloppy pull requests or other bad input from the LLMs. Then the light went on, it was at once so obvious and deeply unsettling: if you have AI agents in production then “traceable and accountable” are extremely important when something goes wrong.

So I’ll close with still more (to me) unfathomable questions:

Could it be that the risk they are talking about here be unsafe deployment of AI agents in live systems? I dearly hope not: but what else could it mean for defense against AI? Certainly the attackers are not going to cooperate and hand us the identities of their agents!
Can anyone please connect the dots for a different interpretation of how that call for action relates to the goal of the letter in another way?
Is this the imminent “limited window to strengthen cyber defenses”, that the signatories refer to, in part at least, defending against our self-deployed AI agents running amok?

Role confusion: one more reason we can’t trust LLMs

Thu, 30 Jul 2026 00:00:00 +0000

Prompt Injection as Role Confusion is my new favorite paper about a very obvious threat in hindsight that’s hard for us humans to see because we anthropomorphize LLMs so naturally. When Obi-Won Kenobi tells the stormtroopers that “These are not the droids you are looking for” to pass the checkpoint that’s role confusion: the guards foolishly think his words are their own thoughts. The very readable blog-style writeup explains the details, but I want to focus on the threat model perspective which is my bread and butter.

I look at software from a security perspective, and as amazing as the technology is, it seems that the list of reasons that modern LLMs are inherently untrustworthy just gets longer. Without limitation, a long list of challenges (that seem to be quite fundamental and not amendable to add-on remediation) includes: poisoned and errant training data, side effects of RLHF, ineffective guardrails[*], hallucination, speculative completion, lacking metacognition, alignment drift, context variation sensitivity, and now (new to me at least) role confusion.

Modern LLMs interfaces partition chat sessions with markers delimiting sequences of tokens as system prompt, user input, thinking, tool use, and its own responses as assistant; these various sections are associated with roles. As the paper’s conclusion explains, Role tags were a formatting trick that became the security architecture and the cognitive scaffolding of modern LLMs.

The phrase “became the security architecture” raises a big red flag because that sounds like nobody thought much about it. What follows is my simplistic take, but the abstract principles involved are so fundamental that details are not important to the basic argument.

Making sense of these sessions (for humans or LLMs) requires keeping track of the roles. Humans know how to understand conversations and easily follow the role markers (like HTML, <user>2+2</user><assistant>4</assistant>), it’s a completely reasonable scheme for us.

But assuming that LLMs interpret roles that way would be naive anthropomorphization; and just such an assumption appears to be how such a weak security architecture came to be. As the paper explains (section 1): … for an LLM, everything arrives through the same channel as one long token soup. Its own thoughts sit next to your instructions, which sit next to the contents of a random webpage it just fetched.

Designing a security architecture where user commands and data sit intermingled with root access only state and commands is already madness, but it gets worse. Classic software might be able to carefully parse such a token sequence accurately into respective roles, though it’s still a risky design, but LLMs do inference on that “token soup” where no hard boundaries of any kind exist or can be enforced. Once there is role confusion all bets are off, and prompt injection is just one of many sources of abuse or confabulation.

It’s hard to think of a murkier trust boundary design.

[*] The inherent fragility of guardrails is worth expanding on because it’s not just an unsolved problem: infallible guardrails are mathematically impossible. NIST has published a proof (behind a firewall; seems wrong for a government entity) that guardrails will inevitably have holes. Based on the summary openly available, “a fixed set of guardrails placed on AI is not universally robust against adaptive adversarial prompts”. The work is especially cool because it builds on the technique Kurt Gödel used to prove his incompleteness theorems. According to the NIST authors, “The findings show that developers and organizations deploying AI systems need to dedicate resources to finding prompts that would break the security of AI systems, and to address them before adversaries can exploit them.”

AI agent parody?

Wed, 29 Jul 2026 00:00:00 +0000

This HuggingFace security incident disclosure has people talking about AI agent security. Today I saw such absolute positive spin that I found myself thinking “this must be a parody”: looking at the context I’m pretty sure that it isn’t … though some parodies stay in character all the way through.

Just one opinion here, and others are quite free to hold different opinions and I won’t identify the source because nobody knows exactly where technology will go next. The post I saw is an excellent foil to serve as a prompt to explain my take at this point in the “AI” ride. I won’t attempt a long article on the perils of AI agents for now, but each point below could be expanded. I’m not intentionally taking anything out of context for this quick take. I’ll be brief and suggest a few edits as bracketed commentary to exact quotes following.

very real [extremely dangerous] implications for the diffusion of AI in the enterprise
need to harden systems and environments [far more than a company with world class expertise can today] to prepare for agents [before using them in production for any purpose not rock solid safely contained; agents are probably better than humans at escalating privileges so a large margin for area is necessary for safety]
There are tons of measures [we know that “guardrails” are never enough] that enterprises must [not may or should] take [what “tons” exactly? what fraction of them are fully mature and demonstrated to be reliable today?]
happy path case [how close are we to achieving that are reliable across a wide range of applications today?] with a good actor, if you give an agent a task [which you always would]
In theory, you had [present tense “have”; humans have not yet been totally replaced] to worry about this for people as well, but with AI [anthropomorphization, ignoring that agents pose new categories of risk incomparable to humans] the risks are inherently amplified [why choose amplified risks?].
[AI agents] don’t have the inherent judgment (yet) [on what basis would anyone think this likely enough to be talking about in 2026?] that a person does [big time anthropomorphization here and the comparison is an enormous stretch]
you need an all new way [agreed: agents already penetrate existing security measures so our current arsenal won’t do it (and I don’t think we can count on AI to figure this out)] of managing data and protecting environments [how many enterprises have the time and resources for this when many struggle just to patch and do updates?]
This obviously creates a huge opportunity for the startup and security ecosystems [yes, software customers need to pay for that — why would they spend even more, for what?]

Finally, I agree with the last sentences in spirit but would make a few adjustments:

It should update everyone’s timelines [which without any clear plan is indeterminate if it’s even possible] on diffusion, though, as it also means that enterprises will have another step of work [that’s a very very long step that I fear not even an astronaut could make] they have to go through for agents to handle more [more? today it’s unsafe] autonomous work.

By the end I think I figured out the one unspoken assumption that flips this around to make perfect sense: AI agents are inevitable. If true, that would explain why we must take on new risks of unknown magnitude and cost to overhaul our systems and security regimes. Again: parody?

More on OpenAI/HuggingFace

Mon, 27 Jul 2026 00:00:00 +0000

The recent AI agent security debacle must be reverberating quite a lot within the walls of the major proponents of AI agentic technology because they announced a brand new “movement” apparently with zero details available yet. If that isn’t a sign of flat-footedness then I don’t know what is.

Did OpenAI hack Hugging Face or didn’t they? and post raises the question of law breaking here which they suggest may be a case of “responsibility laundering” by blurring intention to do harm where the LLM in between OpenAI and the harmful actions done is to blame — and being an intangible, impossible to take to court, as well as opaque piece of software running model of a scale that defies analysis, the buck uselessly stops there. In my personal opinion (IANAL) that such damage occurred shows clearly this was due to insufficient caution wielding a powerful tool granted (intentionally or not) powerful privileges. Had an unintentional bug in complex classic software caused the same harm would/should the reaction be any different?

As Adam Shostack writes, “The team at OpenAI either can’t or won’t slow down to look at the output of these systems. Now, maybe, that’s the right call?” It looks like if you can avoid responsibility for collateral damage then it’s the right call (same as all the “move fast and break things” crowd does) …

My two cents: Sandboxing, perhaps doubled up, would be a good practice going forward - compared to all the model inference the software overhead must be miniscule.

Normalizing cybersecurity facepalms

Wed, 22 Jul 2026 00:00:00 +0000

OpenAI writes: “Last week, Hugging Face disclosed a new kind of security incident⁠(opens in a new window) after they detected and contained an AI agent that compromised their infrastructure, something we expect to become more commonplace with the proliferation of increasingly cyber-capable models. After investigating, we now know that this particular incident was driven by a combination of OpenAI models — including GPT‑5.6 Sol and an even more capable pre-release model, all with reduced cyber refusals for evaluation purposes — while being internally tested on a benchmark⁠(opens in a new window) of cyber capabilities.”

I’m appalled, there’s so much to be said just about this, there are so many hidden details to know what happened, so will be brief since the points should be crystal clear: It’s now OK for their customers’ AI agents to attack your own systems in that we aren’t rethinking if we can really trust them is premature?

If their own cybersecurity practice is flawed how can they train models adept in it, including effective cyber refusal guardrails?

Specifically, the walk back how “safeguards were intentionally not enabled during this evaluation” … why was this necessary in the first place, how can we trust that this time it will be enough? If this happened then it wasn’t running in a “highly isolated” environment. By definition. So many security practices apparently not followed and not mentioned in actions taken: security review, threat modeling, auditing, more sandbox testing, to name a few.

Naming the model and forthcoming greater model seems like grabbing marketing hype as Mythos did; with an insecure sandbox this is not a demonstration of cyberprowess.

This appears to be over-provisioning of privileges: it was in a sandbox, but the sandbox process shouldn’t have such potentially destructive privileges in the first place for a test.

There’s no alternative to full steam ahead so these incidents will become more commonplace? Also last week, OpenAI disclosed that another model mistakenly deletes files which they called an honest mistake. More normalization, more of what we can expect in the AI agentic world.

Toward Better AI Legislation

Sat, 11 Jul 2026 00:00:00 +0000

“People deserve to know whether or not the videos, photos, and content they see and read online is real or original,” said Senator Schatz. “Our bill is simple – if any digital content is made by artificial intelligence, it should be labeled so that people are aware and aren’t fooled or scammed.”

Senator Schatz is doing a great job and this is clearly a sincere effort for an important issue. However, ‘‘AI Labeling Act of 2026’’ is just one example of how poorly the government is equipped to regulate genAI. We must carefully analyze legislation to anticipate serious side effects potentially making things worse.

Will this work well enough? Probably at least for a while, sure. But I’m arguing (a) we should be keenly aware of limitations this fundamental; (b) acknowledging its weaknesses, evaluate how it works and reevaluate it regularly; (c) instead of deferring to administrative regulations left undefined the law should set up a committee representing diverse perspectives to outline regulations and work with government staff to help craft the details; (d) expect that bad actors and crafty lawyers will quickly notice all the loopholes and gray areas to exploit aggressively in order to circumvent responsibility; (e) consider better approaches rather than stick to such a flawed initial conception.

In closing I will offer some different ideas that I think are at least steps in the right direction.

Briefly, the major problems include:

The definition of “AI” (15 U.S.C. § 9401(3)) is overly broad (see below) encompassing all kinds of software we wouldn’t consider “AI”. Saying “I know it when I see it” is fine for subjective judgments but applied to categories of software (100% digital) it’s absurd as a legal criterion.
Model inference is just a (very big) table driven algorithm implemented in code like any other component. All software models information (the concept behind object oriented programming) and produces output that is inferred from those models (often simple but it can be as complex as we make it, however complexity is not a condition for the law anyway).
Ordinary users do not understand genAI at all and have no way of knowing if tools they are using depend on genAI internally or not. (Labeling of software isn’t included. DMCA forbids reverse analysis to find out if the facts are not accurately disclosed by the maker)
There are no exceptions for minor uses that are insignificant (e.g. minor photo touchup).
Reposting content a user believes to be genuine but isn’t would be an unfair violation.
Expecting civil servants to craft regulations based on a weak definition of “AI” and given the complexities of software, unless they have considerable genAI expertise is almost certain to fail. Virtually all genAI experts are in industry or academia and both already biased and also unlikely to commit to advising the government.
Content providers need clear guidelines in order to comply with the law, but have no industry consensus definition of “AI”, are generally unaware of the regulatory definition mentioned (with many faults), and implementing identification and protection at scale is infeasible.
Predictably the $1T scale platforms have technical and legal resources to effectively skirt the law or respond effectively if charged with violation, but smaller competitors will be exposed to arbitrary enforcement risk.

Regarding the definition of “AI” that this legislation is based on (15 U.S.C. § 9401(3)), it says:

The term “artificial intelligence” means a machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations or decisions influencing real or virtual environments. Artificial intelligence systems use machine and human-based inputs to-
(A) perceive real and virtual environments;
(B) abstract such perceptions into models through analysis in an automated manner; and
(C) use model inference to formulate options for information or action.

There being no carve outs for scale, complexity, parameter size, compute complexity, this would apply to a broad set of software categories that must not be the intention and would be totally infeasible to apply much less enforce. Also over-labeling dulls the value of the label itself (consider California’s Proposition 65 toxic exposure right-to-know law).

An easy example of this being overly broad (it applies to all machine learning, of any scale) is web algorithms to maximize engagement: human inputs (posts and choices of content); virtual environment (social graph, advertising demand, other web content, etc); build customized models of what each user engages best with; infers what content to push next to extend platform interaction.
Digital games with 3D world models (that use similar NVIDIA hardware) also have human input, a virtual environment, character behavior as well as physics models, and infer game action or scene visuals.
Small models fit, too; consider grammar checkers: human inputs (text entry); virtual environment (document context); correct grammar model for inference; action options (suggested fixes). Do we seriously want to require labeling documents written with copy editing help from genAI?

It’s easy to find flaws, but here are some alternative approaches I think deserve consideration:

Instead of criminalizing mandatory labeling it would be much more effective to hold hosters responsible for enforcement. (This may be politically infeasible due to lobbying by the $1T scale platforms.
We should punish deceptive behavior, not the technology used. Precedents like gun laws and fraud penalize behavior not technology usage: there is a track record and that approach works quite well.

By no means do I suggest having The Answer but I think there’s a lot to worry about, it calls into question how much expertise is applied in crafting these laws, and we need a clear reckoning of the limitations and risks of passing laws — especially about leading edge technologies.

Paywalls fall thanks to AI Overview (Google search)

Fri, 26 Jun 2026 00:00:00 +0000

The NY Times teases a paywalled article, “A.I. won’t take all our jobs because it can’t reason like a human, Zeynep Tufekci writes.” linking to the article behind a paywall. Simply searching for [it can’t reason like a human, Zeynep Tufekci] provides a nice summary of the article, not only penetrating the paywall but also saving time and skipping the ads.

Could this be one use case that many opponents of “AI” can get behind?

By the way the article is disappointing to the point of self-contradiction. Tufekci assumes that management hiring and firing decisions, including “hiring” AI for jobs, include being aware of and caring about (the claimed without basis) innate human advantages. It goes on to point out more “immediate threats” leading to self contradiction. [1] Erosion of Trust: management is vulnerable to just this deception and likely to make poor decisions replacing humans with AI. [2] Management failures of ethical and moral judgment will result in the very loss of jobs argued against not being a major threat.

(Disclosure: I’m relying on the AI summary since the article is behind a paywall.)

Here’s the Google AI mode experience

Software security with Large Language Models

Wed, 29 Apr 2026 00:00:00 +0000

“AI” on my view is already and will certainly be a massive disruption to software in the coming years. Furthermore, we have an unprecedented wave coming that’s only just now beginning to break. Yet the biggest unknown, as I see it, is how the software community will respond, and that will be more due to social factors than purely technical. This is very much as it should be, however our very human frailties and limitations will inevitably drive how this unfolds.

Thus, I believe that navigating the rough waters ahead successfully requires first awareness of these powerful influences clouding our judgment and the decisions we make starting now.

LLMs dominate just about every discussion of software security these days, and lately I am asked to opine on this increasingly often. Frankly let me emphasize that all I can offer is my opinion, based on my very limited knowledge and experience with this new technology. However, My thinking is based on first principles which I do not think LLMs will change.

At this point (mid-2026) in my opinion things are so volatile that it’s premature for anyone to know very clearly what’s ahead. For example, a few weeks ago the Anthropic Mythos announcement (whether the claims hold up or not) dramatically changed the conversation about both offensive and defensive cybersecurity.

Given this context, I can only offer these high level points (all opinion, I could be wrong of course):

Going back to first principles is the best way I know to escape all the hype (both pro and con).
Threat modeling is essential (using LLMs what could possibly go wrong?), while realizing that LLMs make modeling far more challenging (being unpredictable and challenging to keep within strict guardrails).
Unless the LLM maker takes responsibility (never heard of this), before deployment get consensus for who does when something goes wrong.
Roll out new LLM-based projects carefully and monitor how well they do the job.
Stay flexible because things will change fast.

Transparently sharing results (both success and failures) is the best way for the software community to learn how to best integrate this remarkable technology, as well as better understand what can go wrong and how to mitigate the downsides.

Transparent AI use

Sat, 31 Jan 2026 00:00:00 +0000

How much AI use is acceptable for writing? It’s a hard question because it depends greatly on context, the reader’s expectations, and the fact that it’s difficult to usefully measure “how much”. How we address this matters for several reasons, including but not limited to: creator’s responsibility and originality, honest disclosure about research effort and sources, respecting the broad spectrum of opinion about ethical use of AI.

Consider this recent example. After writing feedback on a document I felt I should come clean that I used AI. Specifically I had Gemini “read the PDF for me” and then grilled it to find parts of particular interest. Additionally, I asked it to critique a draft I wrote (all by myself), edited the draft (all by myself) for some of the suggestions. I thought this was well within bounds, but enough use that concealment would be marginally unethical (though soon norms may change), and saying that “No AI was used” would be outright lying. On the other hand, admitting that “AI was used in creating this document” would suggest much heavier use.

How to describe this middle ground that I consider reasonable, not cheating, but also not zero either? Here I will just state that it’s very hard to think of a useful metric or even subjective scale, and of course each of us will have their own sense of the context, expectations, and norms.

Pondering how to approach transparency in AI usage I thought of Publish your threat models! (coauthored with Adam Shostack) where we posit that the benefits far outweigh the dangers. Closer inspection reveals it’s a very good fit:

security is also notorious difficult to measure, or even define objectively
the paper includes a section on how “Context matters”
existing precedents are surveyed
explains the benefits to the creator, consumer, and others
offers guidance on preparing to publish, including redaction if needed

To be clear, no parallels between AI use and threat model are intended: only the similarity in how transparency is an effective tool for substantiating subjective aspects of a work product.

How these topics apply is very different for AI usage, but initially the approach I will suggest is also far simpler:

share your AI session
write a concise summary of how you used AI (e.g. “for research and draft review”)
ideally (if this idea should ever catch on) mark this with a standard presentation (e.g. with a logo signifying “AI usage declaration” conventionally at the beginning of the document)

This allows honest creators to disclose what they did; most people will just see the summary; anyone who cares is free to look at the full details and call it out should they see a discrepancy.

When you use AI for your work - for research, for editing/review, and more - transparency is best practice (IMHO). Rarely the AI session may need minor redaction, but if you are working on a proprietary you probably wouldn’t publish the work anyway.

FAQ

LLM writing aids(*) such as spelling and grammar suggestions can be exempted (IMHO), or meticulous creators can mention either the use or strict avoidance.
LLM editing (I can only suggest drawing the line conservatively) should be disclosed.
Clearly dishonest people can easily cheat, but accidentally misinforming is hard to imagine.
Precedents do exist (this is hardly a new idea, though proposing this should be standard practice may be), for example: Notre Dame AI Transparency (and surely many more).

(*) FAQ#1 is hard to delineate and difficult to detail (listing all the typo corrections would often be laborious); furthermore, even determining whether spelling corrections are AI or dictionary based (or even drawing a meaningful line between the two) is itself challenging. Arguable minor AI use can and should be exempted in my view for clarity about use but others may have ideas for better navigating this tricky issue. For example, an “emission-free vehicle” claim reasonably excludes any share of emissions incurred by the company that designed and manufactured it (necessary for the vehicle’s existence), transporting the vehicle to the customer (shipping and trucking), and so on (including, as a matter of opinion, tire wear particulates in the atmosphere and other pollutants even in small but measurable amounts).

This approach is easy to do, requires no tools or special skills, and is far less involved and technical than other proposals to address the issue of AI usage disclosure. There is no attempt to establish any scale of degrees of usage, nor draw artificial lines along such scales. The declared usage statement is subjective, but backed by full transparency as evidence. Reasonable people may well differ in interpreting actual usage, doing so looking at the same facts. Outright misleading summaries can be discovered and called out). This method enables creators to proactively disclose AI usage in meaningful terms. If we start regularly practicing disclosure now then as AI collaboration becomes more powerful and commonplace then future tools automate including such metadata.

The question of AI use by creators (defined broadly) is important for many reasons, yet there is no widely recognized way to proactively answer it. This article proposes a simple, effective, low effort method as a starting point to fill that need. Anyone who agrees with this idea can easily start using it on their own — if you think this idea is helpful, start using it now.

AI usage writing this article: review of draft versions with follow-up discussion of criticisms