Better security discussions

#security

(900 words) May 2024 – Loren Kohnfelder

We understand software security best through specific threats and mitigations, articulated by threat models shared openly. Without this context we avoid much needed meaningful security discussions.

Software security articles and opinions fill the tech news and increasingly are covered in mainstream media yet consensus is rare, so often it seems people are talking past each other. This month (May 2024) Microsoft announced "company-wide" commitment "prioritizing security above all else" and then a few weeks later introduced Copilot+ PCs with built-in AI hardware, prominently featuring Recall to "search across time to find the content you need" with the power of AI. Numerous observers immediately reacted that Recall was a privacy nightmare. Rather than weigh in with yet another opinion I'd like to look closely at why the discussion is so contradictory and confusing, and consider how we might begin to find some kind of common ground.

Here's my thesis, as the following discussion should explain fully: We understand software security best through specific threats and mitigations, articulated by threat models shared openly. Without this context we avoid much needed meaningful security discussions.

Years ago I worked at Microsoft and know they had plenty of competent people working diligently on their products and have no reason to believe that has changed. I also want to be very clear that I do not own a Copilot+ PC, so I have never used Recall myself and I certainly don't know the technical details; instead I want to write about a hypothetical product much like it in order to make a larger point and avoid worrying about getting any of the details wrong.

The gist of Recall (or our similar theoretical application for discussion purposes) is that it takes screenshots in the background and then indexes those based on AI powered summary of text on the screen as well as image semantics (recognizing cats or whatever visuals might be captured) so you can find anything later via AI-based query capability.

Microsoft assures us the system is secure because:

Here is a skeleton threat model that seems to be what Microsoft folks are thinking based on the above talking points, expressed as "risk — mitigation" (which I think is a very handy way to quickly express a basic threat model that's easy to understand with special technical expertise).

OK, that makes sense, so why are so many people so upset?

And here's the crux of my focus using Recall just as a convenient example of a very general principle stated above — understanding software security as threats and mitigations.

I haven't seen a cogent technical analysis of Recall but there is plenty of negative reaction out there, and my point here is more about explaining the discrepancy between Microsoft's position and all those critics. Here are some additional risks that I believe people are sensing (whether they articulate them explicitly or not) that appear to be unaddressed — which explains the divergence.

(The above list of potential risks isn't verified on Recall or meant to be an exhaustive list.)

Whether or not these are valid concerns, the point here is that only by enumerating risks and mitigations can we have a meaningful discussion about any security topic. Absent specific risks and mitigations, with both sides assuming things and worrying about different risks, we will invariably be arguing at loggerheads. This is commonplace for software security discussions today but it just doesn't make any sense as I hope this example makes clear, it gets us nowhere.

Perhaps Microsoft has already anticipated and mitigated all the additional risks listed above — if so, they need to show their work for us to understand. Alternatively, if customers raise new risks then we can have that discussion, perhaps to deny the concern (for example, that using Windows implicitly requires trusting Microsoft). Only when we see the risks laid out in black and white can this kind of meaningful discussion go forward: by comparing your list of risks to mine we immediately see where our perspectives differ.

We understand software security best through specific threats and mitigations, articulated by threat models shared openly. Without this context we avoid much needed meaningful security discussions.


More on how to further the security discussion here.