Better security discussions

(900 words) May 2024 – Loren Kohnfelder

We understand software security best through specific threats and mitigations, articulated by threat models shared openly. Without this context we avoid much needed meaningful security discussions.

Software security articles and opinions fill the tech news and increasingly are covered in mainstream media yet consensus is rare, so often it seems people are talking past each other. This month (May 2024) Microsoft announced "company-wide" commitment "prioritizing security above all else" and then a few weeks later introduced Copilot+ PCs with built-in AI hardware, prominently featuring Recall to "search across time to find the content you need" with the power of AI. Numerous observers immediately reacted that Recall was a privacy nightmare. Rather than weigh in with yet another opinion I'd like to look closely at why the discussion is so contradictory and confusing, and consider how we might begin to find some kind of common ground.

Here's my thesis, as the following discussion should explain fully: We understand software security best through specific threats and mitigations, articulated by threat models shared openly. Without this context we avoid much needed meaningful security discussions.

Years ago I worked at Microsoft and know they had plenty of competent people working diligently on their products and have no reason to believe that has changed. I also want to be very clear that I do not own a Copilot+ PC, so I have never used Recall myself and I certainly don't know the technical details; instead I want to write about a hypothetical product much like it in order to make a larger point and avoid worrying about getting any of the details wrong.

The gist of Recall (or our similar theoretical application for discussion purposes) is that it takes screenshots in the background and then indexes those based on AI powered summary of text on the screen as well as image semantics (recognizing cats or whatever visuals might be captured) so you can find anything later via AI-based query capability.

Microsoft assures us the system is secure because:

  • You can turn it off.

  • There are settings to disable it for specific apps and websites.

  • It won't capture private browsing data for certain browsers.

  • You can delete some or all of the captured data anytime.

  • All analysis and data is stored at the edge (in your PC).

  • Data at rest is encrypted (BitLocker).

  • Only available on PCs with Microsoft Pluton security processor.

Here is a skeleton threat model that seems to be what Microsoft folks are thinking based on the above talking points, expressed as "risk — mitigation" (which I think is a very handy way to quickly express a basic threat model that's easy to understand with special technical expertise).

  • Adds attack surface, upping risk — Setting to disable the feature.

  • Malware can easily snoop on screenshots — Use anti-virus etc. to keep your PC safe.

  • Certain activity should be excluded — Settings to selectively disable (apps and websites).

  • Inadvertent capture of sensitive data — Users can delete things later.

  • Microsoft can spy on me — All processing and storage is at the edge.

  • Screenshots can leak if PC is stolen — Encrypted data at rest, plus Pluton security.

OK, that makes sense, so why are so many people so upset?

And here's the crux of my focus using Recall just as a convenient example of a very general principle stated above — understanding software security as threats and mitigations.

I haven't seen a cogent technical analysis of Recall but there is plenty of negative reaction out there, and my point here is more about explaining the discrepancy between Microsoft's position and all those critics. Here are some additional risks that I believe people are sensing (whether they articulate them explicitly or not) that appear to be unaddressed — which explains the divergence.

  • When malware gets into PCs, this makes it easy for attackers to get juicy secrets

  • Shared data from others will be inadvertently captured without their permission

  • People share PCs with others they trust, which may lead to accidental unintended disclosures if not abuse

  • Many people won't configure all sensitive apps and websites that should be excluded

  • Evil Housecleaner attacks are aided by AI, e.g. "find bank account details"

  • If disabled, the Evil Housecleaner enables it (rarely noticed) and returns later

  • Microsoft bugs may inadvertently expose the screenshot data

  • People don't trust Microsoft: future versions my loosen the security mitigations

(The above list of potential risks isn't verified on Recall or meant to be an exhaustive list.)

Whether or not these are valid concerns, the point here is that only by enumerating risks and mitigations can we have a meaningful discussion about any security topic. Absent specific risks and mitigations, with both sides assuming things and worrying about different risks, we will invariably be arguing at loggerheads. This is commonplace for software security discussions today but it just doesn't make any sense as I hope this example makes clear, it gets us nowhere.

Perhaps Microsoft has already anticipated and mitigated all the additional risks listed above — if so, they need to show their work for us to understand. Alternatively, if customers raise new risks then we can have that discussion, perhaps to deny the concern (for example, that using Windows implicitly requires trusting Microsoft). Only when we see the risks laid out in black and white can this kind of meaningful discussion go forward: by comparing your list of risks to mine we immediately see where our perspectives differ.

We understand software security best through specific threats and mitigations, articulated by threat models shared openly. Without this context we avoid much needed meaningful security discussions.

More on how to further the security discussion here.

security