That Fake AI Generated Photo and the Stock Market

Why it is an example of the system working and how it could be better

Here’s what happened. A series of Twitter “verified” or blue check accounts posted AI-generated images of an explosion at the Pentagon on Monday morning. Here is an example.

This caused a dip in the S&P500 of 80 points (0.15%) in the first 4-5 minutes after the posting. Six minutes later, the market had recovered fully.

The incident was short. For instance, relevant emergency service authorities were on it.

Is this a worrying development? Yes and no. The stock market has particularly sensitive to information and has a history of reacting before verifying. In this case, the disruption was short-lived.

That said, the usual cast of characters lept in with “See!”

See what exactly? One reaction is that this has little to do with AI and could have been done in other ways. True, but I will take for the moment that AI is supposed to make this cheaper, easier and more convincing.

But did it?

No, it did not. It was a very poor example of misinformation. It was easily detectable. Just the sort of thing AI-assisted misinformation was not supposed to be.

Could it be convincing? I suspect so, but if you really wanted to have a persistent misinformation attack, it looks like there are real benefits to spending lots more time working on it. AI could get to a stage where it can do this in a hard-to-detect manner, but before we get there, something else will push back.

But Monday’s event was the system working fairly quickly to identify misinformation and communicate it. Thus, the disruption was contained. That is, broadly speaking, how the system should work. It isn’t the ideal reaction, but it is in the right direction and suggests the world is not yet broken.

The fear of those like Marcus is that there will be so much AI misinformation, high quality or not, that, rather than reacting to it as the stock market did, no one will react to any information — misinformation or not.

But that does not seem to be the conclusion to draw from this event. Instead, the conclusion to draw is that no one might react to information immediately and only do so once it is verified by a trusted “to them” source. To be sure, there is a cost to this delay, and that could prove to be an issue. But it won’t be a cost if the information comes from trusted sources in the first place.

Herein lies the problem we face at this moment. Twitter took two steps back when it got rid of its most valuable product — artisanal verification of certain accounts. It looks like a very costly experiment in trying to find a monetisation route without ads; the Twitter Blue subscription model has sullied the signal. Yes, trolling accounts have to spend some money to get a blue check which means that if only blue checks are a signal, that signal is costly. But kicking off or not providing a distinguishing signal for real, verified accounts is a big problem. Put simply, if Twitter’s main selling point is immediacy then basically it has exited that business. The stock market hasn’t quite got the message.

My view is that there is room in the market to fill Twitter’s gap. Hopefully, all of this will facilitate someone moving forward on the verification of trusted sources.