Did exposure apps work?
Welcome to Plugging the Gap (my email newsletter about Covid-19 and its economics). In case you don’t know me, I’m an economist and professor at the University of Toronto. I have written lots of books including, most recently, on Covid-19. You can follow me on Twitter (@joshgans) or subscribe to this email newsletter here. (I am also part of the CDL Rapid Screening Consortium. The views expressed here are my own and should not be taken as representing organisations I work for.)
The other day, when giving a talk for the International Telecommunications Society, I was asked about exposure notification apps and whether they worked. I answered that I thought that they had a ton of potential but it is very hard to develop these things in the midst of a crisis and so that they had not really helped. One problem is that there was a lack of take-up. Another was that many people claimed to receive exposure notifications that caused them stress while waiting for a test where it was likely that they had not really been exposed at all but had just been proximate to a case in a building. In other words, there were false positives. My belief is that this was primarily a result of not tweaking the app as we learned more regarding what led to exposure — notably, indoors versus outdoors — and that we could do better in the future. Let’s face it, if you are trying to predict whether you have been exposed, the fact that we carry around devices that can talk to one another in a privacy-respecting way, and log contacts is a great opportunity for a good predictive algorithm.
But I hadn’t really seen any evidence on this. A new paper appeared in Nature this week that provided some evidence looking at the UK’s NHS app.
Here we investigated the impact of the NHS COVID-19 app for England and Wales, from its launch on 24 September 2020 through to the end of December 2020. It was used regularly by approximately 16.5 million users (28% of the total population), and sent approximately 1.7 million exposure notifications: 4.4 per index case consenting to contact tracing. We estimated that the fraction of app-notified individuals subsequently showing symptoms and testing positive (the secondary attack rate, SAR) was 6.0%, comparable to the SAR for manually traced close contacts.
The last part is the important bit. You can actually see it in (d) in the following figure:
In other words, if you were exposed to a positive person and notified of it, the probability that you tested positive afterwards was 6%. The question is: is that a high or low number?
It is hard to say. Let’s go to the fine print. First of all, the data they used was from iPhone users as they decided Android was less reliable. I wasn’t clear about how adjustments were done for that fact given that you might be exposing any type of user. The way they estimated the secondary attack rate (or SAR) was to observe the probability that an individual notified on a given day tests positive t days later and combines this with the actual number of people notified on that day and correlated this with the number of people reporting a positive test on a given day given that they had received a recent notification. This allows them to estimate the SAR. (The details are not in the paper but in a supplement).
The problem, of course, is that there is a distinction between regular/known contacts and unknown ones. The app’s marginal benefit is concentrated in the latter set as normal contact tracing would take care of the others. But if, say, app installs were more clustered in friendship groups, that means that a disproportionate set of notifications that subsequently test positive will be for regular contacts. What I could not see in the study was any account for this — for instance, conditioning on how many contacts you had with the same person. Of course, I don’t believe that data is available for privacy reasons. But it really matters when thinking about how useful the apps are. And it matters given that the authors took their estimate at face value to calculate what the overall benefits of the app might have been:
We estimated the number of cases averted by the app using two complementary approaches. Modelling based on the notifications and SAR gave 284,000 (108,000-450,000), and statistical comparison of matched neighbouring local authorities gave 594,000 (317,000-914,000). Roughly one case was averted for each case consenting to notification of their contacts. We estimated that for every percentage point increase in app users, the number of cases can be reduced by 0.8% (modelling) or 2.3% (statistical analysis).
My guess is that this overstates the benefits relate to other ways of finding contacts. It is still not nothing but right now it is not clear it is a primary instrument in pandemic management.
But it could get better. For instance, an app that allowed users to specify work and home would allow notifications to be tailored and also for exposure notifications to indicate how ‘close’ the exposure really was to you. Also, it is unclear how long exposure notifications take place following a positive test. This can be tweaked as well. Finally, being able to overlay a map of indoor versus outdoor places would be invaluable. In other words, there is more publicly available information that can be brought to bear.
Exposure tracking has really high potential benefits for pandemic management. It requires lots of app take up which is something we already understood. But it also requires flexibility and thought. It seems to me that in most cases apps have been done and dusted without any evolution. This is something that we can put in the ever-increasing basket of things we can do better with next time.