Why the best isn't best
Many think that the more precise a test is the better it is. That isn't the case, especially when it comes for testing for coronavirus infectiousness.
Welcome to Plugging the Gap (my email newsletter about Covid-19 and its economics). In case you don’t know me, I’m an economist and professor at the University of Toronto. I have written lots of books including, most recently, on Covid-19. You can follow me on twitter (@joshgans) or subscribe to this email newsletter here.
I have a new paper out on Testing Sensitivity for SARS-CoV-2. It is formal but the issue is one that many will want to have explained more broadly. In today’s newsletter, I try a couple of other ways of explaining why Antigen Tests that are cheaper and commonly regarded as “worse” than PCR Tests are likely to be far superior, on their own terms, as a test for infectiousness and hence, as a test to be used in a comprehensive mitigation strategy for Covid-19.
We must have the best!
Let’s start here:
“PCR Tests are the best and we must have the best!” That is what many medical professionals and test regulators argue. Think I’m kidding? Here is a common statement.
The “gold standard” refers to the highest quality, or benchmark, of a specific practice, product or technology.… Other diagnostic testing methods, like culture or serology, may not provide the same level of sensitivity as PCR. Therefore, the risk of false negatives increases in critical testing scenarios where organisms or viruses may be difficult to grow, or detect an immune response to. This is why PCR is considered the gold standard by many across the diagnostic community.
It’s the coronavirus test Harrods would sell. And scientists use the term over and over again.
The PCR Test is a marvel of engineering. It can be used to find RNA and DNA remnants in the smallest of quantities from small samples. It can be used to find them in a group of samples from different people. It can be used to find them in sewage. It is high fidelity. It is high resolution. It can find needles in large haystacks. It’s simply the best.
The best can be bad
Let’s move away from tests for the moment and think of other ways we try and detect things. One way we can sniff out harmful stuff is to use sniffer dogs.
Just as astonishing, to Waggoner, is a dog’s acuity—the way it can isolate and identify compounds within a scent, like the spices in a soup. Drug smugglers often try to mask the smell of their shipments by packaging them with coffee beans, air fresheners, or sheets of fabric softener. To see if this can fool a dog, Waggoner has flooded his laboratory with different scents, then added minute quantities of heroin or cocaine to the mix. In one case, “the whole damn lab smelled like a Starbucks,” he told me, but the dogs had no trouble homing in on the drug. “They’re just incredible at finding the needle in the haystack.”
Dogs can be trained to seek out any chemical substance from stuff to make bombs, to bed bugs and to, this won’t surprise you, coronavirus carriers. But the challenge is that the chemicals that are specific to a particular thing might be elsewhere.
Horacio Maldonado, one of the new recruits, positioned himself under an arched entrance on the west side of the station. His black Lab, Ray, could smell most of the passersby from there—she had a range of about thirty feet—but a crowd like this was full of false leads. The chemicals found in explosives can also be found in drugs, cosmetics, fertilizer, construction supplies, and other mundanities. I’d heard of a police dog driven wild by a table patched with plastic wood filler, and a dog tearing down a wall with nail-gun cartridges hidden inside it. “I remember one time, we stopped a guy in Columbus Circle, he had two hundred nitrogen pills in his pocket,” Maldonado told me. “Turned out he was going to Europe and had just come from his doctor. So you’ve got to use your common sense. The guy’s sixty, seventy years old. He isn’t sweating. Does he look like a suicide bomber?”
As the dog cannot distinguish the chemical from its source nor really the amount of the chemical, there are false positives. This could be disruptive. If someone was moving an explosive around an airport in a bag, you would have to be concerned that a dog might end up targeting the wrong bags.
Being too sensitive is something we are familiar with. Do you want a fire alarm that is set to go off at the slightest hint of smoke? If not, do you need a fire alarm that is so good that it can detect the slightest hint of smoke?
Herein lies the issue with PCR tests. They can be too good. To be sure, if the coronavirus is present in your body, they pick it up. But what they cannot tell is whether the coronavirus is active or dead. When it is active, you care. When it is dead, not so much. A dead virus doesn’t need an anti-viral treatment (that might have side effects). A dead virus is not contagious so you don’t need to isolate the person.
Turning it Down
While one goal of testing is a diagnosis for treatment, there are other goals. The most notable is to inform a decision to isolate someone. Another is to inform a decision to investigate who someone has had contacts with. Both of these actions, if undertaken needlessly, are costly.
But even in diagnosing for treatment, the PCR test can be a bit much. From the New York Times:
One solution would be to adjust the cycle threshold used to decide that a patient is infected. Most tests set the limit at 40, a few at 37. This means that you are positive for the coronavirus if the test process required up to 40 cycles, or 37, to detect the virus.
Tests with thresholds so high may detect not just live virus but also genetic fragments, leftovers from infection that pose no particular risk — akin to finding a hair in a room long after a person has left, Dr. Mina said.
Any test with a cycle threshold above 35 is too sensitive, agreed Juliet Morrison, a virologist at the University of California, Riverside. “I’m shocked that people would think that 40 could represent a positive,” she said.
A more reasonable cutoff would be 30 to 35, she added. Dr. Mina said he would set the figure at 30, or even less. Those changes would mean the amount of genetic material in a patient’s sample would have to be 100-fold to 1,000-fold that of the current standard for the test to return a positive result.
In other words, even for doctors thinking about treatment, the PCR test is detecting smoke where there is no fire.
So who is setting the dial on PCR test thresholds?
The Food and Drug Administration said in an emailed statement that it does not specify the cycle threshold ranges used to determine who is positive, and that “commercial manufacturers and laboratories set their own.”
The Centers for Disease Control and Prevention said it is examining the use of cycle threshold measures “for policy decisions.” The agency said it would need to collaborate with the F.D.A. and with device manufacturers to ensure the measures “can be used properly and with assurance that we know what they mean.”
The C.D.C.’s own calculations suggest that it is extremely difficult to detect any live virus in a sample above a threshold of 33 cycles. Officials at some state labs said the C.D.C. had not asked them to note threshold values or to share them with contact-tracing organizations.
For example, North Carolina’s state lab uses the Thermo Fisher coronavirus test, which automatically classifies results based on a cutoff of 37 cycles. A spokeswoman for the lab said testers did not have access to the precise numbers.
Apparently, it is not clear. That’s a problem given how much it can matter.
Why is a threshold set at all? When you perform a PCR test, you could actually report the Ct number and then a doctor could judge what that means. Instead, these are hard-wired into machines and so even if you knew what the threshold for that machine/lab was for a test to be negative, if you have a positive result, the associated number could be anything above that.
This amounts to an enormous missed opportunity to learn more about the disease, some experts said.
“It’s just kind of mind-blowing to me that people are not recording the C.T. values from all these tests — that they’re just returning a positive or a negative,” said Angela Rasmussen, a virologist at Columbia University in New York.
“It would be useful information to know if somebody’s positive, whether they have a high viral load or a low viral load,” she added.
This really matters for interpretation.
Officials at the Wadsworth Center, New York’s state lab, have access to C.T. values from tests they have processed, and analyzed their numbers at The Times’s request. In July, the lab identified 794 positive tests, based on a threshold of 40 cycles.
With a cutoff of 35, about half of those tests would no longer qualify as positive. About 70 percent would no longer be judged positive if the cycles were limited to 30.
In Massachusetts, from 85 to 90 percent of people who tested positive in July with a cycle threshold of 40 would have been deemed negative if the threshold were 30 cycles, Dr. Mina said. …
“I’m really shocked that it could be that high — the proportion of people with high C.T. value results,” said Dr. Ashish Jha, director of the Harvard Global Health Institute. “Boy, does it really change the way we need to be thinking about testing.”
Dr. Jha said he had thought of the PCR test as a problem because it cannot scale to the volume, frequency or speed of tests needed. “But what I am realizing is that a really substantial part of the problem is that we’re not even testing the people who we need to be testing,” he said.
Those are big numbers. What that means is that you don’t really know what the real-time numbers are on infectious people going about even if you used PCR tests on everyone daily? You might eventually know them but you can’t take action to mitigate historical pandemics!
What all this means is that, apart from anything else, you want to dial those thresholds down on PCR tests. It likely needs this for treatment and probably needs to be dialed down further if testing for infectiousness. Either that, or you report the Ct number so whomever is using the test can match that information for the purpose at hand.
Better than the Best
Saying that the best might have problems is one thing. The question is: can you do better than the best? The fact that you want to take the ‘gold-standard’ test and turn down the threshold for decisions is a strong signal that you can do better.
To see this, look at this comparison table of PCR versus Antigen tests from Eric Topel.
I have amended this table to show the ‘turned down’ threshold on the PCR test. You can now see that there is no longer any advantage to the PCR test compared to cheap, rapid tests. This is because the thing that was putting the ‘gold’ in ‘gold standard’ is not useful for the purpose. And everything else about those tests was expensive which is what you get if you want to gold plate something.
If you want the big, shiny one on the right, go to Harrods!
So here are the argument’s steps” PCR tests are the most precise … but you don’t want tests that precise so you will optimally ignore that precision … but, in that case, you want to find the cheapest way of getting the precision you want … which is not a PCR test but a rapid test.
Explainer: With Simple Words
Now once I started with PCR, Ct and Antigen I wasn’t really explaining all this in the most accessible way. So here is my attempt to explain my own paper using only the top ‘ten hundred’ most commonly used words in English.
One Final Thing
A very interesting piece has just appeared in the New Yorker about Jonathan Rothberg, a successful scientist/entrepreneur who is working on a rapid test. It is worth a read. Here is a passage that stood out in relation to today’s newsletter.
Rothberg, who is sensitive to the criticism that his own innovations are merely low-rent versions of better technologies, was determined in this case to make no compromises. His product would not only do away with the machines required by his competitors’ antigen tests, it would approximate the diagnostic rigor of the PCR standard. The F.D.A.’s willingness to relax its benchmark for rapid tests was, he felt, irrelevant; he liked to quote the old Hebrew National slogan, “We answer to a higher authority.”
So Rothberg’s approach is to match the PCR test going for a no-compromise solution. What I fear, however, is that such a fast test won’t, out of practical necessity, even measure the information required to assess the viral load in a subject let alone report it. In other words, unless it is dialed down, when it registers positive it will be for infection rather than infectiousness and so have too many false positives from the perspective of doing its job — helping determine whether people should be isolated.