There has been a lot of talk lately about a few COVID-19 studies that use antibody testing to estimate the prevalence of infection across the population. In theory, antibody tests detect evidence that a
, which, given the lack of available testing while people are still experiencing symptoms (or may be experiencing only mild symptoms, if they aren’t completely asymptomatic), can be a useful tool for estimating the full scope of the pandemic. person has already had and recovered from COVID-19
In particular, a study of prevalence rates in Santa Clara County, California, has attracted attention for its conclusion that the prevalence of people with antibodies—meaning people who have been infected and gotten better—is much higher than official estimates.
Antibody studies are garnering interest because they might help us
who has immunity against COVID-19, which could be useful as states look to open up again—particularly as we work our way toward potential “herd immunity.” But putting too much faith into a study suggesting a low death rate—particularly one that doesn’t account for flaws and biases in its data—could lead people to think the disease is less dangerous than it really is. estimate
In the Santa Clara study, 3,330 people were recruited through Facebook ads and tested for antibodies against the virus that causes COVID-19. Out of these 3,330 people, 50 tested positive. Extrapolating from this number, the authors estimate that the prevalence of people who have recovered from the disease in Santa Clara County is 50 to 85 times what the official numbers suggested. Using this number, authors have estimated the death rate from COVID-19 to be between 0.12-0.20, versus
, which typically range between 1-3%. Assessing the death rate for this disease is tricky, given how new it is, as well as the lack of widespread testing. But overall, in scenarios where there have been higher rates of testing, the death rate seems to be somewhere in that 1-3% range. other estimates
The Santa Clara study’s results were startling, and have not held up to the scrutiny that followed the initial, heartening headlines. From the moment this preprint was released, one issue after another with the data has come to light. And the Santa Clara study is not the
—although in the past week, it has become the most notable. only antibody test that has sparked controversy
When it comes to analyzing these studies to determine how much stock we can place in them, these are the factors scientists are considering:
How were participants recruited?
For the Santa Clara study, participants were recruited through Facebook ads. Although the researchers did attempt to correct for oversampling of some demographics, this is a sampling technique that will introduce certain biases, not least of which is the fact that many of the participants might have been motivated by wanting to know if a previous illness they suffered had been COVID-19 or not. This factor may have significantly skewed the results, especially considering the study’s conclusions are based on only 50 positive results out of 3,330.
Then there’s the
that some participants for the Santa Clara study were recruited by the wife of one of the author sending an email to a middle school’s email list alongside the suggestion the testing would determine if they could “return to work without fear,” and falsely claiming the test was FDA approved. problematic discovery
Real Life. Real News. Real Voices
Help us tell more of the stories that matter
Become a founding member
A true sampling would be taken at random. In the real world, this isn’t always possible, but it’s important to note the sampling method a study followed and how it corrected for any possible biases.
How accurate is the test?
Tests for antibodies against COVID-19 are still being developed, and there’s still a lot we are learning when it comes to their accuracy. Complicating all of this is the fact that the FDA has
on letting companies sell tests that haven’t been vetted by the government. relaxed its rules
When it comes to test accuracy, there is the false negative rate to consider—when the test results say you don’t have COVID-19 antibodies when in fact you do—as well as the false positive rate—which is when the test results say you have antibodies against COVID-19, when in fact you don’t.
, performed by a team of more than 50 scientists, only three produced consistently reliable results. And of these 14 different tests, only one didn’t generate false positives. False positives are especially problematic, as they could offer those tested a false sense of security—a mistaken belief that one is immune, when in fact the test results were faulty. (Of course, it’s important to note that analysis of 14 different antibody tests to the virus that causes COVID-19.) we don’t yet know the degree to which antibodies confer immunity
It’s the false positive rate that is especially concerning in the Santa Clara study. The authors’ conclusions don’t account for any false positives, which is especially problematic considering the test they used has a
, while their reported findings were that 1.5% of their samples were positive. In theory, confidence interval of between 0.1% to 1.7% false positive rate all of those positive samples could be false positives, a likelihood that the authors didn’t account for. Even if the false positive rate is much lower, with such a small sample size, a slight difference could skew the results by quite a bit.
Do the results line up with reality?
The Santa Clara study uses its findings to estimate a mortality rate between 0.12 to 0.20 percent. However,
, if the death rate were only 0.12 to 0.20, then the number of COVID-19 deaths in New York City would suggest that 12.5 million people were infected, while city’s population is only 8.3 million. as Undark points out
Given that NYC is still recording new cases every day and hasn’t seen an influx of an extra 4 million people in the last month, this suggests that the death rate is not, in fact, between 0.12 to 0.20 percent.
Has this study been through peer review?
Life in a pandemic changes by the day, with new research results being released every hour. Generally speaking, this ongoing research is a very good thing: There is a lot we don’t know yet. That said, given the speed at which all this is happening, as well as our desperate desire for more information, more and more preprints—studies that yet to pass through peer review—are being covered as news.
It’s doubly important to double-check how a study is being covered when it is new and hasn’t been through peer review. What are other scientists saying about the methodology and results? What are they saying about the limitations of the study? No study is perfect. If the news coverage hasn’t included a diversity of voices—including an analysis by experts who weren’t involved in the research, as well as experts who can speak to its limitations—that’s a sign you might want to wait it out before deciding whether the resul
Subscribe to the newsletter news
We hate SPAM and promise to keep your email address safe