Don't Expect Covid-19 Research to Deliver Certainty

(Bloomberg Opinion) -- To assess the safety and effectiveness of their Covid-19 vaccines, BioNTech SE, Pfizer Inc. and Moderna Inc. sponsored clinical trials overseen by independent boards that recruited tens of thousands of participants, randomly assigned them to receive vaccines or placebos and waited to see if there was a discernible difference in outcomes between the two groups.

With the rapid spread of the new coronavirus in the U.S. this fall, they didn’t have to wait long — and the results were stunning. In both trials, about 95% of those who developed Covid symptoms were in the placebo group, meaning that the vaccines appeared to be 95% effective.

A chart from a U.S. Food and Drug Administration briefing document illustrates that effectiveness perhaps more clearly than the percentages do. It shows the cumulative incidence of symptomatic Covid-19 cases in the days following the first dose of the BioNTech-Pfizer vaccine (in blue) versus a placebo (in red).

The chart for the Moderna vaccine is nearly identical. What both show is that for the first 10 days or so after getting a shot, people in the placebo and vaccine groups were infected with the virus at nearly identical rates. After that, the placebo recipients kept on getting Covid-19 at the same pace, while those who received the vaccine hardly got it at all.

These results don’t bring absolute certainty about how well the vaccines will work, or for how long, or what if any long-term side effects they might bring. But they provided convincing enough evidence to persuade the FDA to grant emergency-use approvals for both, with multiple peer agencies abroad taking similar steps, and to offer realistic hope that the pandemic could be over by summer.

If only most of the other virus-related studies delivered results like that! Consider the new mutation of SARS-CoV-2 (the virus that causes Covid-19) that was first identified in the U.K. in September and about which epidemiologists began ringing alarm bells last month. It seems to be more contagious than earlier variants — how else could it have so quickly become the country’s dominant strain? But is it 70% more contagious, as originally estimated? Or 56% more, as one quick-turnaround study concluded last month? Or something more like 10% to 20%, which some experts still hold out hope for?

We’ll probably never know for sure. There’s no plausible and ethically defensible method for comparing the contagiousness of different virus variants in the controlled way that vaccines are tested, plus the exact answer just isn’t that important. Far more crucial is whether existing vaccines can stop it. The early thinking is that they can, and we should get a lot more information on that over the next couple of months.

So it goes with the state of knowledge on Covid-19. There’s certainly a lot of it: 254,798 Covid-related articles, working papers and other publications had been produced as of Tuesday, according to the Dimensions database maintained by Digital Science & Research Solutions, Inc., with 7,871 clinical trials completed or in the works. But there are huge differences in the reliability, importance and durability of the findings.

At one end are large, randomized controlled trials such as the vaccine studies. The great strength of that RCT research method, which physicians first started dabbling with in the 1700s and developed into its modern form in the decades after World War II, is that researchers can hold more or less everything equal except for the vaccine or other treatment being tested, allowing them to draw clear connections between cause and effect.

Those connections are seldom made as clear as they have been in the BioNTech-Pfizer and Moderna vaccine trials, though. The one other big Western vaccine trial for which results have been released found the vaccine developed at the University of Oxford and produced by AstraZeneca Plc to be 70.4% effective. Subjects who were erroneously given a half-dose on their first shot seemed to be better protected than that, but that result may necessitate new trials to determine that it’s not a fluke.

Meanwhile, the large-scale Randomised Evaluation of Covid-19 Therapy (known by the portmanteau “Recovery”) trials in the U.K. have so far found three of the four treatments studied, azithromycin, hydroxychloroquine and lopinavir-ritonavir, to deliver “no clinical benefit” and the fourth, dexamethasone, to reduce deaths by about one-third among patients already on ventilators. Trial results of new antibody therapies from Regeneron Pharmaceuticals Inc. and Eli Lilly and Co. seem to indicate improvements in outcomes of 50% or more, but the trials have been small and there are still lots of questions about how to maximize the effectiveness of the treatments.

Go from pharmaceuticals to the so-called “non-pharmaceutical interventions” — such as masks and distancing — used to slow the pandemic, and things rapidly get more complicated and muddled. In many cases RCTs just aren’t practical, and even when they are they face obstacles that trials of shots and pills generally don’t.

Norwegian public-health researcher Atle Freheim tried to set up an RCT last spring on the effect of school closures on the spread of Covid, for example, but the country’s health minister, while agreeing that it was a good idea, said (in Freheim’s paraphrase) “it would be too difficult to get popular support for it.”

In Denmark, a large group of researchers did succeed in conducting an RCT on the protective benefits of face masks this April through June, but it was tough to draw any conclusions from it. Just 46% of those who were given disposable surgical masks and instructed to wear them outside the home reported following those instructions every week, with another 47% saying they did so most weeks — making the trial a test perhaps more of adherence to mask mandates than of mask effectiveness. Low incidence of the disease in Denmark during the study period also made it hard to produce meaningful results, as did the fact that the study wasn’t even testing what is generally represented to be masks’ main benefit: keeping mask wearers from infecting others.

“Statistical insignificance and/or too-small effect measured are the most likely outcome as an artifact of the study’s design, regardless of the true effect of masks,” three public-health researchers argued in a letter posted online well before the results were published. Sure enough, the study’s authors ended up reporting that while members of the mask group were less likely to test positive for SARS-CoV-2 infections or antibodies than those in the non-mask control group, the difference (1.8% versus 2.1%) wasn’t statistically significant.

Most evidence on masks thus comes not from RCTs but from simple observation of how masks affect air flow, as well as studies that compare real-world Covid-19 outcomes where and when masks are widely used with those where and when they aren’t. The latter approach has great limitations, though, compared to an RCT where every variable but one can be held constant. One early study that made big claims for the effectiveness of masks by comparing the spread of Covid-19 in different parts of the U.S., published in June in the prestigious Proceedings of the National Academy of Sciences, “ignores other clear differences in disease control policy between these areas,” wrote the same three researchers cited above, along with several dozen others, in a letter requesting retraction of the article (it hasn’t been retracted).

Still, I would guess that all the signatories of that letter also believe that masks are effective in the fight against Covid-19. Noah Haber, a postdoctoral fellow at Stanford University’s Meta-Research Innovation Center and a ring-leader of both letter-writing efforts, certainly does. Which begs the question of why. As Haber said last month when I sought him out for advice on weighing Covid evidence, “One of the things that I get asked a lot, especially with masks, is ‘Okay, these studies are bad. What’s a good one that you’re basing all of your opinions on?’”

While there are definitely better mask studies than the one discussed above, there is none that delivers results anywhere near as convincing as the BioNTech-Pfizer and Moderna vaccine trials. Instead, as Haber said in an article for Wired, there is “a collection of evidentiary bits and pieces; and when these are taken all together they say that masks are certainly somewhat effective — maybe even very effective — at slowing the spread of SARS-CoV-2.”

Also key, in my opinion, is that the cost-benefit analysis for masks is a bit like Pascal’s wager on the existence of God. If they do in fact work the gains are huge, while if they don’t the cost of having worn them for nine or 12 months is relatively modest.

This isn’t true of some of the other non-pharmaceutical interventions used to combat Covid: closing schools and banning indoor dining and entertainment are extremely costly, and while the big shift to remote work has brought benefits as well as costs, it’s had wrenching economic consequences for urban downtowns and transit systems. Yet for the most part we’re also having to make do with “a collection of evidentiary bits and pieces” to arrive at such decisions.

As someone who has been struggling since starting to write about the pandemic 10 months ago with how to weigh that evidence, I can attest that there’s no simple answer. Relying on scientific journals to sort things out tends to disappoint. As has been made clear by decades of often-contradictory published studies on whether consuming this, that or the other food or beverage is likely to make you live longer or die young, many top journals have a bias for bold and surprising study results over judiciously expressed uncertainty.

Since April, the 2019 Novel Coronavirus Research Compendium hosted by the Johns Hopkins Bloomberg School of Public Health has provided a happy corrective by evaluating the strengths and limitations of hundreds of Covid-related studies, and developments elsewhere in scientific publishing suggest that such open peer review will become more common. Haber and five other researchers have also recently organized a systematic review of Covid-19 policy evaluation studies to determine whether they at least “meet basic requirements for study design.” And the Centers for Disease Control and Prevention, after often being muzzled by President Donald Trump’s administration, already seems to be returning to its proper role as expert and mostly independent weigher of evidence.

None of that, though, can remove the uncertainty and error inherent in figuring out what to do about a fast-moving disease. At the moment, governments are beginning to announce new rounds of school shutdowns, non-essential business closures and other tough measures to combat the spread of a new SARS-CoV-2 variant that seems to be a lot more contagious than the old one. Can they be sure this is the right approach? No, not yet at least.

It’s quite clear that reducing people’s contacts with other people slows the spread of the disease, but the question of which policies most efficiently accomplish this has generated a cacophony of answers. A group of mostly Vienna-based researchers published an effectiveness ranking of Covid-19 government interventions in November that put canceling small gatherings first and closure of educational institutions second, but the evaluation for the Johns Hopkins research compendium offered a long list of reasons why the conclusions could be off.

In conjunction with the arrival of seemingly very effective vaccines, though, there is at least a logic to taking drastic measures aimed at slowing the spread of a more-contagious SARS-CoV-2 variant until a big enough share of the population is vaccinated that it slows down automatically. Waiting for certainty isn’t really practical in a pandemic.

This column does not necessarily reflect the opinion of the editorial board or Bloomberg LP and its owners.

Justin Fox is a Bloomberg Opinion columnist covering business. He was the editorial director of Harvard Business Review and wrote for Time, Fortune and American Banker. He is the author of “The Myth of the Rational Market.”