Why the Pfizer and BioNTech vaccine is a cause for optimism — and skepticism
On Monday morning, my mom sent me a text message: “Black cloud is lifting … Pfizer has a 90 percent effective vaccine.” She was one of the many people buoyed by the announcement from the drugmaker that early data suggests its Covid-19 vaccine — developed with the German biotech firm BioNTech — can prevent infection with the deadly disease.
What my mom missed, by no fault of her own, was that the news didn’t come with any detailed data, regulatory review, published study, or preprint. In fact, the trial isn’t even completed.
Instead, it was the latest instance of “science by press release” in the coronavirus pandemic.
In the race to stop the virus, drug companies, research groups, and other players involved in the development of Covid-19 vaccines and medicines have repeatedly announced early results from ongoing clinical trials in excited press releases.
There’s some justification for tipping off the public as soon as possible — before the usual scientific review steps are complete: “We’ve been on tenterhooks about information like this, and we want radical transparency,” writer and meta-scientist Hilda Bastian, who has been carefully following the vaccine race for months, reasoned.
But these press releases — picked up by the media, then reported to a public desperate for good news — have often culminated in dashed hope when the necessary steps in the scientific and regulatory process play out and the final results are less stunning than initially reported.
In the case of Pfizer and BioNTech, there certainly appears to be good reason for optimism (more on that in a moment). But it’s still too early to know whether, as my mom suggested, the black cloud is lifting. If the hope keeps getting dampened, however, we risk further eroding the already fragile public trust in science — and in the vaccines and treatments we’ve all been waiting for to finally end this pandemic.
Too many coronavirus medicines and vaccines have arrived with a side of hype
While it feels like the pandemic has been going on for 8,000 years, it was only in December 2019 that the novel coronavirus was discovered. And in this short time, there have been multiple instances of early, and trumped-up, coronavirus vaccine or medicine claims that don’t quite pan out.
Remember back to May, when Moderna — another company with a coronavirus vaccine that’s far along in clinical trials — put out a press release about promising phase 1 results? While it first caused the company’s stock valuation to swell, vaccine researchers pointed out in Stat that the information in the press release was way too preliminary and vague to gauge whether the vaccine was actually working.
Or remdesivir: In April, the National Institutes of Health announced promising results for hospitalized patients via press release. Later, further studies complicated the picture of the drug’s effectiveness, and today it’s not clear that remdesivir carries any benefit.
Or how about the Regeneron drug REGN-COV2, which President Trump called a “cure” in October. Most of what we know about its effectiveness comes from a September 29 Regeneron press release about a multi-phase, randomized, double-blind clinical trial involving only 275 people. The results said nothing of whether the drug cut the risk of death or “cured” people.
There have been some cases where early hype bears out: The press around the promise of dexamethasone for critically sick Covid-19 patients was later backed up by the Recovery Trial. There have also been examples of even peer review in journals failing to root out bad coronavirus science (see: the Lancet hydroxychloroquine controversy).
But press releases and news conferences have been particularly fraught venues for science communication — and the problem predates the pandemic. It also extends into the White House.
We’ve known for years that when news releases are filled with exaggerated claims, there’s greater odds the media that flows from them will be similarly hyped. We’ve also known that when drugmakers don’t publicly disclose detailed data about their methods and results, there’s no way of knowing their findings are reliable. (That’s why there’s a movement in medicine, nearly a decade old now, called the All Trials Campaign that advocates for all clinical trials to be registered and their results to be reported.)
Hype, exaggeration, and unverifiable claims are the last thing we need at a moment when there’s an infodemic going on in parallel with the pandemic, and the most powerful person in the country is also the largest driver of Covid-19 misinformation.
There’s real reason for optimism with the Pfizer and BioNTech vaccine — but it’s too early to know for sure
When asked why Pfizer chose to release the interim results of its phase 3 clinical trial in a press release, a media representative didn’t quite answer the question, instead telling Vox: “Pfizer and BioNTech plan to submit the full efficacy and safety data from our Phase 3 trial for peer-review in a scientific journal once the data is complete and analyzed.”
While that’s great, most of what we know their results is detailed in this press release. In fact, the study is still blinded to preserve the integrity of the trial, and Pfizer hasn’t even seen the results yet.
Here’s what we know. The trial, which began in July and enrolled more than 45,000 participants, was originally set up to allow an independent, external review committee to periodically look at the data, and see how the vaccine is working and whether the study should be stopped, five times: when 32 people were diagnosed with Covid-19, then 62, 92, 120, and 164 cases.
But after discussion with the Food and Drug Administration, the drug company decided to drop the first analysis (at 32 cases) and instead look at the data after 62 people had fallen sick with Covid-19. This meant that any findings about the vaccine’s effectiveness would be even more statistically sound.
By the time the review board actually analyzed the data for the first time, 94 people had been diagnosed with Covid-19, more than halfway to their final review at 164 confirmed cases. Ninety-four cases is a lot of data. And even if it’s early data, it suggests the finding of more than 90 percent efficacy may be particularly statistically powerful, as drug discovery chemist and blogger Derek Lowe explains.
What’s more, over 90 percent efficacy is also a far greater degree of protection than most people expected. (There had been serious discussions about life with a 50 percent effective vaccine.) As Vox’s Umair Irfan explains, these preliminary findings suggest “that if 10 people received the vaccine, at least nine of them will have protection against the virus compared to the placebo.” To date, the external data monitoring committee also hasn’t found any serious safety issues with the vaccine.
If the vaccine turns out to be as safe and effective as it appears now, it would put it in the same league as extremely protective routine childhood immunizations, such as the measles vaccine, the New York Times reports.
But the trial is not finished. And while the press release was quite detailed as far as press releases go, it wasn’t detailed enough, Boston University epidemiology professor Ellie Murray pointed out on Twitter. For example, we don’t know how many people dropped out in each arm of the trial (the group that got the vaccine versus the group that didn’t) — a phenomenon in clinical studies called “loss to follow-up.”
“Loss to follow-up could potentially have inflated their estimate if it occurred differentially between trial arms and I’d like to know how much drop out there was and whether and how they’ve adjusted for it,” Murray tweeted.
“It’s really problematic to be telling people what the data say but not allowing them to see the data for themselves,” University of Maryland School of Pharmacy professor Peter Doshi, who has been a prominent critic of Covid-19 vaccine trials, added. The press release raised a bunch of questions for him that aren’t answerable now: Did the shot cut the risk of serious complications from Covid-19? Did the researchers see fewer hospitalizations or ICU admissions in the vaccinated group? Or did they just see a reduced risk of mild infection? That’s saying nothing of the complexities of using the vaccine in the real world, such as cold storage.
“Even Pfizer presumably can’t see the data here, and only received a positive note from its review board,” Doshi added. “That unnecessarily sends the world into a frenzy talking about what this means, but we don’t really have any way of making sense of what has transpired since we don’t have access to the data.”
In an ideal world, we’d all be a little more patient. Researchers and drug companies running vaccine trials would wait until they were confident in their results to tell the public about them, reducing the risk of flip-flopping. When they did go public, they’d be more restrained in their press releases. And if they’re releasing results before their studies are completed, reviewed by a regulator, or published and peer-reviewed, they’d at least come out with detailed data so the public and policymakers could make informed choices.
For Bastian, this dearth of data is an even greater problem than how and where the news is shared. “Many argue there should be preprints or journal articles first. But we use data from various sources to assess clinical trials all the time, and so I’m not quite as concerned about that aspect,” she reasoned. “It’s the accuracy and detail that matters, and there’s too little detail here.”
This science by press release has to stop
The stakes for an effective vaccine in the pandemic couldn’t be higher: More than 50 million people have been diagnosed with Covid-19, and more than 1.2 million have died. These are just the known cases and deaths. The virus has upended lives and economies around the world, and governments are investing billions in vaccines to end the pain.
When we do finally have a viable Covid-19 immunization, we need the public to be able to trust that it’s safe and effective. And the road to building that confidence isn’t paved with preliminary, sparsely detailed news releases that are later overturned when the real work of vetting the vaccines is done.
“I’m going to hold my applause and continue to insist that to declare any Covid-19 product is based on science requires that the data be publicly available for all to scrutinize,” Doshi said.
When the Pfizer and BioNTech data is eventually released, and the trial is complete, we may learn this vaccine is as effective as it seems right now, and perhaps no harm will be done, Bastian said. “But if it turns out this is giving the world an exaggerated impression,” she added, “then it’s a whole other story.”