Why do the Covid-19 vaccine trials take so long?

The whole world is waiting for a Covid-19 vaccine. More than 100 different vaccines are being investigated, and 42 of them are already being tested in humans, which is lightning-fast progress in the world of vaccine development.

11 vaccines are already in Phase 3 trials, which use thousands of volunteer subjects to test whether a vaccine really works. If any of these 11 trials are successful, as many scientists expect them to be, then the world might finally begin the process of opening back up.

By all accounts, though, we’re still a few months away from having an approved vaccine. Why does this take so long? Today I’ll try to answer this question. A little math is involved, but we don’t need much to get the basic idea across.

In a Phase 3 trial, we give the vaccine to large numbers of people to see if it works. Some of the 11 current trials use as many as 40,000 volunteers, so let’s use that number for the sake of discussion. In the trial, we might give the real vaccine to half the volunteers–20,000 people–and give a placebo to the other 20,000. A placebo is a harmless shot, typically just saline solution, that won’t have any effect. The volunteers don’t know if they’re getting the real thing; this is called “blinding.”

Then we wait. Here’s the problem: we don’t infect anyone intentionally, so we have to wait for naturally-occurring infections, and it might take a long time to see those. Subjects just go about their lives, and if they get sick, the study records that fact.

So the question is, how many people in each group of 20,000 will be infected in the first week? The first month? Two months? The answer is that we simply don’t know. To speed things along, scientists running the trials try to select volunteers who are more likely than most people to get infected, but we can’t really control the number of people who get sick.

Let’s suppose that after just one week of a trial, 3 people in the placebo group come down with Covid-19, and no one in the vaccine group gets sick. So far so good, right? But we can’t possibly conclude that a vaccine works based on just 3 cases. Statistics tells us that those 3 cases might have just happened by chance. (More precisely, if 3 cases occur in the 40,000 subjects, and if the vaccine doesn’t work at all, then there’s still a 12.5% chance that all 3 cases will occur in the placebo group.)

Suppose that 2 months roll by, and now we have 100 people in the placebo group who got sick, and only 10 infections in the vaccine arm. This is much, much better: without going into the math, a difference of 100 versus 10 would be highly significant, suggesting that the vaccine reduced cases by 90%.

But what if 2 months roll by and the placebo group only has 10 cases? Even if the vaccine group has zero cases, such a small number is not going to be enough to give us confidence that we have an effective vaccine. We want to see as many cases as possible–but we can’t force the issue. We have to wait.

In the US, the FDA has announced that a vaccine has to protect at least 50% of people in order to be declared effective. This means we need to see enough cases in to be confident that a vaccine confers that degree of protection. 50% is a pretty low bar, but so far none of the trials have announced even preliminary results showing that they’ve met that standard.

(Aside: “blinding” is really important in these trials. If subjects know they’re getting a placebo, they might be extra-careful to avoid exposure to the virus. This would artificially depress the number of cases in the placebo group. Conversely, if they know they’re getting the vaccine, they might be more reckless, increasing the exposures and cases in that group. In order for the results to be valid, we need all the subjects to behave the same.)

A faster option? There is a way to speed up this process: a “challenge” trial, where subjects are intentionally infected with the virus. The UK is preparing to start such a trial in January, first administering vaccines to healthy volunteers, and then exposing them to the SARS-CoV-2 virus about a month later. This is a far faster way to determine if a vaccine is working, but it creates serious ethical quandaries, because we don’t have a cure for the virus. If the world has an effective vaccine in January, I expect that the challenge trial will be cancelled. That wouldn’t be a bad outcome.

A new Russian Covid-19 vaccine looks promising, but did they fabricate some of their data?

Last week, a team of Russian scientists published the results of two phase 1/2 vaccine trials for a new Covid-19 vaccine developed in Russia. The study appeared in The Lancet, one of the world’s leading medical journals.

This vaccine has already received tremendous attention after Russian leader Vladimir Putin announced they would start administering it widely, before any phase 3 trials were under way. As I wrote last month, it’s not a good idea to skip these Phase 3 trials.

Nevertheless, the results from the early stage trials of both vaccines look quite good. Although the trials were small, with just 76 subjects, 100% of the subjects had a strong antibody response, and none of them had anything more than mild reactions to the vaccine. This suggests that both vaccines might be effective, although it’s too soon (after just 76 people) that it will be safe on a large scale.

There’s another problem, though.

Within 3 days of the paper’s publication, Enrico Bucci from Temple University described a series of apparent duplications in the figures presented in the Russian paper. He published his findings on his website as a “note of concern” that dozens of other scientists have signed.

I’ve read the paper and looked at all the figures, and it’s clear that something is wrong with the data.

Let’s look at one example to see what is going on. Here’s a small part of Figure 2A from the paper:

Each little column of dots shows a distinct group of 9 subjects, where the height of a dot indicates the level of antibodies found in that subjects. Notice that the 9 subjects in the red box (boxes added for emphasis) on the left have an identical pattern to those in the box on the right. These are completely independent subjects, and such a pattern is exceedingly unlikely.

It’s possible that this happened by chance, but then the problem is that this isn’t the only apparently duplication. Prof. Bucci identified at least 13 instances where sets of results are identical or near-identical between two different time points or two different sets of subjects. The other duplications look a lot like the one shown here.

The simplest explanation is that the data for some of the experiments were simply copied over from other experiments. As reported in The Moscow Times, the lead author of the study, Denis Lugonov, said there were no errors in the data. Because the authors of the Russian study didn’t provide their raw data, and The Lancet didn’t require it, other scientists can’t really check.

What are we to make of this? The details of the study are clearly explained, and the Russian vaccines use a design (an adenovirus modified to contain the SARS-CoV-2 spike protein) that is similar to other vaccines that so far seem safe and effective. Thus it’s quite possible that this vaccine will work–and it will be good for the world if it does. But the questionable data raise questions about whether the scientists behind this phase 1/2 trial have really done all of the experiments that they describe. The study concludes by noting that a phase 3 clinical trial with 40,000 participants is planned. Let’s hope that one yields positive–and genuine–results.

[Hat tip to Retraction Watch for drawing my attention to this study.]