COVID-19 vaccine “95% effective”: It doesn’t mean what you think it means!

COVID-19 has the world more than ever in its grip – but there is hope: several vaccines have been developed which promise to deliver “95% efficacy”.

When people read this many assume that it means that 95% of vaccinated persons will be protected from infection – but that is not true. Even many (science) journalists get it wrong! If you want to understand what it really means, read on!

Let us take the example of the German biotech company BioNTech which developed its mRNA vaccine BNT162b2 in cooperation with pharmaceutical giant Pfizer. First of all, let me make it very clear that this is an extraordinary achievement and I am hopeful that it will prove to be a real game-changer in the fight against this horrible virus. I stress this because nowadays you cannot be cautious enough not to be instrumentalized by all kinds of strange people who have an interest in downplaying this devastating pandemic.

Now, what does a so-called efficacy rate of 95% mean? Let me start by stating what it doesn’t mean: it doesn’t mean that 95 out of 100 vaccinated persons will be protected from COVID-19, nor does it mean that it will reduce the severity of the illness in case you contract the virus despite being vaccinated.

To understand the real meaning let us go through the press release of Pfizer:

Primary efficacy analysis demonstrates BNT162b2 to be 95% effective against COVID-19 beginning 28 days after the first dose;170 confirmed cases of COVID-19 were evaluated, with 162 observed in the placebo group versus 8 in the vaccine group

What Pfizer did was to select about 43,000 voluntary participants where about half received the vaccine and the other half (= the control group) received only a placebo, without any active substance. After about a month after the first dose (i.e. one week after the second dose), they started to count the number of confirmed COVID-19 cases for each group: in the placebo group 162 cases were confirmed, whereas in the vaccine group only 8 cases appeared.

To get to the 95%-number the following calculation was performed:

(1 - 8/162) * 100
## [1] 95.06173

So the 95% is the relative risk-reduction in infections, it doesn’t tell us the absolute probability of not getting infected despite being vaccinated!

We already covered relative vs. absolute risk reductions (and its dangers) in this blog. There we saw that personograph plots (also called Kuiper-Marshall plots) are an excellent way to communicate risks.

Let us start by illustrating the confirmed COVID-19 cases in the placebo (= control) group for a better manageable group size of 2,500…

library(personograph) # first install from CRAN
## Loading required package: grImport
## Loading required package: grid
## Loading required package: XML

n <- 2500
inf_wo_vac <- 20 / n

data <- list(first = inf_wo_vac, second = 1-inf_wo_vac)
personograph(data,  colors = list(first = "red", second = "lightgrey"),
             fig.title = "20 of 2500 infected without vaccine",
             draw.legend = FALSE, n.icons = n, dimensions = c(25, 100), 
             plot.width = 0.97)

…and now for the vaccine group of the same size:

inf_w_vac <- 1 / n

data <- list(first = inf_w_vac, second = 1-inf_w_vac)
personograph(data,  colors = list(first = "red", second = "lightgrey"),
             fig.title = "1 of 2500 infected despite of vaccine",
             draw.legend = FALSE, n.icons = n, dimensions = c(25, 100), 
             plot.width = 0.97)

I think that those plots really put matters in perspective.

The main problem is that vaccine studies cannot directly measure what we really want to know: the effectiveness of the vaccine in the real world, i.e. how well it protects us from contracting the disease. Therefore they use efficacy as a proxy instead, i.e. relative risk-reduction of infections in the two study groups. This can give a good indication of the order of magnitude of the real-world effect but is not the same!

So while “95% effective” indeed does mean something different from what most people think it means let us hope that it translates into a real turning point for the better.