Show full content
A study by Flacco et al looked at potentially vaccine-related serious adverse events (PVR-SAEs) in a single province in Italy. It looks like the authors forgot to include some deaths in their calculations, and I’ve also noticed a few issues with timings and cohort sizes.
In brief:
- Every vaccinated cohort is compared to the entire unvaccinated cohort. Their sizes and follow-up times do not match, which casts doubt on the validity of the Cox relative risk calculations in the paper.
- The risk of PVR-SAEs are very high in the ‘1 Dose’ and ‘2 Doses’ categories compared to unvaccinated people. The supplementary data table shows how high the risks are for these vaccinated groups.
- Deaths for the ‘Greater than one dose’ group reported in the data tables don’t match the deaths in the Kaplan-Meier plot. When those deaths are considered, this group has approximately the same risk of all-cause death as the unvaccinated cohort (which disagrees with the authors’ calculations).
An important step in comparing different treatment groups in any study of drug effectiveness is matching follow-up times and group sizes. In this case, the authors were using all of the health records from one province in Italy (page 2) and were conducting a retrospective cohort analysis. Unlike a randomised controlled trial (RCT) where two matched patient groups are recruited and followed up for a set time, the authors used all of the health records from a single province between 2 January 2021 and 31 July 2022. This places some limitations on the data available. The authors couldn’t keep on recruiting patients until a target cohort size was reached. They had to work with what they had from the whole population in the province.
One problem is that the sizes of the different groups are not well matched (see graph below).

Another limitation is that the follow-up times are different. Note that the authors use the term “follow-up time” to mean “time to event” – the time measured from the start of the study until an event happens that fits the PVR-SAE criteria. The follow-up times (times to event) varied across each cohort.

This graph is based on the information in Table 1 (page 5). Curiously, it shows that the majority of unvaccinated people suffered some kind of PVR-SAE near the end of the study at around day 561. The standard deviation of 111 days takes these follow-up times to beyond the end date of the study.
The biggest problem I see with the way the authors have compared different groups is that all of the unvaccinated cohort are compared to all of each vaccinated group. This is show in the flowchart from the paper (Figure 1, page 3) below.

This means that instead of considering just the unvaccinated from July 2021 until the end of the study for the ‘Greater Than or Equal to 3 Doses’ group, all of the unvaccinated people from before this vaccine group existed were also used to calculate hazard ratios.
One assumption of the Cox method used by the authors is that of proportional hazards. It appears to me that comparing the unvaccinated group to the ‘Greater than one dose’ group breaks that assumption. Hazards will change across the period of the study in that vaccine group. I think that comparing ‘1 Dose’, ‘2 Doses’ and ‘Greater Than or Equal to 3 Doses’ to only those unvaccinated people who were recorded during those three different time periods would have made a more useful study.
Another Cox assumption is that the length of the study is the same for each group considered. As discussed above, this assumption is broken for all except the ‘Greater than one dose’ group. More information on problems with how Cox survival analysis has been used in other studies on COVID-19 treatment is discussed by Piovani et al.
How risky was vaccination?The authors state
…the likelihood of none of the individual PVR-SAEs was significantly higher among subjects who received at least one dose of vaccine…
page 10
and
…all subjects who received at least one dose, showed no increase in the incidence of any outcome…
page 11
yet the data tables show that for the ‘1 Dose’ and ‘2 Doses’ groups, this was not the case. The hazard ratios for each group are graphed below. These data are taken from Table 3 on page 9 of the paper.

In 10 out of 14 outcomes, the ‘1 Dose’ group had larger hazard ratios than the unvaccinated (up to 20x larger for myocarditis or pericarditis!). In 11 out of 14 outcomes, the ‘2 Doses’ group had larger hazard ratios than the unvaccinated.
The ‘Greater Than or Equal To 1 Dose’ and ‘Greater Than or Equal To 3 Doses’ groups always had lower risks of every PVR-SAE than the unvaccinated group. However, the authors may have forgotten to include some deaths in the ‘Greater Than or Equal To 1 Dose’ group.
How many died in the ‘Greater Than or Equal To 1 Dose’ group?The authors include a Kaplan-Meier (KM) plot (page 8) showing the survival estimates of time to death for this group and the unvaccinated group. The line on the KM plot labelled ‘One or more vaccine doses’ shows that 98.3% of this group survived to the end of the study. This is compared to 95.8% of the unvaccinated group.
This means that 4417 (1.8%) of the ‘Greater Than or Equal To 1 Dose’ group died during the study according to the KM plot. This disagrees with the results in Table 2 (page 6) of the paper which states that 3351 (1.29%) of this group died.
I read the data from the KM plot using WebPlotDigitizer to extract the times of deaths from the unvaccinated and ‘Greater Than or Equal To 1 Dose’ groups. I did a Cox Regression to calculate proportional hazards between the two groups using R (with the survival package). My results show that there was no significant difference in the risk of all-cause death between the two groups.

Risk of all-cause death in the ‘Greater Than or Equal To 1 Dose’ group is 1.07 (95%CI 0.99 – 1.15) compared to the unvaccinated group. The confidence intervals contain a hazard ratio of 1, therefore there is no difference between the two groups. My result with the extra deaths included is different to the authors’ result – they said that the hazard ratio for all-cause death in this group was a far lower 0.19 (95%CI 0.18 – 0.20).
Note that the total size of the cohort in my analysis (n=316329) differs from that in the paper (n=316314) by 15 people due to slight errors in the graph reading process when using WebPlotDigitiser. This makes little difference to the overall result. If the authors released the raw data (I’ve asked with no response from them yet) that would be very helpful in understanding this more.
One helpful feature of the studyThe authors define “vaccinated” as “vaccinated”.
…the follow-up started the day of the first (or single) dose of vaccine for vaccinated subjects…
Flacco et al, page 2
There is no “two weeks to let the vaccine take effect” waiting time before someone is considered to be vaccinated. This is the first study I’ve seen that does this – every other study I’ve heard of uses two or three weeks after a vaccine dose before someone is considered vaccinated.
What does the study say about vaccine safety?The authors finish their abstract with this statement:
Further research is warranted to evaluate the long-term safety of COVID-19 vaccines.
Flacco et al, page 1
Sadly, this is the main problem with all studies on COVID-19 vaccines. Long-term safety data should have been known before any vaccination programme began. The study reviewed here ignores a massive safety signal in the greatly increased hazard ratios for the ‘1 Dose’ group, and glosses over them by considering the whole of the vaccinated group instead.
It’s clear from this study that COVID-19 vaccination carries an increased risk of various types of circulatory and respiratory adverse events. It’s unhelpful that this study has inconsistencies in the data used and how it is reported. This appears to be a really promising study that would help to understand how frequently PVR-SAEs occur following vaccination. The missing deaths data and the problems with the authors’ analysis make this a missed opportunity to help us to understand COVID-19 vaccine safety.




































