INTRODUCTION

With an extended global vaccination campaign against SARS-CoV-2 infection and limited clinical experience, there is an understandable skepticism among doctors and lay people on potential side effects of these vaccines ⁽¹⁾. Moreover, almost any neurological symptom that occurs after vaccination has been attributed to an adverse effect of the vaccine.

Before the SARS-CoV-2 pandemic, the World Health Organization published the Assessment of the Causality of an Adverse Event Following Immunization (AEFI) manual, to standardize criteria on vaccine-related adverse effects ⁽²⁾. The scale has two main items, one referring to the severity of the event and the other to the possible causal relationship between the vaccine and the event. To improve AEFI criteria for establishing causality, a modified version of the AEFI manual (published in 2018) included a new algorithm and an extensive checklist ⁽³⁾. These criteria are widely used to establish vaccines’ adverse effects. However, there are reasonable doubts about AEFI usefulness in clinical practice when attempting to reach an agreement about severity and causality of suspected side effects of SARS-CoV-2 vaccines. This study aims to analyze consistency of the AEFI to designate probable side effects among patients who develop neurological symptoms after vaccination.

METHODS

The study included a series of consecutive patients evaluated at our Institution from January to April 2021, who presented neurological-related manifestations within 21 days following SARS-CoV-2 vaccination. After clinical cases were collected, a virtual meeting was organized to present them in full to a panel composed of 17 physicians that included five neurologists, four infectious disease specialists, and eight internal medicine residents. Cases were independently evaluated by all staff members in the light of AEFI criteria for opinion on severity and causality.

We first rated the percentage of agreement between the panel members regarding severity and causality for each case separately. Then, we analyzed cases using Fleiss's Kappa for more than two raters ⁽⁴⁾. Fleiss Kappa statistics were selected because agreement to measure has a nominal quality, the units are independent, and categories on the rating scale are independent and mutually exclusive.

The average proportion of agreeing with pairs out of all the possible pair assignments was the raters' random mean proportion of agreement, as defined by Fleiss. For this work, we used concordance criteria as follows: No agreement k ≤0.20, minimal agreement k 0.21-0.39, k 0.40-0.59, moderate agreement k 0.60-0.79, strong agreement 0.80-0.90, and almost perfect agreement k >0.90. ⁽⁴⁾

In a first Kappa analysis, we used the data of the 17 raters in two different tests following AEFI criteria, including the severity test (serious or non-serious), and the causality test (classified into five sub-items), after taking into account the Kappa (with 95% C.I.) value for each. For the second analysis, we calculated the Kappa index in three different sub-groups according to the raters’ medical specialty in order to observe and compare meaningful differences that may have occurred between the different subgroups.

RESULTS

The 11 patients were five women (mean age: 64.5 years; age range: 36-82 years) and six men (mean age: 55.6 years; age range: 22-72 years). Neurological manifestations appeared from one hour to seven days after vaccination (average 168 hours). Four of them were immunosuppressed for different reasons, three had cardiovascular risk factors, two had a benign intracranial tumor resected several years ago, and the remaining patient had no previous medical conditions.

The final diagnosis was manifestations of autoimmune origin in four patients (two autoimmune lymphocytic encephalitis, one acute disseminated encephalomyelitis, and one transverse medullary syndrome), three were strokes (two involving the anterior territory and one the posterior territory), two had mono-neuropathies (one peripheral facial paralysis and one Tolosa Hunt syndrome), one developed an encephalopathic event attributed to hyponatremia and benzodiazepine withdrawal syndrome, and the remaining patient had Herpes Zoster infection. Mean follow-up was 41 days (range: 7 to 90 days).

The agreement percentage analysis within observers showed that for the severity variable, the minimum percentage of agreement to consider the case "serious" was 45% (case 7) and the highest was 100% (case 11). Four cases had high discrepancy rates in this category, resulting in agreement percentages between 45% and 62%, while the other seven showed agreement percentages above 69%. Regarding the five categories of the causality variable, the minimum coincidence percentage was 25.7% (case 6) and the maximum 45.6% (case 2). In none of the 11 cases, the observers agreed in more than 50%.

The first kappa analysis to find the overall agreement included all the cases and all the observers. In the severity category, the Kappa index was 0.236 (95% C.I.: 0.064-0.408), which corresponded to a minimal concordance. Considering the items of causality, the Kappa index was 0.086 (95% C.I.: 0.037-0.135), which indicated no-concordance (Table 1). For neurologists, Kappa for severity was 0.383 (95% C.I.: 0.076-0.690) with a minimum concordance level, and for causality, it was 0.056 (95% C.I. -0.095-0.207) with no agreement at all. For infectious disease specialists, Kappa for severity was 0.185 (95% C.I.: -0.359 - 0.729) and for causality 0.082 (95% C.I.: -0.142-0.305) with minimal and no concordance, respectively. Among internal medicine residents, Kappa for severity was 0.117 (95% C.I.: -0.136-0.370) and for causality 0.115 (95% C.I.: 0.034-0.196), meaning minimal agreement in both cases.

DISCUSSION

Anti-vaccine movements have gained strength during the past decade ⁽⁵⁾. During the SARS-CoV-2 pandemic, up to 30% of the population presented reluctance to vaccination due to political, ideological, and social factors ⁽⁶⁾. The most common barriers are related to vaccines' severe potential side effects. Due to a social environment full of doubts and suspicions, it appears quite common for patients and their relatives to associate new neurological conditions with the vaccine.

The AEFI manual has been used in clinical and population settings to determine the severity and causality of post-vaccination adverse events. The self-reported incidence of adverse effects to SARS-CoV-2 vaccines shows that about 25% of people believe they developed systemic adverse effects and 66% local adverse effect ⁽⁷⁾. A meta-analysis of 14 studies describes the incidence of AEFI as 23% in inactivated inoculated vaccines, 48% in mRNA-based vaccines, and 76% in viral vector vaccines. Approximately 80% of the AEFIs were local and 36% systemic adverse reactions ⁽⁸⁾. In Mexico, neurological complications have been reported in less than 1% of vaccinated individuals, and in most cases, they have been categorized as non-serious. Severe events had an observed frequency of 2.4/100,000 applied doses ⁽⁹⁾. In our opinion, neurological-related events are less common but tend to be more serious, harder to diagnose, and even more difficult to classify.

In the present study, quantitative results showed a minimum level of agreement for the item "severity" (k=0.236), which slightly increased when only the subgroup of neurologists was considered (k=0.383). The category of serious events includes the words significant or persistent disability; the former seems to be sensitive to interpretation, and the latter constitutes a concept dependent on time. Apparently, disability was perceived differently among the observers, which did not allow complete agreement to classify the event as non-serious. Only in one case, there was 100% concordance as the patient died during hospitalization. Second, if the patient needs hospitalization or his hospital stay is delayed due to the adverse effect, AEFI must be classified as serious. In our series, 10 of 11 cases required hospitalization; therefore, all the cases should have been classified as serious. However, the qualitative observer's opinion considered that other factors could have influenced criteria for admission.

Causality criteria were even less consistent due to multiple factors. Nine of eleven cases had some comorbidity that can be sufficient to justify the adverse symptoms of the vaccine, which would automatically rule out a causal relationship. However, some raters considered that the vaccine could have a specific influence on the patient's status. We presented four cases with compromise of the immune system and systemic comorbidities in which the adverse effect could be related to either the vaccination or the underlying disease without being mutually exclusive. Three patients developed a stroke from five hours to three days after vaccination. Two of them had evidence of an embolic source as the cause of the stroke. In no case was it possible to ascertain a causal relationship between vaccination and the event.

Discrepancies between raters indicate insufficient evidence to determine causality with the perception that adverse effects need evaluation over time. As stated in the AEFI manual update, it is not possible to establish a definitive causal relationship between a particular AEFI and a particular vaccine based on a single AEFI ⁽²⁾. Many neurological AEFIs fall into the category of "adverse events of special interest" ⁽¹⁰⁾, which is a valuable tool for epidemiological purposes, but not oriented to determine the causality of the event in clinical practice.

The main limitation of the present work is the small number of cases and observers. However, this limitation is also a strength since it reflects the situation in everyday medical practice. The present study suggests that the AEFI criteria often does not allow to establish a relationship between the SARS-CoV-2 vaccine and an adverse neurological effect. Further studies are needed to corroborate our findings.