ABSTRACT
Background The purpose of this study was to evaluate the effectiveness of the influenza vaccine during the 2024-2025 respiratory viral season.
Methods Employees of Cleveland Clinic in employment in Ohio on October 1, 2024, were included. The cumulative incidence of influenza among those in the vaccinated and unvaccinated states was compared over the following 25 weeks. Protection provided by vaccination (analyzed as a time-dependent covariate) was evaluated using Cox proportional hazards regression.
Results Among 53402 employees, 43857 (82.1%) had received the influenza vaccine by the end of the study. Influenza occurred in 1079 (2.02%) during the study. The cumulative incidence of influenza was similar for the vaccinated and unvaccinated states early, but over the course of the study the cumulative incidence of influenza increased more rapidly among the vaccinated than the unvaccinated. In an analysis adjusted for age, sex, clinical nursing job, and employment location, the risk of influenza was significantly higher for the vaccinated compared to the unvaccinated state (HR, 1.27; 95% C.I., 1.07 – 1.51; P = 0.007), yielding a calculated vaccine effectiveness of −26.9% (95% C.I., −55.0 to −6.6%).
Conclusions This study found that influenza vaccination of working-aged adults was associated with a higher risk of influenza during the 2024-2025 respiratory viral season, suggesting that the vaccine has not been effective in preventing influenza this season.
Summary Among 53402 working-aged Cleveland Clinic employees, we were unable to find that the influenza vaccine has been effective in preventing infection during the 2024-2025 respiratory viral season.
INTRODUCTION
Influenza is a common respiratory viral infection with potential for substantial mortality and morbidity, and was estimated to be responsible for 145 000 deaths worldwide among all ages in 2017 [1]. The mortality could be much higher when there are pandemics of illness, which occur periodically, the most devastating recorded one being the influenza pandemic of 1918 which was estimated to have a case fatality rate of 2.5% and was considered to be responsible for more than 50 million deaths worldwide [2]. There is a seasonal pattern to illness, with most infections occurring during the winter months [3]. The influenza virus evolves over time [4], and as this happens an increasingly larger proportion of the population becomes susceptible to the newly evolved strains.
Influenza is also a vaccine-preventable illness. However, influenza vaccines do not induce long-lasting antibody titers, and annual influenza vaccination is recommended at the beginning of each respiratory viral season in the autumn months in the northern hemisphere. Additionally, the effectiveness of the vaccine in any given year depends on how similar the strains contained in the vaccine are to the strains causing infection that year. The most widely used seasonal influenza vaccine is the trivalent inactivated vaccine (TIV), which is composed of two influenza A virus types (H3N2 and H1N1) and an influenza B virus type [5]. A new vaccine is produced each year in an attempt to match the vaccine strains to the strains projected to be most prominent in the upcoming influenza season. Since the current process of developing the vaccine typically takes a few months, a decision on which strains to include in the vaccine must be made several months in advance. In years where there is a good match between the vaccine strains and the infecting strain, vaccine effectiveness is expected to be good. In years where there is a poor match between vaccine strains and the circulating infecting strain, vaccine effectiveness is expected to be poor.
Given the high morbidity and mortality burden of influenza, universal annual vaccination against the infection is recommended by the Advisory Committee on Immunization Practices [6]. Over the last couple of decades, policies of mandatory annual vaccination of healthcare personnel have been increasingly adopted across healthcare institutions [7].
Healthcare resource utilization, including hospitalizations, and resource needs such as quantity of antiviral medications needed, are strongly affected by how effective the vaccine is during any respiratory viral season. Early estimates of vaccine effectiveness of the influenza vaccine during any respiratory viral season can provide information that can help healthcare institutions and pharmacies prepare for the remainder of the season.
The purpose of this study was to evaluate the effectiveness of the influenza vaccine during the 2024-2025 respiratory viral season in North America.
METHODS
Study design
This was a prospective cohort study conducted at the Cleveland Clinic Health System (CCHS) in the United States.
Patient Consent Statement
The study was approved by the Cleveland Clinic Institutional Review Board as exempt research (IRB no. 23-625). A waiver of informed consent and waiver of HIPAA authorization were approved to allow the research team access to the required data.
Setting
For several years Cleveland Clinic has had a mandatory participation influenza vaccination program, which requires employees to either receive an annual influenza vaccine or seek an exemption on medical or religious grounds. The vaccine is provided to healthcare personnel free of charge. When healthcare personnel develop acute respiratory illnesses, they are encouraged to seek medical attention and the decision to test for influenza is made on a case-by-case basis by the treating provider either in the occupational health clinics or at their personal providers’ offices.
Participants
CCHS employees in employment at any Cleveland Clinic location in Ohio on the study start date were included in the study. Those for whom age or sex data were missing were excluded.
Variables
Variables collected were influenza vaccination date, age, sex, job location, job type categorization into clinical nursing or other, and date of positive test for influenza. Institutional data governance around employee data limited our ability to collect additional clinical variables.
Influenza was defined as a positive nucleic acid amplification test for influenza A or B any time after the study start date. Only molecular (including molecular point-of-care tests) performed within Cleveland Clinic Health System were included.
Outcome
The study outcome was time to influenza. Outcomes were followed until March 26, 2025.
Statistical analysis
For the 2024-2025 influenza season, the vaccine became available on 1 October 2024. This date was considered the study start date.
To assess whether there was a difference in the propensity to get tested among the vaccinated and the unvaccinated, the ratio of the proportion of the vaccinated who got tested to the proportion of the unvaccinated who got tested on each day of the study was examined, as was the ratio of the proportion of vaccinated persons’ tests that were positive to the proportion of unvaccinated persons’ tests that were positive on each day of the study.
A Simon-Makuch hazard plot [8] was created to compare the cumulative incidence of influenza in the vaccinated and unvaccinated states, by treating influenza vaccination as a time-dependent covariate [9,10]. Individuals were considered vaccinated 7 days after receipt of a single dose of an influenza vaccine. Subjects who had not developed influenza were censored at the end of the study follow-up period. Those whose employment was terminated during the study period before they had influenza were censored on the date of termination of employment. Curves for the unvaccinated state were based on data while the vaccination status of subjects remained “unvaccinated”. Curves for the vaccinated state were based on data from the date the influenza vaccination status changed to “vaccinated”.
Multivariable Cox proportional hazards regression models were fit to examine the association of various variables with time to influenza. Influenza vaccination was included as a time-dependent covariate. Variance inflation factors were evaluated to ensure that there was no multicollinearity in the models. The proportional hazards assumption was checked by examining Schonfeld residuals and there were no significant violations. Vaccine effectiveness (VE) was calculated from the hazard ratios (HR) for influenza vaccination in the models using the formula, VE = 1-HR.
The analysis was performed by N. K. S. and A. S. N. using the survival package and R version 4.4.2 (R Foundation for Statistical Computing) [11].
RESULTS
A total of 53402 employees in Ohio remained after excluding 1700 subjects (3.1%) for whom age or gender were missing. These employees formed the study cohort and a total of 43857 (82.1%) were vaccinated by the end of the study. The vaccine was the inactivated 3-valent influenza vaccine in 98.7% of those vaccinated. Altogether, 1079 employees (2.02%) acquired influenza during the 25 weeks of the study. Of these, 1066 (98.8%) were influenza A infections, the remaining being influenza B infections. A total of 2740 subjects (5.13%) were censored during the study period because of termination of employment before the end of the study.
Baseline characteristics
Table 1 shows the characteristics of subjects included in the study. Notably, this was a relatively young population, with a mean age of 42 years, and 75% were female. About 20% had a clinical nursing job.
Testing differences between the vaccinated and unvaccinated
The ratio of the proportion of the vaccinated who got tested to the proportion of the unvaccinated who got tested for influenza on each day of the study was significantly higher than 1.00 for most of the study (Figure 1), suggesting that the vaccinated were more likely to be tested than the unvaccinated on any given day. After excluding outlier values (> 3 SDs away from the mean), the slope of the regression line was 0.0009 and the slope was not significantly different from zero (P value 0.38), suggesting that the tendency for the vaccinated to be tested more than the unvaccinated did not change significantly over time.
Comparison of the ratio of the proportion of the vaccinated who got tested to the proportion of the unvaccinated who got tested for influenza on each day of the study. Each day is represented by a dot. The dashed line represents the reference line where the testing proportions are the same for those vaccinated and unvaccinated. Dots representing days on which a higher proportion of vaccinated than non-vaccinated individuals were tested for influenza will fall above the reference line, and dots for days on which a lower proportion of vaccinated than non-vaccinated individuals were tested for influenza will fall below the reference line. The red line represents the best fit line for the above ratio by linear regression, after excluding outliers (values >3 standard deviations from the mean ratio), with the shaded areas representing its 95% confidence interval.
However, the ratio of the proportion of vaccinated persons’ tests that were positive to the proportion of unvaccinated persons’ tests that were positive on each day of the study was not significantly different from 1.00, during the period when most of the infections occurred (Figure 2), suggesting that the additional testing among the vaccinated was not from a higher propensity to get tested but rather from a higher number of infections itself.
Comparison of the ratio of the proportion of vaccinated persons’ tests that were positive to the proportion of unvaccinated persons’ tests that were positive on each day of the study. Each day is represented by a dot. The dashed line represents the reference line where the proportion of tests positive are the same for those vaccinated and unvaccinated. Dots representing days on which the vaccinated had a higher proportion of tests positive than the unvaccinated will fall above the reference line, and dots for days on which the vaccinated had a lower proportion of tests positive than the unvaccinated will fall below the reference line. The red line represents the best fit line for the above ratio by linear regression, after excluding outliers (values >3 standard deviations from the mean ratio), with the shaded areas representing its 95% confidence interval. This was based on data for days where both vaccinated and unvaccinated had at least one test done. Data were inadequate to obtain data points prior to day 76 of the study.
Simon-Makuch plot comparing the cumulative incidence of influenza for subjects stratified by vaccination status. Day zero was 1 October 2024, the day the influenza vaccine began to be offered to employees for the respiratory viral season. Point estimates and 95% confidence intervals are jittered along the x-axis to improve visibility.
Influenza vaccine effectiveness
Very few subjects developed influenza A in the first two months of the study and the daily number of infections began to increase steadily about 70 days after the study start date. The cumulative incidence of influenza did not appear to be significantly different between the vaccinated and unvaccinated states early on, but over the course of the study the cumulative incidence of infection increased more rapidly among the vaccinated than among the unvaccinated (Figure 1). The risk of influenza was significantly higher for the vaccinated compared to the unvaccinated state on unadjusted Cox proportional hazards regression (HR, 1.27; 95% C.I., 1.07 −1.51; P = 0.007). In a multivariable model which adjusted for age, sex, clinical nursing job, and primary employment location, the risk of influenza remained significantly higher for the vaccinated compared to the unvaccinated state (HR, 1.27; 95% C.I., 1.07 – 1.51; P = 0.007). Point estimates and 95% confidence intervals for hazard ratios for acquisition of influenza, for the various variables in unadjusted and adjusted Cox proportional hazards regression models, are shown in Table 2. Based on the multivariable model, the influenza vaccine would have had an effectiveness of −26.9% (95% C.I., −51.0 to −6.6%).
DISCUSSION
This study found a significantly higher risk of influenza among the vaccinated compared to the unvaccinated state in northern Ohio during the 2024-2025 influenza season.
The strengths of our study include a sample size that was large enough to find a significant difference in incidence of influenza between the vaccinated and unvaccinated states, and a study design that allowed for actual calculation of risk rather than an extrapolation from odds ratios obtained from “test-negative” design studies as has become the trend in recent vaccine effectiveness studies. “Test-negative” design studies are case-control studies, and one cannot obtain relative risks from case control studies. One can obtain odds ratios, but odds ratios always exaggerate the size of the effect compared with relative risks and when the event is not rare, as is usually the case in published “test-negative” design studies, this difference can be substantial [12]. That is why estimates of vaccine effectiveness from “test-negative” design studies, which treat odds ratios as if they are relative risks in order to estimate vaccine effectiveness, systematically overestimate true vaccine effectiveness. An important strength of the study was its consideration of the possibility that testing behavior might differ between the vaccinated and unvaccinated. This analysis found that over the course of the study, despite people in the vaccinated state being more likely to get tested for influenza than those in the unvaccinated state, the proportion of tests positive among the vaccinated was not different from the proportion of tests positive among the unvaccinated, suggesting that the excess tests among the vaccinated were from an excess of infections rather than from differences in testing behavior. The study methodology of treating vaccination as a time-dependent covariate also allowed for determining vaccine effectiveness in real time, which provided us with very early signals about the magnitude of vaccine effectiveness within a few weeks of the first cases of influenza being diagnosed.
The study has several limitations. The vaccine was the 3-valent inactivated influenza vaccine in about 99% of our study cohort. The possibility that other influenza vaccines might have been more effective cannot be excluded. Infections diagnosed on the basis of home testing kits alone would have been missed. The study was not designed to compare the risk of influenza-associated hospitalization or mortality, or to examine if the vaccine decreased severity of illness, because these outcomes were not expected to occur in numbers large enough to allow for a meaningful analysis. Our study of healthcare personnel included no children and few elderly subjects and primarily consisted of individuals who were healthy enough to be employed. A minority would have been expected to have been severely immunocompromised.
The results are generalizable to relatively healthy adults in the USA, which is a major target of adult influenza vaccination efforts. Although the study was done in northern Ohio, there is little reason to assume that the effectiveness of the vaccine would have been different in a different geographic region within the continental USA.
Given all the variables that can influence the effectiveness of the influenza vaccine in any given year, and our current processes for developing the vaccine, it may be asking for too much to expect the vaccine to be highly effective year after year. It therefore becomes important to evaluate the effectiveness of the vaccine every year. This study found that influenza vaccination was associated with a higher risk of influenza among adults in the healthcare workforce in northern Ohio, USA, during the 2024-2025 winter season, suggesting that the vaccine has not been effective in preventing influenza this season.
Data Availability
All data produced in the present study are available upon reasonable request to the authors
Notes
Author contributions
N. K. S.: Conceptualization, methodology, validation, investigation, data curation, software, formal analysis, visualization, writing-original draft preparation, writing-reviewing and editing, supervision, project administration. P. C. B.: Resources, investigation, validation, writing-reviewing and editing. A. S. N.: Methodology, formal analysis, visualization, validation, writing-reviewing and editing. S. M. G.: Resources, writing-reviewing and editing.
Potential conflicts of interest
The authors: No reported conflicts of interest. Conflicts that the editors consider relevant to the content of the manuscript have been disclosed.
Funding
None.
Footnotes
1. A longer duration of study is reported. 2. Removed "Interim" from the title, as the study duration now spans most of the respiratory viral season studied. 3. Job type is now categorized into clinical nursing vs. other, rather than clinical vs. non-clinical as done previously, as this provides a better representation of closeness of patient contact. 4. A more meaningful assessment of testing differences among the vaccinated and unvaccinated states is presented. This is really important because results could be affected significantly if there are meaningful differences in testing between the groups that is unrelated to occurrence of the outcome. Given the importance of this assessment, it is presented in the main study itself rather than in supplementary material.
