How We Hope Vaccines Will Be Tracked

Vaccine shipments have begun to arrive in states across the country. Initial waves of recipients have received their first doses. And that means the data about these vaccinations has begun to flow. Beginning on December 20, the Centers for Disease Control and Prevention are publishing national counts of total vaccine doses distributed and administered in the United States. 

We expect that over the next few weeks, the CDC will also release data broken down by state/territory, providing two essential pieces of information about the US COVID-19 vaccination campaign: How many vaccines have made it out of the warehouses to each state, and of those, how many shots have made it into arms. 

This is very good news. As we’ve said from the start of our work tracking tests in early March, we believe that the collection and publication of standardized COVID-19 data for the country should be done by the federal government, not aggregated from public state reporting. When you have to resort to combining data from 56 different state dashboards, each with its own data definitions, even simple variations can prevent the construction of a comprehensive and fully consistent national picture. The federal government, on the other hand, has direct access to raw data from states, allowing many of those differences to fall away. 

As heartening as the promise of a federal vaccine dataset is, the agencies that manage it have a lot of work ahead to prevent the kinds of problems we’ve seen in COVID-19 testing data from recurring in the vaccine data. Specifically, the federal government will need to release more data than a simple count of vaccine doses distributed and administered. Even a minimal federal dataset must answer key questions about who is getting vaccinated. 

We know that many states already plan to release more detailed data than topline counts of doses. As we’ve been recording in our list of planned and extant state vaccine data dashboards, some have already launched dashboards with data like individuals vaccinated in addition to doses administered (an important distinction when some vaccines require two shots), county-level counts of vaccine administration, and data on the race, age and gender of vaccine recipients. 

The federal government can also release these more detailed metrics using the data it receives from state immunization information systems, in a standardized format. When the CDC releases its state-level vaccine tracker, we hope to see more data points than doses delivered and administered, including the following:

• The number of people fully vaccinated, also known as people with “series completion,” counting individuals who have received the intended number of doses of COVID-19 vaccine. For the Pfizer/BioNTech and Moderna vaccines which are currently available under Emergency Use Authorization, this would mean the number of people who have been fully vaccinated with the appropriate two doses.

• The number of doses distributed, administered, and series completions by vaccine manufacturer, including a complete historical time-series. Ideally this data will be broken down with the following details: 

  • Long-term-care facility level data via the Pharmacy Partners program (and any other program for distribution and administration in these facilities), broken down pharmacy partner name, facility name and location, manufacturer, and resident and staff. 

  • Aggregate demographic data in absolute numbers (not percentages) including: age data broken down by standard age brackets, race and ethnicity data broken down by standard categories from the Office of Management and Budget’s standards for the classification of federal data on race and ethnicity, and gender data, with cross-analysis available for as many categories as possible but at least including by age and race/ethnicity.

  • County-level data for distribution and administration, reported in a standardized and clearly defined way. (E.g., county of residence of the vaccine recipient for administration.)

The data we’ve seen from states

In the meantime, although we are not compiling state vaccine statistics as we have done for previous metrics, we have begun to provide a set of annotations on some basic features of the vaccination metrics that states are providing on their dashboards. (We also include New York City, since vaccines administered there are not tracked by New York State’s immunization information system.)

Already, we see quite a bit of variation in what states are reporting: Many track series completion, but in slightly different ways. Each state publishing demographic breakdowns has adopted its own version of age brackets and race categories, confounding cross-state analysis. A few states offer data at the level of the local health department, others by county. Some of these data differences may be unsolvable, caused by structural differences in state immunization information systems, but many others are likely just differences in presentation. 

With the launch of the CDC’s initial COVID-19 vaccine metrics, it appears that the federal government is poised to provide high-quality data to track the inoculation of millions of Americans in the coming months. This would be a great improvement over a decentralized dataset working around all the limitations of state dashboards. We hope to see a more comprehensive and fully standardized COVID-19 vaccination dataset from federal public health agencies in the very near future.

Additional research by Rebecca Glassman and Michal Mart.