It’s Our Fault Covid Death Forecasts Keep Changing

It’s Our Fault Covid Death Forecasts Keep Changing

On May 4th, the University of Washington’s Institute for Health Metrics and Evaluation (IHME) updated the estimation framework used for projecting Covid related deaths in the United States.

On May 4th, the University of Washington’s Institute for Health Metrics and Evaluation (IHME) updated the estimation framework used for projecting Covid related deaths in the United States. The substantial changes reported on their site reflect mobility in relation to social distancing policies, correcting reported cases to account for increased testing and drivers such as temperature, population density and testing per capita.

They also revised estimates upwards to 134,475 deaths. That projection was 72,433 just 3 days prior.

The fact is,** the IHME forecasts, upon which the US administration and health officials have relied have been a roller coaster ride since March, it’s just not obvious.**

Since the forecast visualization on the IHME site is dynamic, it’s challenging for the casual visitor to see these fluctuations over time. And it doesn’t help that most news cycles only report the most recent delta which obfuscate the issue. As the IHME doesn’t provide archives of their visualizations, I’ve used their freely available data to produce the graph below of forecast deaths over time.

Image for post

Data Source: IHME Image Credit: David Leibowitz


Ending up at the “same numbers”

In a March 30th press briefing, Dr. Deborah Birx, coronavirus response coordinator, referenced 12 global models that had been under review. Though no reasons were shared, they dismissed those and instead developed a new model from the “ground up” and then subsequently learned of research coming from the IHME. Dr. Birx ultimately landed on the IHME forecasts going forward as they “ended up at the same numbers.”

Since then, the administration and health officials have relied upon IHME models for coronavirus planning, to extend social distancing guidelines, for containment strategies, and most recently for plans to ease shelter-in-place restrictions at the state level to support economic recovery. As the IHME has modified their figures, their best-case predictions have been parroted at briefings.

For example, at a press conference May 1st, President Donald Trump said of the coronavirus, “hopefully we are going to come in below that 100,000 lives lost”, referring to the upper range of the average projection. Just 4 days prior, the average death toll shared by the administration was upwards of 70,000. And two weeks earlier on April 20th, that estimate was 60,000. All based upon the IHME models.

Though there may be justifiable concern about the perceived goal post movement, it diverts from the deeper issue: that few of us (from the administration, to news outlets, to every social media armchair data scientist) have been scrutinizing the soccer field itself.

As such, by way of omission, most news reports, press briefings and blind tweets of the death forecast have been open to interpretation.


What’s Going On?

The IHME models utilize data from across the globe, along with assumed adherence to response strategies in the United States (school closures, shelter-in-place orders, non-essential business closings) to predict peak healthcare demand, and a forecast death toll. Since the IHME provides academic analysis funded in part by well-known philanthropists, I ask in the words of Marvin Gaye, “_What’s going on?_”

“the IHME projections are based not on transmission dynamics but on a statistical model with no epidemiologic basis.”

There are several factors to consider. First, as the The Annals of Internal Medicine have simply put, “the IHME projections are based not on transmission dynamics but on a statistical model with no epidemiologic basis.” Rather than use an epidemiology discipline, they opted instead to base their model on the trajectory of cases and deaths from Wuhan, China along with indicators from Italy and South Korea.

Second, *the IHME uses an overly optimistic forecast *rendered in such simplicity that it obfuscates some rather complex assumptions and limitations, some of which had not been enforced consistently in the United States in late March and early April. As you’ll see in some screenshots captured below, the descriptions, context and visualization format keep changing.

Last, we are to blame, as we have blindly seized upon the headlines without scrutinizing the source and demanding context.


In some fairness to the IHME, the spartan projection page which contains the published results of each model change do provide links to update notes. But I suspect that reporting has been reviewed with the same rigor as an iOS license agreement preceding a software update. Like so many application Terms & Conditions, we simply scroll to the bottom and click “Agree” so we can get on to the next thing. The devil though, is in all of those details and assumptions, especially when the goal posts and context keep changing.

For example, take note of this screenshot of the IHME projections page on May 1st. The overly broad disclaimer “social distancing until infection minimized and containment implemented” displays at the top of the fold.

COVID-19 Deaths Per Day Forecast per IHME

author screenshot of IHME Covid-19 Projections (as of May 1, 2020)

But on May 4th, the revised model forecast nearly doubling the estimate just days prior carries no cautionary assumptions. They’ve also chosen to smooth the visualization, rather than their previous straight line plot based upon lags in reporting.

Image for post

technology politics research health science data science

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

50 Data Science Jobs That Opened Just Last Week

Data Science and Analytics market evolves to adapt to the constantly changing economic and business environments. Our latest survey report suggests that as the overall Data Science and Analytics market evolves to adapt to the constantly changing economic and business environments, data scientists and AI practitioners should be aware of the skills and tools that the broader community is working on. A good grip in these skills will further help data science enthusiasts to get the best jobs that various industries in their data science functions are offering.

Top 20 Latest Research Problems in Big Data and Data Science

Even though Big data is into main stream of operations as of 2020, there are still potential issues or challenges the researchers.

Applications Of Data Science On 3D Imagery Data

The agenda of the talk included an introduction to 3D data, its applications and case studies, 3D data alignment and more.

Data Science Course in Dallas

Become a data analysis expert using the R programming language in this [data science](https://360digitmg.com/usa/data-science-using-python-and-r-programming-in-dallas "data science") certification training in Dallas, TX. You will master data...

32 Data Sets to Uplift your Skills in Data Science | Data Sets

Need a data set to practice with? Data Science Dojo has created an archive of 32 data sets for you to use to practice and improve your skills as a data scientist.