Shelter-in-place orders in the U.S. did not significantly prevent projected coronavirus cases or deaths during the first wave of the COVID-19 pandemic, researchers from the University of Chicago have found, contradicting at least two previous studies. 

The orders also slightly reduced mobility and slightly increased unemployment, the researchers wrote in a peer-reviewed paper published in the Proceedings of the National Academy of Sciences on March 25. 

"A reasonable person could read our paper and conclude that the way we responded to the pandemic was not ideal," said co-author Anthony Fowler, a political scientist at the University of Chicago's Harris School of Public Policy, in an interview with The Academic Times. Fowler wrote the paper alongside his University of Chicago colleague Christopher Berry and graduate students Tamara Glazer, Samantha Handel-Meyer and Alec MacMillen.

The study is not evidence that the act of sheltering in place is ineffective at reducing coronavirus cases and deaths, Fowler and his co-authors said, but rather that county and state-level governments' orders to do so did not make a meaningful difference. Many Americans had already begun to alter their behavior in response to the pandemic prior to the orders being introduced, according to the researchers. 

One potential reading is that shelter-in-place orders should have never been implemented, while another is that the orders should've been enforced more strictly, Fowler said.

"We don't want to be on Fox News as people saying COVID wasn't a big deal," said Fowler. 

The researchers looked at data from Feb. 24, 2020, well before stay-at-home orders were implemented, through May 30, after many were removed. They used cellphone mobility data from Unacast, case and death data from Johns Hopkins University and unemployment data from the U.S. Department of Labor. They tracked case, death, mobility and unemployment data in comparison to the percentage of each state's population that was under a shelter-in-place order. 

In the paper, Fowler and his colleagues wrote that they "do not find detectable effects of these policies on disease spread or deaths."

The researchers also accounted for "spillover effects," through which stay-at-home orders in one state may affect behavior in neighboring states; for example, the implementation of a stay-at-home order in Illinois could affect behavior in Chicago suburbs across states lines in Indiana. The results were robust even while accounting for such situations, the researchers found. 

The work contradicts two previous studies claiming shelter-in-place orders reduced cases and saved lives, including a widely covered paper in the leading scientific journal Nature and another in Economic Inquiry. 

The Nature article, published in June, found that across six countries, including the United States and China, government interventions such as shelter-in-place orders prevented or delayed approximately 495 million coronavirus infections, including 60 million in the U.S. The paper received coverage in publications such as The Washington Post, Reuters and NPR.

But Fowler claims that the Nature article, authored by a large group of researchers from the University of California, Berkeley and led by Solomon Hsiang, included a fundamental methodological mistake. 

The Berkeley researchers' statistical model used day-of-week fixed effects to account for the growth in infections within each country. By contrast, the University of Chicago researchers used day-fixed effects, which Fowler said more accurately accounted for nationwide coronavirus trends. "When we include day-fixed effects, we're trying to account for all the nationwide trends of COVID at the time," said Fowler. "[The Berkeley researchers] are in effect not doing that." 

"That's something I would view as a pretty sloppy mistake," he added.  

However, Hsiang refuted Fowler's claims about the paper.

"First, the claims by Berry et al regarding our work are not factually correct nor supported by their analysis," Hsiang told The Academic Times. "Specifically, they do not report a replication result that is statistically different from our finding. Second, omitting day-fixed effects was deliberate, it was in no way a mistake."

Hsiang also called into question the University of Chicago researchers' results, saying, "They do not account for a very large number of non-shelter-in-place policies that were implemented during their study period. Berry et al also assume that cumulative cases would grow linearly in the absence of policy, which is at odds with well-established epidemiology. Infections grow exponentially early in epidemics if populations do not take actions to slow contagion."  

"Given these issues, I do not see how we can infer anything about policy effectiveness from the Berry et al analysis," he said.

When the University of Chicago team submitted their own paper to Nature that included criticisms of the Berkeley researchers' methodology, the journal rejected it without sending it out for review, according to Fowler.

"I'm sure that the review process for one of the other papers like the Shiang et al, that found politically pleasing results, I think looked different from the review process for our paper," said Fowler of Nature's alleged decision to reject the paper without review. "Results that confirm our priors receive much less scrutiny." 

A Nature spokesperson denied Fowler's claims of bias.

"We cannot confirm or deny reports of submissions that may or may have not been made to us, as we treat this information as confidential to the authors," the spokesperson said. "For COVID-19-related submissions, as with all other submissions, our editors make decisions based solely on whether research meets our criteria for publication — robust original scientific research, of outstanding scientific importance, which reaches a conclusion of interest to a multidisciplinary readership — and remain completely independent. Editorial decisions are never made based on whether they support a particular political position or policy, or previously published research."

The University of Chicago researchers' paper was also rejected from Science without review, according to Fowler. He said that some of his peers, editors and reviewers suggested that even if the paper's results are compelling and sound, they should not be released because they could be used to criticize coronavirus restrictions. Even some of the reviewers from the Proceedings of the National Academy of Sciences, which eventually published the paper, expressed hesitation, according to Fowler. 

"It was interesting to see how much people's political predispositions seemed to influence their evaluation of our paper," he said. 

Fowler cautioned that he doesn't know for sure whether Nature's rejection of his paper was politically motivated, saying, "For any particular instance, it's very difficult to know why an editor made the decision they did." But overall, he worries about politics influencing what research gets published. 

"One unfortunate thing is the extent to which so many things have become politicized these days," he said. "We would very much like to fight against that as much as we can."

The paper, "Evaluating the effects of shelter-in-place policies during the COVID-19 pandemic," published March 25 in the the Proceedings of the National Academy of Sciences, was authored by Christopher Berry, Anthony Fowler, Tamara Glazer, Samantha Handel-Meyer and Alec MacMillen, University of Chicago.