"An Egregious Failure of Scientific Integrity"
The first independent evaluation of NOAA's "billion dollar disasters"
On Tuesday, the U.S. National Oceanic and Atmospheric Administration (NOAA) will release with great fanfare the year-end update of its “billion dollar disaster” tally. If past is prologue, NOAA will vigorously promote the dataset in collaboration with environmental NGOs, reporters on the climate beat will uncritically parrot and amplify NOAA’s claims, and before long, the dataset will find itself cited in the peer-reviewed literature, identified by the U.S. government as a key indicator of human-caused climate change, and perhaps even cited by the U.S. president in support of the claim that all U.S. disaster costs are attributable to climate change.
What you will not see is any scientific critique, evaluation or independent peer review of the dataset.1
That is, until now.
Today, I am very happy to share with you a new preprint of a paper that I have just submitted to the new Nature journal, npj Natural Hazards. My paper, which was invited by the journal’s editors, is titled, Scientific Integrity and U.S. “Billion Dollar Disasters.”
Here is the abstract:
For more than two decades, the U.S. National Oceanic and Atmospheric Administration (NOAA) has popularized a count of weather-related disasters in the United States that it estimates have exceeded one billion dollars (inflation adjusted) in each calendar year starting in 1980. The dataset is widely cited and applied in research, assessment and invoked to justify policy in federal agencies, Congress and by the U.S. President. This paper performs an evaluation of the dataset under criteria of procedure and substance defined under NOAA’s Information Quality and Scientific Integrity policies. The evaluation finds that the “billion dollar disaster” dataset falls comprehensively short of meeting these criteria. Thus, public claims promoted by NOAA associated with the dataset and its significance are flawed and misleading. Specifically, NOAA incorrectly claims that for some types of extreme weather, the dataset demonstrates detection and attribution of changes on climate timescales. Similarly flawed are NOAA’s claims that increasing annual counts of billion dollar disasters are in part a consequence of human caused climate change. NOAA’s claims to have achieved detection and attribution are not supported by any scientific analysis that it has performed. Given the importance and influence of the dataset in science and policy, NOAA should act quickly to address this scientific integrity shortfall.
The full submission will soon appear at Nature’s preprint platform, Research Square, and meantime I have provided a link to the PDF.2
My paper is about scientific integrity, and not about the reality or importance of climate change — as I write:
The point here is not to call into question the reality or importance of human-caused climate change – it is real, and it is important.
Just because climate change is important does not mean that scientific claims related to climate change get a free pass or are somehow insulated from scrutiny. To the contrary, it is because of the importance of climate change in policy and politics that scientific claims related to climate change should adhere to the highest scientific standards. Each of us have a responsibility to ensure that happens. Far too often, that is not the case.
You can explore the details of the evaluation in the paper, and here is the bottom line:
The NOAA billion dollar disaster dataset comprehensively falls short of NOAA’s guidelines for scientific integrity. The shortfalls documented here are neither small nor subtle. They represent a significant departure from NOAA’s long-term history of scientific integrity and excellence, which has saved countless lives and facilitated the nation’s economy. A course correction is in order.
I conclude the paper with recommendations for that course correction. This is not rocket science — these recommendations could be implemented easily today or tomorrow.
Here is how the paper ends:
NOAA is a crucially important agency that sits at the intersection of science, policy and politics. It has a long and distinguished history of providing weather, climate, water, ocean and other data to the nation. These data have saved countless lives, supported the economy and supported significant scientific research. The agency is far too important to allow the shortfalls in scientific integrity documented in this paper to persist. The billion dollar disaster dataset is an egregious failure of scientific integrity. However, science and policy are both self-correcting.
You can download a PDF of the submission here. Note that it has not yet gone through peer review. I welcome and invite scrutiny, critique, and comment. Let’s “steel man” this thing. You can comment here at THB, send me an email, or enter your comments at Research Square when the preprint is posted.3
Thanks for reading! THB is reader-supported and so too is the research that results from that support. The “billion dollar disaster” paper is the first paper of mine in which I will formally acknowledge THB subscribers for their support. Assuming it survives the review process, it might also be the first peer-reviewed paper EVER supported by a Substack and acknowledged in print. You helped make that happen. Stay tuned for coming news about the future of THB, which is a bright one, and one that will depend more than ever on reader support. Meantime, please click the little heart at the top, share this post with your favorite reporter on the climate beat, and share also on your favorite social media platform. And keep those comments coming — Honest brokering is a group effort.
Disclosure: I was a fellow of a NOAA Cooperative Institute from 2001-2016, and I have great respect for NOAA’s research and operations.
I guess I am providing a pre-pre-print?
I’ll update this post with the Research Square link when it is eventually posted there.
From a slightly older KS friend:
In 1963 I was living on the top floor of Ellworth Hall at KU. KU is located on Mount Oread which is the highest point in Eastern Kansas, and Ellsworth in the highest place there. There was a magnificent view of farms and woodlands to the West. One Spring afternoon there was a storm front that approached from the West. That afternoon, you could see 7 tornados at the same time from the dorm windows. We were all Kansans and most had seen tornados before, but even we remarked that surely there would be a big write up in the local papers. There was nothing. A note in a small town local paper from mentioned that some farmer had lost a small out building.
Today the entire horizon as seen from Ellsworth is ranches homes, small businesses, schools and churches. If that event happed today, if would provide some Weather Channel specials that would last the rest of the decade. In 2011 one large tornado and several smaller ones tore thru Joplin Missouri. Killing several and doing millions of dollars worth of damage. I have bid and built a lot of work in the area. My best friend in 1962 was born there and we visited several times. I know where the tornados struck. In t4he 1960's a few cows would have died and no one would have noticed.
Sorry I came to this late. I'd prefer if you focused on specific aspects of "scientific integrity" and defined the concept. It might be an interesting grad student project to look at different agency's definitions. Ideas like "transparency of public information" were around long before the Data Quality Act or whatever is linked to current concepts of "scientific integrity."
I think we need look no further than some of the government scientist "let's not use our work emails" discussions during Covid to know there is a major transparency issue in government science, above and beyond other behaviors that might be called "against scientific integrity." But as you may recall I always thought the way that "scientific integrity" was promoted was basically political.. err.. signaling without much substance.