"An Egregious Failure of Scientific Integrity"

The first independent evaluation of NOAA's "billion dollar disasters"

ChatGPT’s fever dream envisioning NOAA’s “billion dollar disaster” database. Do you spot any errors? I sure did.

On Tuesday, the U.S. National Oceanic and Atmospheric Administration (NOAA) will release with great fanfare the year-end update of its “billion dollar disaster” tally. If past is prologue, NOAA will vigorously promote the dataset in collaboration with environmental NGOs, reporters on the climate beat will uncritically parrot and amplify NOAA’s claims, and before long, the dataset will find itself cited in the peer-reviewed literature, identified by the U.S. government as a key indicator of human-caused climate change, and perhaps even cited by the U.S. president in support of the claim that all U.S. disaster costs are attributable to climate change.

What you will not see is any scientific critique, evaluation or independent peer review of the dataset.¹

That is, until now.

Today, I am very happy to share with you a new preprint of a paper that I have just submitted to the new Nature journal, npj Natural Hazards. My paper, which was invited by the journal’s editors, is titled, Scientific Integrity and U.S. “Billion Dollar Disasters.”

Here is the abstract:

For more than two decades, the U.S. National Oceanic and Atmospheric Administration (NOAA) has popularized a count of weather-related disasters in the United States that it estimates have exceeded one billion dollars (inflation adjusted) in each calendar year starting in 1980. The dataset is widely cited and applied in research, assessment and invoked to justify policy in federal agencies, Congress and by the U.S. President. This paper performs an evaluation of the dataset under criteria of procedure and substance defined under NOAA’s Information Quality and Scientific Integrity policies. The evaluation finds that the “billion dollar disaster” dataset falls comprehensively short of meeting these criteria. Thus, public claims promoted by NOAA associated with the dataset and its significance are flawed and misleading. Specifically, NOAA incorrectly claims that for some types of extreme weather, the dataset demonstrates detection and attribution of changes on climate timescales. Similarly flawed are NOAA’s claims that increasing annual counts of billion dollar disasters are in part a consequence of human caused climate change. NOAA’s claims to have achieved detection and attribution are not supported by any scientific analysis that it has performed. Given the importance and influence of the dataset in science and policy, NOAA should act quickly to address this scientific integrity shortfall.

The full submission will soon appear at Nature’s preprint platform, Research Square, and meantime I have provided a link to the PDF.²

My paper is about scientific integrity, and not about the reality or importance of climate change — as I write:

The point here is not to call into question the reality or importance of human-caused climate change – it is real, and it is important.

Just because climate change is important does not mean that scientific claims related to climate change get a free pass or are somehow insulated from scrutiny. To the contrary, it is because of the importance of climate change in policy and politics that scientific claims related to climate change should adhere to the highest scientific standards. Each of us have a responsibility to ensure that happens. Far too often, that is not the case.

You can explore the details of the evaluation in the paper, and here is the bottom line:

The NOAA billion dollar disaster dataset comprehensively falls short of NOAA’s guidelines for scientific integrity. The shortfalls documented here are neither small nor subtle. They represent a significant departure from NOAA’s long-term history of scientific integrity and excellence, which has saved countless lives and facilitated the nation’s economy. A course correction is in order.

I conclude the paper with recommendations for that course correction. This is not rocket science — these recommendations could be implemented easily today or tomorrow.

Here is how the paper ends:

NOAA is a crucially important agency that sits at the intersection of science, policy and politics. It has a long and distinguished history of providing weather, climate, water, ocean and other data to the nation. These data have saved countless lives, supported the economy and supported significant scientific research. The agency is far too important to allow the shortfalls in scientific integrity documented in this paper to persist. The billion dollar disaster dataset is an egregious failure of scientific integrity. However, science and policy are both self-correcting.

You can download a PDF of the submission here. Note that it has not yet gone through peer review. I welcome and invite scrutiny, critique, and comment. Let’s “steel man” this thing. You can comment here at THB, send me an email, or enter your comments at Research Square when the preprint is posted.³

Thanks for reading! THB is reader-supported and so too is the research that results from that support. The “billion dollar disaster” paper is the first paper of mine in which I will formally acknowledge THB subscribers for their support. Assuming it survives the review process, it might also be the first peer-reviewed paper EVER supported by a Substack and acknowledged in print. You helped make that happen. Stay tuned for coming news about the future of THB, which is a bright one, and one that will depend more than ever on reader support. Meantime, please click the little heart at the top, share this post with your favorite reporter on the climate beat, and share also on your favorite social media platform. And keep those comments coming — Honest brokering is a group effort.

1. Disclosure: I was a fellow of a NOAA Cooperative Institute from 2001-2016, and I have great respect for NOAA’s research and operations.

2. I guess I am providing a pre-pre-print?

3. I’ll update this post with the Research Square link when it is eventually posted there.