Billion Dollar Disasters: Not the Highest Standards

NOAA responds to my request for correction, finding serious problems

Last January, upon submission of my paper (since published) — Scientific integrity and U.S. “Billion Dollar Disasters” — I submitted a “request for correction” to the National Oceanic and Atmospheric Administration (NOAA) under the agency’s policies for scientific integrity and information quality.

Today, I heard back from NOAA and they largely agree with my concerns:

“NOAA will undertake actions to improve the documentation and transparency of the data set for greater compliance with NOAA’s Information Quality Guidelines.”

I applaud NOAA for taking my request seriously and, as a result, identifying some serious problems in transparency and information quality.

NOAA agrees that the tabulation “meets the threshold for influential scientific information (ISI) under NOAA Information Quality Guidelines.” This means that the dataset matters for policy, and thus must meet the “highest standards” of procedure and content. NOAA finds that the dataset has not always met those highest standards and is taking steps to remedy.

I share the entire 10 page response at the bottom, but before that, I share some thoughts on NOAA’s response. First, a high-level summary of key points:

• NOAA has failed to fully document its methods for tabulating disaster losses. NOAA admits that its methodology for tabulating losses is unbounded, its methods include a category of losses titled “OTHER” — which could be anything;

• NOAA has systematically failed to disclose its sources, listing in its response more than 20 previously undisclosed sources;

• The “billion dollar disaster” tabulation has not been externally peer-reviewed since 2015, which violates NOAA’s requirement that “influential scientific information” be peer-reviewed at least every 5 years. NOAA is thus initiating a new peer review process;

• NOAA’s internal “robustness check” involves a simple comparison of their estimates to press releases of Aon, Munich Re, and Gallagher Re. NOAA agrees that “these robustness check procedures should be formalized and stated on the Billion Dollar Disaster website”;

• Historical versions of the tabulation (and thus new entries and changes to it) are only available since 2020. This means — remarkably — that 40 of (almost) 44 years of the tabulation lack basic information on the loss estimates and how they were originally compiled;

• NOAA has a tabulation of sub-billion-dollar-events but refuses to share that information;

• NOAA asserts that its claims that the dataset demonstrates the detection and attribution of changes in climate are a matter of policy and not information quality;

These responses and admissions clearly indicate that the NOAA tabulation is not suitable for research. There is no area of science where it would be acceptable to use a dataset of unknown provenance prior to 2020 to study anything from 1980 to 2024. The dataset is a clever public relations gimmick, to be sure, but it should never be used in scientific research, climate assessment reports, or as a grounding for policy.

One part of NOAA’s response illustrates clearly why it is not a good idea to have climatologists doing economics. I explained in my request to NOAA that they claimed incorrectly to be tabulating direct losses from disasters, but in fact, they were also including indirect losses (those removed from the event in time and space) such as business interruption, loss of living quarters, and commodity price changes. This error artificially boosts and mischaracterizes loss estimates.

NOAA in its response simply restated the mistake by citing a 2015 paper by two NOAA climatologists on the economics of “direct losses” from disasters:

. . . regarding indirect costs, the Billion Dollar Disaster data set only relies on direct costs in accordance with the peer reviewed publications. The publications state that “the loss estimates reflect direct effects of weather and climate events (i.e., not including indirect effects) and constitute total losses (i.e., both insured and uninsured).”

NOAA then further quotes from the paper, which asserts that “time element losses (i.e., businesses interruption)” is an example of a direct loss. This is just plain wrong — don’t take it from me, here is the National Association of Insurance Commissioners (emphasis added):

“Examples of indirect loss are loss of use, additional living expenses, and business interruption.”

One funny admission, NOAA reveals that it includes in its tabulation another indirect loss — the “winter feed costs for beef cows” (presumably in the OTHER category). You don’t need to be an economist to know that feed costs for beef cows someplace far removed in place and time from an extreme weather event is not a direct loss.

A distinction between direct and indirect costs may seem like just a detail, but details are supposed to matter when making putatively scientific claims that are highly influential in policy and politics.

NOAA largely avoided discussion of substance — though it did defend its tally for Hurricane Idalia, which I had identified as problematic (at 10x insured losses). But without the input data, this little debate will not be possible to resolve. More generally, until NOAA subjects the dataset to an independent peer review by relevant experts, its many substantive problems are unlikely to be addressed.¹

Finally, on NOAA’s frequent claims to be able use the dataset to detect and attribute human-caused climate change, NOAA replied:

“NOAA appreciates the suggestion to align with the Intergovernmental Panel on Climate Change methods and standards for detection and attribution. Whether further alignment is warranted is a question of policy. This request falls outside the scope of NOAA’s Information Quality Guidelines.”

The detection and attribution of changes in climate is actually a scientific matter, but it is good to have NOAA admit to misalignment with the IPCC and that the agency views such claims as a matter of policy, not information quality. It is a remarkable admission.

I have little doubt that the “billion dollar disaster” dataset will continue to be promoted heavily and serve well as a public relations gimmick irresistible to the climate beat. At the same time, kudos to NOAA for taking a serious look at the tabulation. Based on what NOAA has found, no one should be using the dataset in research or in a scientific assessment — Unless of course the goal is PR, not science.

NOAA’s full 10-page response is below.

Response Billion Dollar Disaster Rfc Letter Ncei Aug 2024

334KB ∙ PDF file

Download

Download

Thanks for reading! Efforts like this exploration of the “billion dollar disaster” tabulation are possible because of your support of THB. Science is self-correcting — sometimes slowly and incrementally — and is only possible if people are willing to challenge claims and show their work. I welcome your comments and questions. Please consider also supporting THB by liking this post (click the heart and it’ll get seen by more Substack readers), subscribing, or, if you are already a subscriber, by upgrading your subscription to support independent research and analysis.

1. All 19 versions of the dataset since 2020 are available — NOAA points us here — for anyone wanting to engage in extended peer review.