The challenge, I think, is effectively managing the nexus of politics and policy in conditions of crisis. There are so many conflicting interests that even an optimal scientific advisory apparatus will (too) often be rendered ineffective.
Using Covid as an example, it's reasonable to think that nobody <wanted> to promote the means to maximize deaths. Even still, there was excess mortality in the U.S. and elsewhere. Managing the economic impacts, however, produced competing interests and likely sub-optimal outcomes due to varying levels of influence in the policy making process.
In other words, while it is a good and necessary thing to look for ways to improve science advisory processes under conditions of crisis, the messiness of the political processes and the eternal competition for resources should lead us to expect results which are too often, uh, uneven.
The question that was not asked nearly often enough during COVID was why did the CDC and WHO abandon their previous advice on both masking and lockdowns? If you look at earlier studies and advice, written prior to the epidemic and after quite a bit of study, they advised that neither masking nor lockdowns were useful. Yet in the panic of the moment, they were recommended and implemented in haste. And as we well know, they really had zero positive effect, and very substantial negative effect, especially on adolescents, which were in fact the people least likely to suffer dangerous consequences from the virus. Worst of all, we were initially told the lockdowns would only be a week or two, which would have had minimal negative effects, but many places were locked down for months. The lesson there is dont abandon analysis made in depth over long term because of knee jerk reactions by politicians. And remember that long term senior civil servants are really politicians. just not elected politicians.
In general, I think it is important to separate "experts" into specialists, generalists, and systems analysts. Most experts are specialists. That is where the paying jobs are. They see a narrow slice of the big picture. Generalists see the big picture, but usually they are retired (there are few paying jobs for generalists) and only they comment from afar. System analysts represent a very small percentage of the total and they attempt to analyze and project the future. They are usually tied to gigantic computer programs based on assumptions that are highly speculative. So, be very careful of which experts you are hearing from.
Roger: I once counted up the federal government employees for health and disease control to be about 10,000. During the height of the Covid pandemic, the information released mas minimal and often misleading, and to this day, I would say that I learned very little from the government. The six-foot rule rests on what? And it totally neglects duration of exposure. It is simple logic to believe that six feet at one hour is far worse than 1 foot for five seconds. Masks! To this day we don’t have a clues how effective different types of masks are in different circumstances. Vaccinations! How effective were they against which variants, and for how long? Clearly, they were over-hyped from the beginning. And then the claims of protection were obviously skewed toward effectiveness of vaccines, just like news releases on climate are skewed toward alarmism. Who was dying? The demographics was difficult to discern, but I had the impression that if you subtracted off the deaths from nursing homes, the pandemic didn’t seem quite so bad for the rest of the population. The effectiveness of shutdowns? There seems to be evidence that the cost/benefit was very unfavorable but the 10,000 employees haven’t done squat to clarify that. Ancillary factors: washing your hands (like we’ve been told a hundred times): is that useful at all? As I look back on the past three years, I can’t find hardly anything useful in the form of expert advice regarding Covid, and all told, I would give the government a grade of F-minus
The insularity of government agencies that persist in dubious agendas while ignoring shadow activities outside of their limited world view was clearly dominant in the Covid era. But it is not limited to Covid. There are shadow groups (blogs) regarding climate change. While these are all politically biased (so are the government programs) they do obviously include some smart people with system analysis expertise which seems to be lacking in government? The government pretends that these are invisible and blithely persists, while ignoring everything in the shadow world. In an area of my expertise (human missions to Moon or Mars) the government is also insular and ignores anything contrary outside their sphere. My book “Human Missions to Mars” third edition recently published in January, demonstrates fallacies in government thinking that will never be considered by NASA.
The main things that distinguish the primary attitude of government agencies is insularity, near-sightedness, and not invented here.
Important subject....but I too was stunned by the recommendation that essentially 'experts' just need to be more expert and to do their jobs better. Re covid ,the CDC, NIH and hundreds if not thousands of state and local health agencies, massively populated with 'experts' and 'scientists' made a hash of it. It is clear from personal discussions with a number of those employed within these expert and scientific communities that many disagreed with the 'expert consensus' but were not able or willing to express their concerns. So too with climate. I don't know what the solution might be but I failed to see one in your piece. S Wilson
I’ve been interested in the idea that evidence-based public policy can help us transcend policy dictated by political interests. But so far, it looks more like at best we have evidence-influenced politics.
So, there are experts in various scientific disciplines who can be better prepared to inform policy makers. That’s one thing. But is there any chance of transitioning from scientific information used as a tool for politicians to enhance their power, toward an actual science of public policy making?
I agree with others, that “science advisory mechanisms” such as they are don’t function particularly well. I think the fundamental question is “why do you think you need “science” advice separate from other kinds of advice?” Because ultimately it is one element in the mix, and isn’t how the mix is conducted the key element of policy? And there is an imbedded privilege/authority of raising science advice over other kinds of advice (economic, what could go wrong with that?) plus the fact that Roger and others have pointed out that the scientific communities have their own disagreements within and among disciplines, and often reflect only one political perspective, whereas people in our country are close to half and half with another political perspective.
So before we rate science advice I think we ought to discuss and articulate what it is we want from science advice, which can’t be separate from determining how it should be developed and used with other information. Because I think people can sense when political views are imbedded in that advice and certainly practitioners know how reality-based it is.
I had to read this 3 times to make sure I wasn't suffering from brain fog. Are you serious?
"In general, science advisory mechanisms function well on topics such as vaccine approval, climate change........ "
Do you really think that the vaccine approval process for Covid functioned well? For young people/children? Boosters for those with natural immunity? Prioritizing early doses? How about other aspects of the Covid response (which by the way you teased upcoming content on this about a year ago)?
Do you really think that the science advisory mechanisms are working well for climate policy? Policy makers find scientists (in and out of government) to tell them what they want to hear. As you've pointed out many scientists advising on policy lack integrity and have financial and reputational conflicts of interest.
Do you really believe that having more permanent unaccountable government bureaucrats charged with providing science advice to policy makers who ultimately decide their professional fates is a net plus? Is the State Department and diplomacy really a model to be followed when many have criticized it as an echo chamber?
Mar 7, 2023·edited Mar 7, 2023Liked by Roger Pielke Jr.
Rather than how to give advice better, the real question is how to give better advice (Better leaders would be a big help, but that's a whole other bucket of poo.). One of the major causes of the Covid fiasco (the response as opposed to the pandemic) was that the advice was given by hedgehogs, to use Tetlock's analogy. Their focus was on response, but the real focus should always be on recovery. During and after Katrina, law enforcement and other first responders stopped out of state businesses from coming in to help with clean up and recovery. Their intent was laudable - keep out the hucksters and the flim-flammers. But NOLA's recovery was slower than it needed to be because of a lack of construction professionals. In Mississippi, since out-of-state handlers and haulers weren't allowed in, the downed timber caused a new forest fire literally every day.
Comprehensive stress tests might be a good way to improve both advice and advice-giving, leading to both better responses and better recoveries. I've written about this (resilientcommunities.home.blog/2021/01/07/rising-after-the-fall/). In short, a group - perhaps coming from the group in your first recommendation - develops a scenario for some sort of crisis that will require a massive national response. The scenario should be realistic not worst case. They might also identify what recovery should look like, in concrete terms. Then let the experts (we need to include the states in this) have at it. Their job is to identify a realistic path through the crisis to the recovery goal. If one agency takes the lead in response (e.g., CDC) then they're forced to look at the consequences of their actions through the other agencies' eyes.
Stress testing of this type offers some real positives:
• It is based on the risks the nation actually faces.
• It uses all of the nation’s expertise and knowledge.
• It provides a time to recover based on the resources actually available to the nation.
• It indicates opportunities for mitigating action to reduce the time to recovery.
• It avoids the trap of having responders from different agencies exchanging business cards in the midst of crisis.
• Not least, it breaches the bureaucratic bastions and forces the hedgehogs to think more like foxes!
"In general science advisory mechanisms function well on topics such as ...climate change.." I find it surprising you would say this Roger. I strikes me that the various conferences where decision makers gather are prone to issuing apocalyptic statements and these statements seem to drive target setting that doesn't adequately balance costs and benefits. Doesn't that imply that the advisory mechanism is not functioning adequately? Could you comment?
A difficult aspect of generating effective policy is that it requires subjective values judgements. Scientific discipline advice is often relevant to our choices, but most experts don't really have expertise in the intersecting issues. e.g., what are the impacts of dietary changes to fight climate change on human nutrition? What are the impacts of coronavirus masks on child learning and socialization? Too often, scientists who have a moral goal in the mix make too many flawed assumptions about the intersecting issues they know little about. e.g., they might claim that masks don't inhibit communication, or that phasing out meat will only improve human nutrition. This is why it is unwise for elected officials to pass relevant policy decisions solely to domain experts.
I often wonder if Roger feels his efforts border on Sisyphean. Over the last 20 years I've viewed various crises (e.g., Great Financial Crisis, COVID-19, climate) more objectively since taking his science and policy course in 2002. From my perspective it appears that political power as well as the politization of science have only increased.
Interesting stuff. However, many of the suggestions for how previous failures might be remedied in the future, depend upon a retrospective view of what the failures and successes were in relation to one particular event - the global Covid pandemic. Such retrospective reflections are of course valuable in thinking about how we might respond in the future. But the future is unknowable. The particular combination of emergency, scientific knowledge, political configurations both national and international, distribution of expertise, and so on, is unlikely ever to be repeated. That is not counsel of despair, but realism.
One suggestion for improvement is that we might try to make candid admission of failure and misjudgement less politically costly. At the moment, retrospective judgement of failure is treated with a sanctimonious ferocity. This makes passing the buck more important, for many actors, than learning from complex situations. How do we change that?
The challenge, I think, is effectively managing the nexus of politics and policy in conditions of crisis. There are so many conflicting interests that even an optimal scientific advisory apparatus will (too) often be rendered ineffective.
Using Covid as an example, it's reasonable to think that nobody <wanted> to promote the means to maximize deaths. Even still, there was excess mortality in the U.S. and elsewhere. Managing the economic impacts, however, produced competing interests and likely sub-optimal outcomes due to varying levels of influence in the policy making process.
In other words, while it is a good and necessary thing to look for ways to improve science advisory processes under conditions of crisis, the messiness of the political processes and the eternal competition for resources should lead us to expect results which are too often, uh, uneven.
Great comments and discussion! I'll work my way through with some responses as I catch up . . . Thanks all
The question that was not asked nearly often enough during COVID was why did the CDC and WHO abandon their previous advice on both masking and lockdowns? If you look at earlier studies and advice, written prior to the epidemic and after quite a bit of study, they advised that neither masking nor lockdowns were useful. Yet in the panic of the moment, they were recommended and implemented in haste. And as we well know, they really had zero positive effect, and very substantial negative effect, especially on adolescents, which were in fact the people least likely to suffer dangerous consequences from the virus. Worst of all, we were initially told the lockdowns would only be a week or two, which would have had minimal negative effects, but many places were locked down for months. The lesson there is dont abandon analysis made in depth over long term because of knee jerk reactions by politicians. And remember that long term senior civil servants are really politicians. just not elected politicians.
In general, I think it is important to separate "experts" into specialists, generalists, and systems analysts. Most experts are specialists. That is where the paying jobs are. They see a narrow slice of the big picture. Generalists see the big picture, but usually they are retired (there are few paying jobs for generalists) and only they comment from afar. System analysts represent a very small percentage of the total and they attempt to analyze and project the future. They are usually tied to gigantic computer programs based on assumptions that are highly speculative. So, be very careful of which experts you are hearing from.
Roger: I once counted up the federal government employees for health and disease control to be about 10,000. During the height of the Covid pandemic, the information released mas minimal and often misleading, and to this day, I would say that I learned very little from the government. The six-foot rule rests on what? And it totally neglects duration of exposure. It is simple logic to believe that six feet at one hour is far worse than 1 foot for five seconds. Masks! To this day we don’t have a clues how effective different types of masks are in different circumstances. Vaccinations! How effective were they against which variants, and for how long? Clearly, they were over-hyped from the beginning. And then the claims of protection were obviously skewed toward effectiveness of vaccines, just like news releases on climate are skewed toward alarmism. Who was dying? The demographics was difficult to discern, but I had the impression that if you subtracted off the deaths from nursing homes, the pandemic didn’t seem quite so bad for the rest of the population. The effectiveness of shutdowns? There seems to be evidence that the cost/benefit was very unfavorable but the 10,000 employees haven’t done squat to clarify that. Ancillary factors: washing your hands (like we’ve been told a hundred times): is that useful at all? As I look back on the past three years, I can’t find hardly anything useful in the form of expert advice regarding Covid, and all told, I would give the government a grade of F-minus
The insularity of government agencies that persist in dubious agendas while ignoring shadow activities outside of their limited world view was clearly dominant in the Covid era. But it is not limited to Covid. There are shadow groups (blogs) regarding climate change. While these are all politically biased (so are the government programs) they do obviously include some smart people with system analysis expertise which seems to be lacking in government? The government pretends that these are invisible and blithely persists, while ignoring everything in the shadow world. In an area of my expertise (human missions to Moon or Mars) the government is also insular and ignores anything contrary outside their sphere. My book “Human Missions to Mars” third edition recently published in January, demonstrates fallacies in government thinking that will never be considered by NASA.
The main things that distinguish the primary attitude of government agencies is insularity, near-sightedness, and not invented here.
Dr Pielke
Important subject....but I too was stunned by the recommendation that essentially 'experts' just need to be more expert and to do their jobs better. Re covid ,the CDC, NIH and hundreds if not thousands of state and local health agencies, massively populated with 'experts' and 'scientists' made a hash of it. It is clear from personal discussions with a number of those employed within these expert and scientific communities that many disagreed with the 'expert consensus' but were not able or willing to express their concerns. So too with climate. I don't know what the solution might be but I failed to see one in your piece. S Wilson
Hi Roger,
I’ve been interested in the idea that evidence-based public policy can help us transcend policy dictated by political interests. But so far, it looks more like at best we have evidence-influenced politics.
So, there are experts in various scientific disciplines who can be better prepared to inform policy makers. That’s one thing. But is there any chance of transitioning from scientific information used as a tool for politicians to enhance their power, toward an actual science of public policy making?
I agree with others, that “science advisory mechanisms” such as they are don’t function particularly well. I think the fundamental question is “why do you think you need “science” advice separate from other kinds of advice?” Because ultimately it is one element in the mix, and isn’t how the mix is conducted the key element of policy? And there is an imbedded privilege/authority of raising science advice over other kinds of advice (economic, what could go wrong with that?) plus the fact that Roger and others have pointed out that the scientific communities have their own disagreements within and among disciplines, and often reflect only one political perspective, whereas people in our country are close to half and half with another political perspective.
So before we rate science advice I think we ought to discuss and articulate what it is we want from science advice, which can’t be separate from determining how it should be developed and used with other information. Because I think people can sense when political views are imbedded in that advice and certainly practitioners know how reality-based it is.
I had to read this 3 times to make sure I wasn't suffering from brain fog. Are you serious?
"In general, science advisory mechanisms function well on topics such as vaccine approval, climate change........ "
Do you really think that the vaccine approval process for Covid functioned well? For young people/children? Boosters for those with natural immunity? Prioritizing early doses? How about other aspects of the Covid response (which by the way you teased upcoming content on this about a year ago)?
Do you really think that the science advisory mechanisms are working well for climate policy? Policy makers find scientists (in and out of government) to tell them what they want to hear. As you've pointed out many scientists advising on policy lack integrity and have financial and reputational conflicts of interest.
Do you really believe that having more permanent unaccountable government bureaucrats charged with providing science advice to policy makers who ultimately decide their professional fates is a net plus? Is the State Department and diplomacy really a model to be followed when many have criticized it as an echo chamber?
Rather than how to give advice better, the real question is how to give better advice (Better leaders would be a big help, but that's a whole other bucket of poo.). One of the major causes of the Covid fiasco (the response as opposed to the pandemic) was that the advice was given by hedgehogs, to use Tetlock's analogy. Their focus was on response, but the real focus should always be on recovery. During and after Katrina, law enforcement and other first responders stopped out of state businesses from coming in to help with clean up and recovery. Their intent was laudable - keep out the hucksters and the flim-flammers. But NOLA's recovery was slower than it needed to be because of a lack of construction professionals. In Mississippi, since out-of-state handlers and haulers weren't allowed in, the downed timber caused a new forest fire literally every day.
Comprehensive stress tests might be a good way to improve both advice and advice-giving, leading to both better responses and better recoveries. I've written about this (resilientcommunities.home.blog/2021/01/07/rising-after-the-fall/). In short, a group - perhaps coming from the group in your first recommendation - develops a scenario for some sort of crisis that will require a massive national response. The scenario should be realistic not worst case. They might also identify what recovery should look like, in concrete terms. Then let the experts (we need to include the states in this) have at it. Their job is to identify a realistic path through the crisis to the recovery goal. If one agency takes the lead in response (e.g., CDC) then they're forced to look at the consequences of their actions through the other agencies' eyes.
Stress testing of this type offers some real positives:
• It is based on the risks the nation actually faces.
• It uses all of the nation’s expertise and knowledge.
• It provides a time to recover based on the resources actually available to the nation.
• It indicates opportunities for mitigating action to reduce the time to recovery.
• It avoids the trap of having responders from different agencies exchanging business cards in the midst of crisis.
• Not least, it breaches the bureaucratic bastions and forces the hedgehogs to think more like foxes!
"In general science advisory mechanisms function well on topics such as ...climate change.." I find it surprising you would say this Roger. I strikes me that the various conferences where decision makers gather are prone to issuing apocalyptic statements and these statements seem to drive target setting that doesn't adequately balance costs and benefits. Doesn't that imply that the advisory mechanism is not functioning adequately? Could you comment?
A difficult aspect of generating effective policy is that it requires subjective values judgements. Scientific discipline advice is often relevant to our choices, but most experts don't really have expertise in the intersecting issues. e.g., what are the impacts of dietary changes to fight climate change on human nutrition? What are the impacts of coronavirus masks on child learning and socialization? Too often, scientists who have a moral goal in the mix make too many flawed assumptions about the intersecting issues they know little about. e.g., they might claim that masks don't inhibit communication, or that phasing out meat will only improve human nutrition. This is why it is unwise for elected officials to pass relevant policy decisions solely to domain experts.
I often wonder if Roger feels his efforts border on Sisyphean. Over the last 20 years I've viewed various crises (e.g., Great Financial Crisis, COVID-19, climate) more objectively since taking his science and policy course in 2002. From my perspective it appears that political power as well as the politization of science have only increased.
Just stumbled across one of your experts about expert advice and how contributing research papers can be improved. Quite interesting. I’ve not thought much about this stuff https://www.degruyter.com/document/doi/10.1515/peps-2023-0002/html
Interesting stuff. However, many of the suggestions for how previous failures might be remedied in the future, depend upon a retrospective view of what the failures and successes were in relation to one particular event - the global Covid pandemic. Such retrospective reflections are of course valuable in thinking about how we might respond in the future. But the future is unknowable. The particular combination of emergency, scientific knowledge, political configurations both national and international, distribution of expertise, and so on, is unlikely ever to be repeated. That is not counsel of despair, but realism.
One suggestion for improvement is that we might try to make candid admission of failure and misjudgement less politically costly. At the moment, retrospective judgement of failure is treated with a sanctimonious ferocity. This makes passing the buck more important, for many actors, than learning from complex situations. How do we change that?