The challenge, I think, is effectively managing the nexus of politics and policy in conditions of crisis. There are so many conflicting interests that even an optimal scientific advisory apparatus will (too) often be rendered ineffective.
Using Covid as an example, it's reasonable to think that nobody <wanted> to promote the means to maximize deaths. Even still, there was excess mortality in the U.S. and elsewhere. Managing the economic impacts, however, produced competing interests and likely sub-optimal outcomes due to varying levels of influence in the policy making process.
In other words, while it is a good and necessary thing to look for ways to improve science advisory processes under conditions of crisis, the messiness of the political processes and the eternal competition for resources should lead us to expect results which are too often, uh, uneven.
The question that was not asked nearly often enough during COVID was why did the CDC and WHO abandon their previous advice on both masking and lockdowns? If you look at earlier studies and advice, written prior to the epidemic and after quite a bit of study, they advised that neither masking nor lockdowns were useful. Yet in the panic of the moment, they were recommended and implemented in haste. And as we well know, they really had zero positive effect, and very substantial negative effect, especially on adolescents, which were in fact the people least likely to suffer dangerous consequences from the virus. Worst of all, we were initially told the lockdowns would only be a week or two, which would have had minimal negative effects, but many places were locked down for months. The lesson there is dont abandon analysis made in depth over long term because of knee jerk reactions by politicians. And remember that long term senior civil servants are really politicians. just not elected politicians.
In general, I think it is important to separate "experts" into specialists, generalists, and systems analysts. Most experts are specialists. That is where the paying jobs are. They see a narrow slice of the big picture. Generalists see the big picture, but usually they are retired (there are few paying jobs for generalists) and only they comment from afar. System analysts represent a very small percentage of the total and they attempt to analyze and project the future. They are usually tied to gigantic computer programs based on assumptions that are highly speculative. So, be very careful of which experts you are hearing from.
Roger: I once counted up the federal government employees for health and disease control to be about 10,000. During the height of the Covid pandemic, the information released mas minimal and often misleading, and to this day, I would say that I learned very little from the government. The six-foot rule rests on what? And it totally neglects duration of exposure. It is simple logic to believe that six feet at one hour is far worse than 1 foot for five seconds. Masks! To this day we don’t have a clues how effective different types of masks are in different circumstances. Vaccinations! How effective were they against which variants, and for how long? Clearly, they were over-hyped from the beginning. And then the claims of protection were obviously skewed toward effectiveness of vaccines, just like news releases on climate are skewed toward alarmism. Who was dying? The demographics was difficult to discern, but I had the impression that if you subtracted off the deaths from nursing homes, the pandemic didn’t seem quite so bad for the rest of the population. The effectiveness of shutdowns? There seems to be evidence that the cost/benefit was very unfavorable but the 10,000 employees haven’t done squat to clarify that. Ancillary factors: washing your hands (like we’ve been told a hundred times): is that useful at all? As I look back on the past three years, I can’t find hardly anything useful in the form of expert advice regarding Covid, and all told, I would give the government a grade of F-minus
The insularity of government agencies that persist in dubious agendas while ignoring shadow activities outside of their limited world view was clearly dominant in the Covid era. But it is not limited to Covid. There are shadow groups (blogs) regarding climate change. While these are all politically biased (so are the government programs) they do obviously include some smart people with system analysis expertise which seems to be lacking in government? The government pretends that these are invisible and blithely persists, while ignoring everything in the shadow world. In an area of my expertise (human missions to Moon or Mars) the government is also insular and ignores anything contrary outside their sphere. My book “Human Missions to Mars” third edition recently published in January, demonstrates fallacies in government thinking that will never be considered by NASA.
The main things that distinguish the primary attitude of government agencies is insularity, near-sightedness, and not invented here.
The US approach to science advice in the pandemic was extremely poor. I have not written up an evaluation (yet) but I have given some talks on it, and there is not a lot to praise here.
On this, I would add that as far as I know, at no time did the CDC advise people to boost their immune system - get out in the sun, take Vitamin D and so on. And yet we knew very early on that it was the immuno-compromised that were the most susceptible to the virus. Why was this not communicated to the public and factored into the advice on lockdowns?
Important subject....but I too was stunned by the recommendation that essentially 'experts' just need to be more expert and to do their jobs better. Re covid ,the CDC, NIH and hundreds if not thousands of state and local health agencies, massively populated with 'experts' and 'scientists' made a hash of it. It is clear from personal discussions with a number of those employed within these expert and scientific communities that many disagreed with the 'expert consensus' but were not able or willing to express their concerns. So too with climate. I don't know what the solution might be but I failed to see one in your piece. S Wilson
Thanks, on the US, I completely agree. The EScAPE project looked at more than a dozen countries and the effectiveness of science advisory mechanisms varied across them. Of course, it is important to distinguish advice from decisions and their outcomes -- we focused on advisory mechanisms, and not the ultimate COVID outcomes informed (or not) by advice.
Steve: In sensitive matters, the government will usually not allow individuals to voice opinions, no matter how 'expert" they are. The party line is administered from above. A couple of examples from NASA: (1) There are two ways to do a human mission to Mars: short stay and long stay. The system analysts long since gave up on short stay because it required nuclear propulsion (politically very difficult), required gravity assist from Venus on return flight (heat and radiation) and the total return on investment was low (plant flag and run). But in 2019, the Administrator told the analysts that short stay was the new front runner. The analysts reluctantly had to align like weather vanes. (2) NASA is developing recycling systems for waste gas, liquids and solids for deep space. The program for years has concentrated on increasing percent closure, while neglecting reliability. One maverick in NASA spent years pointing out that 90% closure and 95% reliability is far more important than 95% closure and 90% reliability. The NASA program goes on as planned. Impressively, the maverick is still employed - because he presents no danger - those who run the program don't listen to him.
I’ve been interested in the idea that evidence-based public policy can help us transcend policy dictated by political interests. But so far, it looks more like at best we have evidence-influenced politics.
So, there are experts in various scientific disciplines who can be better prepared to inform policy makers. That’s one thing. But is there any chance of transitioning from scientific information used as a tool for politicians to enhance their power, toward an actual science of public policy making?
My view is that "evidence-influenced politics" is probably the best we can ever hope for, and in democracies, that is probably appropriate. That said, we still want robust evidence and not evidence curated or invented to serve political ends.
I agree with others, that “science advisory mechanisms” such as they are don’t function particularly well. I think the fundamental question is “why do you think you need “science” advice separate from other kinds of advice?” Because ultimately it is one element in the mix, and isn’t how the mix is conducted the key element of policy? And there is an imbedded privilege/authority of raising science advice over other kinds of advice (economic, what could go wrong with that?) plus the fact that Roger and others have pointed out that the scientific communities have their own disagreements within and among disciplines, and often reflect only one political perspective, whereas people in our country are close to half and half with another political perspective.
So before we rate science advice I think we ought to discuss and articulate what it is we want from science advice, which can’t be separate from determining how it should be developed and used with other information. Because I think people can sense when political views are imbedded in that advice and certainly practitioners know how reality-based it is.
I had to read this 3 times to make sure I wasn't suffering from brain fog. Are you serious?
"In general, science advisory mechanisms function well on topics such as vaccine approval, climate change........ "
Do you really think that the vaccine approval process for Covid functioned well? For young people/children? Boosters for those with natural immunity? Prioritizing early doses? How about other aspects of the Covid response (which by the way you teased upcoming content on this about a year ago)?
Do you really think that the science advisory mechanisms are working well for climate policy? Policy makers find scientists (in and out of government) to tell them what they want to hear. As you've pointed out many scientists advising on policy lack integrity and have financial and reputational conflicts of interest.
Do you really believe that having more permanent unaccountable government bureaucrats charged with providing science advice to policy makers who ultimately decide their professional fates is a net plus? Is the State Department and diplomacy really a model to be followed when many have criticized it as an echo chamber?
Some science advisory mechanisms work better than others, to be sure. That is why we undertook this comparative assessment. The US stands out as a poor performer on science advice in the pandemic (under DT & JB, IMO), to be sure. Science advisory mechanisms are imperfect and difficult to implement (like diplomacy), but what is the alternative?
The alternative is to elect people to leadership positions that have the intellect, energy, integrity and courage to solicit the best available advice and to work hard to understand it so that they can make responsible decisions. Neither DT nor JB fit this mold. While I am sure you are not a DeSantis fan, his model for obtaining expert advice and implementing thoughtful policies during Covid is the best model that I can think of.
Well, I can't argue with this list of characteristics! As a policy scholar I often tell my students that we operate in conditions that are not the ones we wish we had but the ones that we get. And in democracies that includes the elected officials present at he time that we are offering advice, evidence and the like.
Some of my colleagues believe that we academics should spend more time politicking for our favorite candidates, I'm more of the view that whomever is elected is going to need expertise, so some of us should spend more time on research and advice, and less worrying about elections. But I know this can be unpopular!
Don't take this the wrong way, but you really should find a way to get out of the academic bubble that you have lived and worked in all of your life. Viewing the world from the vantage point of the academic ivory tower can't be good for your health. You're only in your early fifties, there's till time!
Mar 7, 2023·edited Mar 7, 2023Liked by Roger Pielke Jr.
Rather than how to give advice better, the real question is how to give better advice (Better leaders would be a big help, but that's a whole other bucket of poo.). One of the major causes of the Covid fiasco (the response as opposed to the pandemic) was that the advice was given by hedgehogs, to use Tetlock's analogy. Their focus was on response, but the real focus should always be on recovery. During and after Katrina, law enforcement and other first responders stopped out of state businesses from coming in to help with clean up and recovery. Their intent was laudable - keep out the hucksters and the flim-flammers. But NOLA's recovery was slower than it needed to be because of a lack of construction professionals. In Mississippi, since out-of-state handlers and haulers weren't allowed in, the downed timber caused a new forest fire literally every day.
Comprehensive stress tests might be a good way to improve both advice and advice-giving, leading to both better responses and better recoveries. I've written about this (resilientcommunities.home.blog/2021/01/07/rising-after-the-fall/). In short, a group - perhaps coming from the group in your first recommendation - develops a scenario for some sort of crisis that will require a massive national response. The scenario should be realistic not worst case. They might also identify what recovery should look like, in concrete terms. Then let the experts (we need to include the states in this) have at it. Their job is to identify a realistic path through the crisis to the recovery goal. If one agency takes the lead in response (e.g., CDC) then they're forced to look at the consequences of their actions through the other agencies' eyes.
Stress testing of this type offers some real positives:
• It is based on the risks the nation actually faces.
• It uses all of the nation’s expertise and knowledge.
• It provides a time to recover based on the resources actually available to the nation.
• It indicates opportunities for mitigating action to reduce the time to recovery.
• It avoids the trap of having responders from different agencies exchanging business cards in the midst of crisis.
• Not least, it breaches the bureaucratic bastions and forces the hedgehogs to think more like foxes!
I think John brings up an important point.. anytime “science” advice is separated from “advice by people who actually do the work” there is a problem. To change topics, I recently was on a webinar with climate guy Stanford Mark Jacobson, who firmly believes there are no practical problems with building out industrial wind and solar today... they can be built in 1-3 years.. well not on federal land, and not without changes to regulatory regimes. Out of his view comes policy direction (and many news stories as I was on the webinar for reporters).. One of the structural problems with “science” advisory committees is that there is no place for practitioners in the dialogue.
Jacobson is an academic, involved in a non-profit that is sort of a marketeer for renewables. There is a lot out there debunking some of his claims, but he's a near-perfect example of what you get from a hedgehog. Yes, you could potentially stand up wind and solar generators in a few years. However, California's continuing problems with brownouts and rolling blackouts are dark testimony [intentional bad pun] that he is ignoring the fragility this introduces into the California power grid. I.e., just because you "can" doesn't mean it's a good idea. In short, for a renewable-intensive grid, we still haven't figured out how to reliably provide electric power when physical conditions such as those in CA in August 2020 lead to energy demands that intermittents can't fill. In other words, Jacobson - as a hedgehog - seems to provide good information about the renewables, but doesn't really have the experience or knowledge to be able to say how well these can be integrated into a power system for a public that expects consistent electricity for their homes and businesses. His analyses of the CO2 emissions of renewables vs nuclear are also flawed in my opinion - but look at the assumptions and then decide for yourself.
Not to beat this horse any deader (?), CA's experience in moving toward 100% renewables by 2050 is a good example of why something like my suggestion is useful. Their current "plan" for the transition is sort of like
• We'll mandate the increasing use of renewables;
• We'll ban the use of everything else (of course ignoring lots of other side effects, e.g., low income families will have to do expensive retrofits converting from natural gas to electric, AND pay more for the electricity per month than they did natural gas);
• And somewhere between now and then a couple of miracles will occur - affordable MW-scale batteries will become available AND the engineers will figure out how to handle all of the intermittency issues.
Governor Newsom's recent decision not to close CA's only surviving nuclear plant is a tacit admission that the "transition" plan wasn't (a plan, that is).
This is just another case where academics (whose views are privileged in this society) don't ask questions like "what do people think who work with this every day?" and "what do people think who live in the places where buildout occurs?".
I wonder if part of this is unclear linkages between practitioners and academics at places like Stanford (not so much in ag because of the land grants) and the continuing bias against social sciences.
I agree with this 100%. IMO this is a big reason why expert advice on technologies is often more effective than expert advice on policy. Whether or not a rocket can go to Mars is a lot easier to make expert judgments on than, say, what are options for responding to an emerging pandemic?
"In general science advisory mechanisms function well on topics such as ...climate change.." I find it surprising you would say this Roger. I strikes me that the various conferences where decision makers gather are prone to issuing apocalyptic statements and these statements seem to drive target setting that doesn't adequately balance costs and benefits. Doesn't that imply that the advisory mechanism is not functioning adequately? Could you comment?
The IPCC and US NCA are of course imperfect institutions, and as I often say, if they did not exist they would have to be invented. They work well in the sense that they have clear mandates, processes and can be improved. It is also important to distinguish process from substance ... the IPCC has really done a poor job on scenarios in its most recent assessments, but it did much bettter in the past with SRES. The fact that we can make such judgments and seek to course correct is a strength of science advisory bodies.
That's what I was looking for. Thanks. Although few would say CO2 emissions have zero effect on temperature and, through that, climate, there is a very wide range of informed opinion as to how significant an issue this is and the level of disruption justified to "address it". I have the impression that those who have the most influence within the IPCC don't adequately represent the full range of opinion on this in that they tilt to the alarmist side of the continuum. The result is a widely accepted but economically damaging set policies that, to many in the public, defy commonsense. Even if politicians are onside with the public the fact that the IPCC is assumed to be the final arbitrator of "the science" makes dissent "anti-science" by definition. I wish I had the answer but am convinced we're headed in the wrong direction on the climate issue and the very existence of the IPCC is partly responsible.
But do those actually give "science advice?" As a person in a subfield I might use their views on say whether rain will increase, but how that translates into what I do is a function of my own discipline's choices... say a water manager vs. a wildlife biologist vs. transportation planners. I would call the IPCC and the NCA as "research roundups" or meta-reviews.
A difficult aspect of generating effective policy is that it requires subjective values judgements. Scientific discipline advice is often relevant to our choices, but most experts don't really have expertise in the intersecting issues. e.g., what are the impacts of dietary changes to fight climate change on human nutrition? What are the impacts of coronavirus masks on child learning and socialization? Too often, scientists who have a moral goal in the mix make too many flawed assumptions about the intersecting issues they know little about. e.g., they might claim that masks don't inhibit communication, or that phasing out meat will only improve human nutrition. This is why it is unwise for elected officials to pass relevant policy decisions solely to domain experts.
I think that "scientists" seldom interact directly with policy decision makers. Instead they interact with their managers who interact with their department heads who interact with their division heads who interact with their agency heads who interact with the administration science advisor who probably gets 10 minutes with the decision maker.
I often wonder if Roger feels his efforts border on Sisyphean. Over the last 20 years I've viewed various crises (e.g., Great Financial Crisis, COVID-19, climate) more objectively since taking his science and policy course in 2002. From my perspective it appears that political power as well as the politization of science have only increased.
Interesting stuff. However, many of the suggestions for how previous failures might be remedied in the future, depend upon a retrospective view of what the failures and successes were in relation to one particular event - the global Covid pandemic. Such retrospective reflections are of course valuable in thinking about how we might respond in the future. But the future is unknowable. The particular combination of emergency, scientific knowledge, political configurations both national and international, distribution of expertise, and so on, is unlikely ever to be repeated. That is not counsel of despair, but realism.
One suggestion for improvement is that we might try to make candid admission of failure and misjudgement less politically costly. At the moment, retrospective judgement of failure is treated with a sanctimonious ferocity. This makes passing the buck more important, for many actors, than learning from complex situations. How do we change that?
Exactly, we have an ‘image is everything’ political culture, this could be a leverage point for systemic change.
I’m a pilot. The aviation industry has wholeheartedly adopted a just culture, with safety and error reporting encouraged with no jeopardy, the first thing you do when you make a mistake is own up to it. The details are then used in an evidence based training system so others can benefit from lessons learned. It’s so obviously simple and optimised for the good of the system as a whole, it is extremely frustrating to see the blatant lies and coverups by some of our leaders. Imagine if the travelling public thought the same of us as pilots, the evidence speaks for itself.
We wouldn't be flying if the aviation industry operated the same way.. what is this telling the public.. "your government really cares about aviation and wildfire improvements and safety- public health.. not so much."
I agree with Sharon F. (below). We also do this well on NTSB investigations and also, at least in the past, on post-disaster assessments. I agree that judgments of failure are often not welcomed. Just look at the COVID origins issue . . .
Yes, to me the academic question is "how did the NTSB and the wildland fire community and others design a safe space for failure and how can it be replicated?" I don't know how the committee on the Challenger would it into this.
The wildland fire community has managed to do that.. for example they have an entire lessons learned website https://www.wildfirelessons.net/home. Another example, last year prescribed burns got out of control and did substantial amounts of damage in New Mexico. The Forest Service did a stand-down on prescribed burning and did not start until a serious review had been conducted (with experts in the public) and the ensuing recommendations were in place. Think about that approach, then think about errors in gain of function research. Academics could study how that culture (wildland fire) came to be, and why that kind of continual improvement does not happen in other disciplines/areas. I think if you asked many people it would be “political polarization” but wildfire folks work under a variety of state and federal executives.. so...
The challenge, I think, is effectively managing the nexus of politics and policy in conditions of crisis. There are so many conflicting interests that even an optimal scientific advisory apparatus will (too) often be rendered ineffective.
Using Covid as an example, it's reasonable to think that nobody <wanted> to promote the means to maximize deaths. Even still, there was excess mortality in the U.S. and elsewhere. Managing the economic impacts, however, produced competing interests and likely sub-optimal outcomes due to varying levels of influence in the policy making process.
In other words, while it is a good and necessary thing to look for ways to improve science advisory processes under conditions of crisis, the messiness of the political processes and the eternal competition for resources should lead us to expect results which are too often, uh, uneven.
Great comments and discussion! I'll work my way through with some responses as I catch up . . . Thanks all
The question that was not asked nearly often enough during COVID was why did the CDC and WHO abandon their previous advice on both masking and lockdowns? If you look at earlier studies and advice, written prior to the epidemic and after quite a bit of study, they advised that neither masking nor lockdowns were useful. Yet in the panic of the moment, they were recommended and implemented in haste. And as we well know, they really had zero positive effect, and very substantial negative effect, especially on adolescents, which were in fact the people least likely to suffer dangerous consequences from the virus. Worst of all, we were initially told the lockdowns would only be a week or two, which would have had minimal negative effects, but many places were locked down for months. The lesson there is dont abandon analysis made in depth over long term because of knee jerk reactions by politicians. And remember that long term senior civil servants are really politicians. just not elected politicians.
Neither CDC nor WHO get high marks for science advisory mechanisms in the pandemic, my view.
In general, I think it is important to separate "experts" into specialists, generalists, and systems analysts. Most experts are specialists. That is where the paying jobs are. They see a narrow slice of the big picture. Generalists see the big picture, but usually they are retired (there are few paying jobs for generalists) and only they comment from afar. System analysts represent a very small percentage of the total and they attempt to analyze and project the future. They are usually tied to gigantic computer programs based on assumptions that are highly speculative. So, be very careful of which experts you are hearing from.
Agreed
Roger: I once counted up the federal government employees for health and disease control to be about 10,000. During the height of the Covid pandemic, the information released mas minimal and often misleading, and to this day, I would say that I learned very little from the government. The six-foot rule rests on what? And it totally neglects duration of exposure. It is simple logic to believe that six feet at one hour is far worse than 1 foot for five seconds. Masks! To this day we don’t have a clues how effective different types of masks are in different circumstances. Vaccinations! How effective were they against which variants, and for how long? Clearly, they were over-hyped from the beginning. And then the claims of protection were obviously skewed toward effectiveness of vaccines, just like news releases on climate are skewed toward alarmism. Who was dying? The demographics was difficult to discern, but I had the impression that if you subtracted off the deaths from nursing homes, the pandemic didn’t seem quite so bad for the rest of the population. The effectiveness of shutdowns? There seems to be evidence that the cost/benefit was very unfavorable but the 10,000 employees haven’t done squat to clarify that. Ancillary factors: washing your hands (like we’ve been told a hundred times): is that useful at all? As I look back on the past three years, I can’t find hardly anything useful in the form of expert advice regarding Covid, and all told, I would give the government a grade of F-minus
The insularity of government agencies that persist in dubious agendas while ignoring shadow activities outside of their limited world view was clearly dominant in the Covid era. But it is not limited to Covid. There are shadow groups (blogs) regarding climate change. While these are all politically biased (so are the government programs) they do obviously include some smart people with system analysis expertise which seems to be lacking in government? The government pretends that these are invisible and blithely persists, while ignoring everything in the shadow world. In an area of my expertise (human missions to Moon or Mars) the government is also insular and ignores anything contrary outside their sphere. My book “Human Missions to Mars” third edition recently published in January, demonstrates fallacies in government thinking that will never be considered by NASA.
The main things that distinguish the primary attitude of government agencies is insularity, near-sightedness, and not invented here.
The US approach to science advice in the pandemic was extremely poor. I have not written up an evaluation (yet) but I have given some talks on it, and there is not a lot to praise here.
On this, I would add that as far as I know, at no time did the CDC advise people to boost their immune system - get out in the sun, take Vitamin D and so on. And yet we knew very early on that it was the immuno-compromised that were the most susceptible to the virus. Why was this not communicated to the public and factored into the advice on lockdowns?
Dr Pielke
Important subject....but I too was stunned by the recommendation that essentially 'experts' just need to be more expert and to do their jobs better. Re covid ,the CDC, NIH and hundreds if not thousands of state and local health agencies, massively populated with 'experts' and 'scientists' made a hash of it. It is clear from personal discussions with a number of those employed within these expert and scientific communities that many disagreed with the 'expert consensus' but were not able or willing to express their concerns. So too with climate. I don't know what the solution might be but I failed to see one in your piece. S Wilson
Thanks, on the US, I completely agree. The EScAPE project looked at more than a dozen countries and the effectiveness of science advisory mechanisms varied across them. Of course, it is important to distinguish advice from decisions and their outcomes -- we focused on advisory mechanisms, and not the ultimate COVID outcomes informed (or not) by advice.
Steve: In sensitive matters, the government will usually not allow individuals to voice opinions, no matter how 'expert" they are. The party line is administered from above. A couple of examples from NASA: (1) There are two ways to do a human mission to Mars: short stay and long stay. The system analysts long since gave up on short stay because it required nuclear propulsion (politically very difficult), required gravity assist from Venus on return flight (heat and radiation) and the total return on investment was low (plant flag and run). But in 2019, the Administrator told the analysts that short stay was the new front runner. The analysts reluctantly had to align like weather vanes. (2) NASA is developing recycling systems for waste gas, liquids and solids for deep space. The program for years has concentrated on increasing percent closure, while neglecting reliability. One maverick in NASA spent years pointing out that 90% closure and 95% reliability is far more important than 95% closure and 90% reliability. The NASA program goes on as planned. Impressively, the maverick is still employed - because he presents no danger - those who run the program don't listen to him.
Hi Roger,
I’ve been interested in the idea that evidence-based public policy can help us transcend policy dictated by political interests. But so far, it looks more like at best we have evidence-influenced politics.
So, there are experts in various scientific disciplines who can be better prepared to inform policy makers. That’s one thing. But is there any chance of transitioning from scientific information used as a tool for politicians to enhance their power, toward an actual science of public policy making?
My view is that "evidence-influenced politics" is probably the best we can ever hope for, and in democracies, that is probably appropriate. That said, we still want robust evidence and not evidence curated or invented to serve political ends.
I agree with others, that “science advisory mechanisms” such as they are don’t function particularly well. I think the fundamental question is “why do you think you need “science” advice separate from other kinds of advice?” Because ultimately it is one element in the mix, and isn’t how the mix is conducted the key element of policy? And there is an imbedded privilege/authority of raising science advice over other kinds of advice (economic, what could go wrong with that?) plus the fact that Roger and others have pointed out that the scientific communities have their own disagreements within and among disciplines, and often reflect only one political perspective, whereas people in our country are close to half and half with another political perspective.
So before we rate science advice I think we ought to discuss and articulate what it is we want from science advice, which can’t be separate from determining how it should be developed and used with other information. Because I think people can sense when political views are imbedded in that advice and certainly practitioners know how reality-based it is.
I have started calling for "expert advice" rather than "science advice" for such reasons.
I had to read this 3 times to make sure I wasn't suffering from brain fog. Are you serious?
"In general, science advisory mechanisms function well on topics such as vaccine approval, climate change........ "
Do you really think that the vaccine approval process for Covid functioned well? For young people/children? Boosters for those with natural immunity? Prioritizing early doses? How about other aspects of the Covid response (which by the way you teased upcoming content on this about a year ago)?
Do you really think that the science advisory mechanisms are working well for climate policy? Policy makers find scientists (in and out of government) to tell them what they want to hear. As you've pointed out many scientists advising on policy lack integrity and have financial and reputational conflicts of interest.
Do you really believe that having more permanent unaccountable government bureaucrats charged with providing science advice to policy makers who ultimately decide their professional fates is a net plus? Is the State Department and diplomacy really a model to be followed when many have criticized it as an echo chamber?
Some science advisory mechanisms work better than others, to be sure. That is why we undertook this comparative assessment. The US stands out as a poor performer on science advice in the pandemic (under DT & JB, IMO), to be sure. Science advisory mechanisms are imperfect and difficult to implement (like diplomacy), but what is the alternative?
The alternative is to elect people to leadership positions that have the intellect, energy, integrity and courage to solicit the best available advice and to work hard to understand it so that they can make responsible decisions. Neither DT nor JB fit this mold. While I am sure you are not a DeSantis fan, his model for obtaining expert advice and implementing thoughtful policies during Covid is the best model that I can think of.
Well, I can't argue with this list of characteristics! As a policy scholar I often tell my students that we operate in conditions that are not the ones we wish we had but the ones that we get. And in democracies that includes the elected officials present at he time that we are offering advice, evidence and the like.
Some of my colleagues believe that we academics should spend more time politicking for our favorite candidates, I'm more of the view that whomever is elected is going to need expertise, so some of us should spend more time on research and advice, and less worrying about elections. But I know this can be unpopular!
I appreciate your comments and interactions here.
Don't take this the wrong way, but you really should find a way to get out of the academic bubble that you have lived and worked in all of your life. Viewing the world from the vantage point of the academic ivory tower can't be good for your health. You're only in your early fifties, there's till time!
Rather than how to give advice better, the real question is how to give better advice (Better leaders would be a big help, but that's a whole other bucket of poo.). One of the major causes of the Covid fiasco (the response as opposed to the pandemic) was that the advice was given by hedgehogs, to use Tetlock's analogy. Their focus was on response, but the real focus should always be on recovery. During and after Katrina, law enforcement and other first responders stopped out of state businesses from coming in to help with clean up and recovery. Their intent was laudable - keep out the hucksters and the flim-flammers. But NOLA's recovery was slower than it needed to be because of a lack of construction professionals. In Mississippi, since out-of-state handlers and haulers weren't allowed in, the downed timber caused a new forest fire literally every day.
Comprehensive stress tests might be a good way to improve both advice and advice-giving, leading to both better responses and better recoveries. I've written about this (resilientcommunities.home.blog/2021/01/07/rising-after-the-fall/). In short, a group - perhaps coming from the group in your first recommendation - develops a scenario for some sort of crisis that will require a massive national response. The scenario should be realistic not worst case. They might also identify what recovery should look like, in concrete terms. Then let the experts (we need to include the states in this) have at it. Their job is to identify a realistic path through the crisis to the recovery goal. If one agency takes the lead in response (e.g., CDC) then they're forced to look at the consequences of their actions through the other agencies' eyes.
Stress testing of this type offers some real positives:
• It is based on the risks the nation actually faces.
• It uses all of the nation’s expertise and knowledge.
• It provides a time to recover based on the resources actually available to the nation.
• It indicates opportunities for mitigating action to reduce the time to recovery.
• It avoids the trap of having responders from different agencies exchanging business cards in the midst of crisis.
• Not least, it breaches the bureaucratic bastions and forces the hedgehogs to think more like foxes!
Good stuff!
I think John brings up an important point.. anytime “science” advice is separated from “advice by people who actually do the work” there is a problem. To change topics, I recently was on a webinar with climate guy Stanford Mark Jacobson, who firmly believes there are no practical problems with building out industrial wind and solar today... they can be built in 1-3 years.. well not on federal land, and not without changes to regulatory regimes. Out of his view comes policy direction (and many news stories as I was on the webinar for reporters).. One of the structural problems with “science” advisory committees is that there is no place for practitioners in the dialogue.
Point well taken, BUT...
Jacobson is an academic, involved in a non-profit that is sort of a marketeer for renewables. There is a lot out there debunking some of his claims, but he's a near-perfect example of what you get from a hedgehog. Yes, you could potentially stand up wind and solar generators in a few years. However, California's continuing problems with brownouts and rolling blackouts are dark testimony [intentional bad pun] that he is ignoring the fragility this introduces into the California power grid. I.e., just because you "can" doesn't mean it's a good idea. In short, for a renewable-intensive grid, we still haven't figured out how to reliably provide electric power when physical conditions such as those in CA in August 2020 lead to energy demands that intermittents can't fill. In other words, Jacobson - as a hedgehog - seems to provide good information about the renewables, but doesn't really have the experience or knowledge to be able to say how well these can be integrated into a power system for a public that expects consistent electricity for their homes and businesses. His analyses of the CO2 emissions of renewables vs nuclear are also flawed in my opinion - but look at the assumptions and then decide for yourself.
Not to beat this horse any deader (?), CA's experience in moving toward 100% renewables by 2050 is a good example of why something like my suggestion is useful. Their current "plan" for the transition is sort of like
• We'll mandate the increasing use of renewables;
• We'll ban the use of everything else (of course ignoring lots of other side effects, e.g., low income families will have to do expensive retrofits converting from natural gas to electric, AND pay more for the electricity per month than they did natural gas);
• And somewhere between now and then a couple of miracles will occur - affordable MW-scale batteries will become available AND the engineers will figure out how to handle all of the intermittency issues.
Governor Newsom's recent decision not to close CA's only surviving nuclear plant is a tacit admission that the "transition" plan wasn't (a plan, that is).
Sorry - off the soapbox for now.
This is just another case where academics (whose views are privileged in this society) don't ask questions like "what do people think who work with this every day?" and "what do people think who live in the places where buildout occurs?".
I wonder if part of this is unclear linkages between practitioners and academics at places like Stanford (not so much in ag because of the land grants) and the continuing bias against social sciences.
I agree with this 100%. IMO this is a big reason why expert advice on technologies is often more effective than expert advice on policy. Whether or not a rocket can go to Mars is a lot easier to make expert judgments on than, say, what are options for responding to an emerging pandemic?
"In general science advisory mechanisms function well on topics such as ...climate change.." I find it surprising you would say this Roger. I strikes me that the various conferences where decision makers gather are prone to issuing apocalyptic statements and these statements seem to drive target setting that doesn't adequately balance costs and benefits. Doesn't that imply that the advisory mechanism is not functioning adequately? Could you comment?
The IPCC and US NCA are of course imperfect institutions, and as I often say, if they did not exist they would have to be invented. They work well in the sense that they have clear mandates, processes and can be improved. It is also important to distinguish process from substance ... the IPCC has really done a poor job on scenarios in its most recent assessments, but it did much bettter in the past with SRES. The fact that we can make such judgments and seek to course correct is a strength of science advisory bodies.
That's what I was looking for. Thanks. Although few would say CO2 emissions have zero effect on temperature and, through that, climate, there is a very wide range of informed opinion as to how significant an issue this is and the level of disruption justified to "address it". I have the impression that those who have the most influence within the IPCC don't adequately represent the full range of opinion on this in that they tilt to the alarmist side of the continuum. The result is a widely accepted but economically damaging set policies that, to many in the public, defy commonsense. Even if politicians are onside with the public the fact that the IPCC is assumed to be the final arbitrator of "the science" makes dissent "anti-science" by definition. I wish I had the answer but am convinced we're headed in the wrong direction on the climate issue and the very existence of the IPCC is partly responsible.
But do those actually give "science advice?" As a person in a subfield I might use their views on say whether rain will increase, but how that translates into what I do is a function of my own discipline's choices... say a water manager vs. a wildlife biologist vs. transportation planners. I would call the IPCC and the NCA as "research roundups" or meta-reviews.
Yes, there are many flavors of what is generally called "science advice" and that also includes "shadow science advice"
I am a stickler for greater precision in what is actually meant by "science advice" as that general phrase doesn't really do much for us by itself
A difficult aspect of generating effective policy is that it requires subjective values judgements. Scientific discipline advice is often relevant to our choices, but most experts don't really have expertise in the intersecting issues. e.g., what are the impacts of dietary changes to fight climate change on human nutrition? What are the impacts of coronavirus masks on child learning and socialization? Too often, scientists who have a moral goal in the mix make too many flawed assumptions about the intersecting issues they know little about. e.g., they might claim that masks don't inhibit communication, or that phasing out meat will only improve human nutrition. This is why it is unwise for elected officials to pass relevant policy decisions solely to domain experts.
Yes, I agree with this. I discuss these issues a bit here: https://rogerpielkejr.substack.com/p/series-making-sense-of-science-in
I think that "scientists" seldom interact directly with policy decision makers. Instead they interact with their managers who interact with their department heads who interact with their division heads who interact with their agency heads who interact with the administration science advisor who probably gets 10 minutes with the decision maker.
What could possibly go wrong with this?
I often wonder if Roger feels his efforts border on Sisyphean. Over the last 20 years I've viewed various crises (e.g., Great Financial Crisis, COVID-19, climate) more objectively since taking his science and policy course in 2002. From my perspective it appears that political power as well as the politization of science have only increased.
Maybe more like Camus:
"The struggle itself toward the heights is enough to fill a man's heart. One must imagine Sisyphus happy."
Just stumbled across one of your experts about expert advice and how contributing research papers can be improved. Quite interesting. I’ve not thought much about this stuff https://www.degruyter.com/document/doi/10.1515/peps-2023-0002/html
Thanks for the link!
Interesting stuff. However, many of the suggestions for how previous failures might be remedied in the future, depend upon a retrospective view of what the failures and successes were in relation to one particular event - the global Covid pandemic. Such retrospective reflections are of course valuable in thinking about how we might respond in the future. But the future is unknowable. The particular combination of emergency, scientific knowledge, political configurations both national and international, distribution of expertise, and so on, is unlikely ever to be repeated. That is not counsel of despair, but realism.
One suggestion for improvement is that we might try to make candid admission of failure and misjudgement less politically costly. At the moment, retrospective judgement of failure is treated with a sanctimonious ferocity. This makes passing the buck more important, for many actors, than learning from complex situations. How do we change that?
Exactly, we have an ‘image is everything’ political culture, this could be a leverage point for systemic change.
I’m a pilot. The aviation industry has wholeheartedly adopted a just culture, with safety and error reporting encouraged with no jeopardy, the first thing you do when you make a mistake is own up to it. The details are then used in an evidence based training system so others can benefit from lessons learned. It’s so obviously simple and optimised for the good of the system as a whole, it is extremely frustrating to see the blatant lies and coverups by some of our leaders. Imagine if the travelling public thought the same of us as pilots, the evidence speaks for itself.
We wouldn't be flying if the aviation industry operated the same way.. what is this telling the public.. "your government really cares about aviation and wildfire improvements and safety- public health.. not so much."
I agree with Sharon F. (below). We also do this well on NTSB investigations and also, at least in the past, on post-disaster assessments. I agree that judgments of failure are often not welcomed. Just look at the COVID origins issue . . .
Yes, to me the academic question is "how did the NTSB and the wildland fire community and others design a safe space for failure and how can it be replicated?" I don't know how the committee on the Challenger would it into this.
The wildland fire community has managed to do that.. for example they have an entire lessons learned website https://www.wildfirelessons.net/home. Another example, last year prescribed burns got out of control and did substantial amounts of damage in New Mexico. The Forest Service did a stand-down on prescribed burning and did not start until a serious review had been conducted (with experts in the public) and the ensuing recommendations were in place. Think about that approach, then think about errors in gain of function research. Academics could study how that culture (wildland fire) came to be, and why that kind of continual improvement does not happen in other disciplines/areas. I think if you asked many people it would be “political polarization” but wildfire folks work under a variety of state and federal executives.. so...