Writing on Prof. Garnauts Heinz Arndt Lecture, Peter Gallagher pens a sensible comment on the lack of attempt to strike a balanced risk assessment. Unlike the 1996 inaugural speech of Adrian Smith as President of the Royal Statistical Society, who held out evidence-based practices as an exemplar for all public policy, Prof Garnaut dismissed the conflicting scientific evidence for global warming.

Evidence-Based Practice uses techniques from science, engineering, and statistics, such as meta-analysis of literature, risk-benefit analysis, and independent tests. EBP aims for clear understand of the relative quality of evidence used in decisions.

Generally, there are three distinct, but interdependent, areas of EBP. The first is the application of the most well-evidenced studies. This requires a basis for judging best research by some objective critieria. Thus second area is the systematic review of literature to evaluate the best studies. Finally, EBP is a “movement” where advocates work to popularize the method and usefulness, both by highlighting instances of good and bad evidence-based practise.

Evidence-based medicine has demoted statements of the “medical expert” to the least valid form of evidence. Thus, statements by the worlds leading organizations would rank poorly as a basis for public policy unless they are based on an evidence-based approach.

Systems to stratify evidence by quality have been developed, such as this one by the U.S. Preventive Services Task Force for ranking evidence about the effectiveness of treatments or screening:

Level I: Evidence obtained from at least one properly designed randomized controlled trial.
Level II-1: Evidence obtained from well-designed controlled trials without randomization.
Level II-2: Evidence obtained from well-designed cohort or case-control analytic studies, preferably from more than one center or research group.
Level II-3: Evidence obtained from multiple time series with or without the intervention. Dramatic results in uncontrolled trials might also be regarded as this type of evidence.
Level III: Opinions of respected authorities, based on clinical experience, descriptive studies, or reports of expert committees.

A more general approach from the Oxford Centre for Evidence-based Medicine might be more adaptable to earth sciences in general:

Level I: Well designed, randomised controlled trial, a decision rule validated in different conditions.
Level II: Well designed, comprehensive and independently tested studies; or extrapolations from level I studies.
Level III: Hypothesis tested theories or models with observations, or extrapolations from level II studies.
Level IV: Computer simulations, observational, based on physical principles, bench research or first principles.
Level V: Expert opinion without explicit, independent critical appraisal.

The difficulty with climate science and economics is the virtual impossibility of level I randomised controlled trials. This serves as a reminder the quality of evidence is generally never above level III. Human ingenuity may develop ways to improve the conduct of natural experiments, and the classification scheme serves to motivate better generation of evidence.

Based on the scheme above, Garnaut’s statement that the recommendations of the IPCC:

“Is not contested by the large majority of specialists, and by the leaders of the relevant learned academies in the countries” (page 6)

would put the evidence on level V, even though individual research examined by the report could occupy higher levels.

Consider other cases of sources of evidence, Global Climate Models (GCMs) and Paleoecological reconstructions.

GCMs would generally sit at level IV, one level higher than expert opinion, and so present low quality evidence. As an illustration that this evidence is indeed low quality, consider the Climate Impact Assessment (ACIA) report by an international team of 300 researchers for the Arctic Council, predicting the Arctic will lose 50% to 60% of its ice distribution. However, Arctic ice extent has since returned to long term averages, justifying its classification as level IV poor evidence of global warming.

Palaeoecology includes reconstructing past climates from tree-rings and other proxies of climate. The origin of the famous hockey stick graph claiming temperatures are the highest experienced in 600 (and then 1000) years, and this could only be attributed to human emission. This view on historic temperatures, particularly in the Medieval Warm Period, has been reversed by a number of more rigorous studies. At best, one could say that some of these studies sit at level III, and some at level IV.

When judged against these standards, one may well ask just where is the reliable evidence of global warming? What can be confidently attributed to human factors, and represents a serious threat? At the very least, the quality sources of evidence and their confidence would be clearly identified in an EBP.

Categories of recommendations

Recommendations for action on global warming, as Prof. Garnaut is doing, should be based on balance of risk versus benefit and the level of evidence on which this information is based. These levels, drawing on the U.S. Preventive Services Task Force are:

Level A: Good scientific evidence suggests that the benefits of the policy substantially outweighs the potential risks. Clinicians should discuss the service with eligible patients.
Level B: At least fair scientific evidence suggests that the benefits of the clinical service outweighs the potential risks. Clinicians should discuss the service with eligible patients.
Level C: At least fair scientific evidence suggests that there are benefits provided by the clinical service, but the balance between benefits and risks are too close for making general recommendations. Clinicians need not offer it unless there are individual considerations.
Level D: At least fair scientific evidence suggests that the risks of the clinical service outweighs potential benefits. Clinicians should not routinely offer the service to asymptomatic patients.
Level E: Scientific evidence is lacking, of poor quality, or conflicting, such that the risk versus benefit balance cannot be assessed. Clinicians should help patients understand the uncertainty surrounding the clinical service.

A large and growing body of scientists would suggest that despite all of the research, global warming is at Level E:

“the scientific evidence is lacking, of poor quality, or conflicting, such that the risk versus benefit balance cannot be assessed.”

The advice to Clinicians in this case is that they

“should help patients understand the uncertainty surrounding the clinical service”.

Ross Garnaut suggests that we should take up Pascal’s Wager out of fear of the possible consequences of global warming. A parallel can be drawn as in environmental science, as in medical science, where the treatment has attendant risks (costs). Garnaut wants us to take a bitter pill, costing at least $45 billion at this time. And while Garnaut acknowledges that

“The scientific assessments are highly uncertain, and their impacts on human activity and welfare even more so.” (page 4)

It does not follow that we are forced to take it:

“We have no alternative to making decisions on complex issues of valuation under great uncertainty.” (page 4)

While it is defensible to argue for taking the pill on the basis that public opinion wants and demands the treatment, evidence-based practise suggests that Garnaut should pay far more attention to understanding the uncertainties surrounding global warming and explaining these uncertainties to the public.

Adapted from wikipedia