Table of contents for Reducing Research Bias
- Evidence Based Practise
- Bias in Research
- Bias Examples
- Interpretation Bias
- IPCC Fraud Solutions
Writing on Prof. Garnauts Heinz Arndt Lecture, Peter Gallagher pens a sensible comment on the lack of attempt to strike a balanced risk assessment. Unlike the 1996 inaugural speech of Adrian Smith as President of the Royal Statistical Society, who held out evidence-based practices as an exemplar for all public policy, Prof Garnaut dismissed the conflicting scientific evidence for global warming.
Evidence-Based Practice uses techniques from science, engineering, and statistics, such as meta-analysis of literature, risk-benefit analysis, and independent tests. EBP aims for clear understand of the relative quality of evidence used in decisions.
Generally, there are three distinct, but interdependent, areas of EBP. The first is the application of the most well-evidenced studies. This requires a basis for judging best research by some objective critieria. Thus second area is the systematic review of literature to evaluate the best studies. Finally, EBP is a “movement” where advocates work to popularize the method and usefulness, both by highlighting instances of good and bad evidence-based practise.
Evidence-based medicine has demoted statements of the “medical expert” to the least valid form of evidence. Thus, statements by the worlds leading organizations would rank poorly as a basis for public policy unless they are based on an evidence-based approach.
Systems to stratify evidence by quality have been developed, such as this one by the U.S. Preventive Services Task Force for ranking evidence about the effectiveness of treatments or screening:
Level I: Evidence obtained from at least one properly designed randomized controlled trial.
Level II-1: Evidence obtained from well-designed controlled trials without randomization.
Level II-2: Evidence obtained from well-designed cohort or case-control analytic studies, preferably from more than one center or research group.
Level II-3: Evidence obtained from multiple time series with or without the intervention. Dramatic results in uncontrolled trials might also be regarded as this type of evidence.
Level III: Opinions of respected authorities, based on clinical experience, descriptive studies, or reports of expert committees.
A more general approach from the Oxford Centre for Evidence-based Medicine might be more adaptable to earth sciences in general:
Level I: Well designed, randomised controlled trial, a decision rule validated in different conditions.
Level II: Well designed, comprehensive and independently tested studies; or extrapolations from level I studies.
Level III: Hypothesis tested theories or models with observations, or extrapolations from level II studies.
Level IV: Computer simulations, observational, based on physical principles, bench research or first principles.
Level V: Expert opinion without explicit, independent critical appraisal.
The difficulty with climate science and economics is the virtual impossibility of level I randomised controlled trials. This serves as a reminder the quality of evidence is generally never above level III. Human ingenuity may develop ways to improve the conduct of natural experiments, and the classification scheme serves to motivate better generation of evidence.
Based on the scheme above, Garnaut’s statement that the recommendations of the IPCC:
“Is not contested by the large majority of specialists, and by the leaders of the relevant learned academies in the countries” (page 6)
would put the evidence on level V, even though individual research examined by the report could occupy higher levels.
Consider other cases of sources of evidence, Global Climate Models (GCMs) and Paleoecological reconstructions.
GCMs would generally sit at level IV, one level higher than expert opinion, and so present low quality evidence. As an illustration that this evidence is indeed low quality, consider the Climate Impact Assessment (ACIA) report by an international team of 300 researchers for the Arctic Council, predicting the Arctic will lose 50% to 60% of its ice distribution. However, Arctic ice extent has since returned to long term averages, justifying its classification as level IV poor evidence of global warming.
Palaeoecology includes reconstructing past climates from tree-rings and other proxies of climate. The origin of the famous hockey stick graph claiming temperatures are the highest experienced in 600 (and then 1000) years, and this could only be attributed to human emission. This view on historic temperatures, particularly in the Medieval Warm Period, has been reversed by a number of more rigorous studies. At best, one could say that some of these studies sit at level III, and some at level IV.
When judged against these standards, one may well ask just where is the reliable evidence of global warming? What can be confidently attributed to human factors, and represents a serious threat? At the very least, the quality sources of evidence and their confidence would be clearly identified in an EBP.
Categories of recommendations
Recommendations for action on global warming, as Prof. Garnaut is doing, should be based on balance of risk versus benefit and the level of evidence on which this information is based. These levels, drawing on the U.S. Preventive Services Task Force are:
Level A: Good scientific evidence suggests that the benefits of the policy substantially outweighs the potential risks. Clinicians should discuss the service with eligible patients.
Level B: At least fair scientific evidence suggests that the benefits of the clinical service outweighs the potential risks. Clinicians should discuss the service with eligible patients.
Level C: At least fair scientific evidence suggests that there are benefits provided by the clinical service, but the balance between benefits and risks are too close for making general recommendations. Clinicians need not offer it unless there are individual considerations.
Level D: At least fair scientific evidence suggests that the risks of the clinical service outweighs potential benefits. Clinicians should not routinely offer the service to asymptomatic patients.
Level E: Scientific evidence is lacking, of poor quality, or conflicting, such that the risk versus benefit balance cannot be assessed. Clinicians should help patients understand the uncertainty surrounding the clinical service.
A large and growing body of scientists would suggest that despite all of the research, global warming is at Level E:
“the scientific evidence is lacking, of poor quality, or conflicting, such that the risk versus benefit balance cannot be assessed.”
The advice to Clinicians in this case is that they
“should help patients understand the uncertainty surrounding the clinical service”.
Ross Garnaut suggests that we should take up Pascal’s Wager out of fear of the possible consequences of global warming. A parallel can be drawn as in environmental science, as in medical science, where the treatment has attendant risks (costs). Garnaut wants us to take a bitter pill, costing at least $45 billion at this time. And while Garnaut acknowledges that
“The scientific assessments are highly uncertain, and their impacts on human activity and welfare even more so.” (page 4)
It does not follow that we are forced to take it:
“We have no alternative to making decisions on complex issues of valuation under great uncertainty.” (page 4)
While it is defensible to argue for taking the pill on the basis that public opinion wants and demands the treatment, evidence-based practise suggests that Garnaut should pay far more attention to understanding the uncertainties surrounding global warming and explaining these uncertainties to the public.
Adapted from wikipedia
8 responses so far ↓
David, ‘evidence-based medicine is an interesting and pertinent framework for considering the obligations of policy-makers. Thank you for pointing to its relevance. The systems of the “Preventative Services Taskforce” seem especially apt.
I suspect that RG would respond that the analogy is weakened by the time-scales involved in gathering data about the climate (hence the ‘wager’ element in his recommendations). But that does not really change the conclusion that the nature of the evidence and of the assurance (even his assurance, as you note) does not warrant drastic action.
I think the only responsible course for RG would be to “hedge the bet” on warming by recommending measures more proportionate to Level V (or Level E in the clinical practice scale) as a start until we have a better understanding of the physics of climate (warming, for example) and need to rely less on probabilistic techniques based on time-series analysis to project future climate.
But I’m not optimistic that this will be his recommendation.
systematising types of evidence is very useful. Teh difficulty with climate seems to be that, like economics, you rarely get to do trials, let alone randomised trials. This suggest that the quality of the evidence will never rise above level III except in the case of “natural” experiments. We may of course now be conducting such a natural experiment, but we have to wait a long time before teh answer is in. Hence the problem.
That is the nature of observational sciences, so never rising above level III is a worthwhile reminder just how much evidence is needed to provide certainty comparable with experimental sciences. I should put a paragraph in the post about that, as I hadn’t overlooked the issue.
Another issue is that there are fewer issues in climate science, while clinical medicine is very ‘fine grained’. We are really looking at one supposed problem with a few symptoms, and few possible treatments.
David, the “evidence-based medicine” slogan and many articles originated in southern Ontario, with one of the key authors, Gordon Guyatt, even coming from a squash-playing family. Something in the water, I guess. I’ve read some of the articles and there are some surprising parallels to issues in the multiproxy climate articles. Guyatt is mentioned in a couple of early CA comments http://www.climateaudit.org/?p=110#comment-577 ff. Cheers, Steve
It never hurts to remember that it was real people who brought about such improvements, even when the size of the opposition, such as in climate science, is large. It is a positive direction for climate skeptics, and doable to implement something like this. People can understand these levels and can even classify the quality of papers even when they don’t grasp the statistical details. But I don’t think it would happen unless the perception of the IPCC changed from acquiescence to scandal (like oil-for-food).
I think there are a few more posts in this topic. Cheers
[...] davids @ Evidence Based Practise [...]
[...] Science What happens if the science of the IPCC is flawed, as we often find here on this site? The idea that modern “science” is somehow pure and immune from bias is simply not realistic. There is a tendency to believe what suits one politically regardless of the science and I for one am no different. So bias is unsurprising since the member countries Australia, Canada, China, Russia, and Saudi Arabia, all of which have had a hand in editing the science and the political statement, are whittling away anything contrary. If only the climate science community was as as conscious of human bias as the medical community and adopted the standards of Evidence Based Practice in reviews of the evidence supporting their claims. If the science turns out to be flawed I expect carbon credits will continue to be a commodity, only you can’t eat them like hog’s bellies. [...]
[...] Formal processes are intrinsically expensive, and they also reduce the attractiveness of the work, implying greater compensation. Such a change will in fact attract different people as well as different methods. Something will be lost in the process, and an eye to preserving as much of the collegial culture as possible is also worth considering. Am I saying, “we are in trouble, send money”? Not really. I don’t think climate science is first order important as of now, as the big picture is pretty clear. It’s those of you who don’t trust us who should be willing to invest in the matter. I suggest recruiting people from other sciences who don’t have a dog in the hunt. But I’m afraid you’ll get the same answer you always do. The sensitivity to CO2 doubling on a century time scale is about 3 C. Sloppy methods or not we have this thing nailed. Now you can let us keep thinking about our angles and pins, or you can hire somebody to replicate our work. But if you insist on your sport of sniping at our informality, if you insist that we become more formal, you need to invest a lot of money to train us and/or replace us, because we weren’t trained as MDs or pharmacologists or (a few exceptions like myself notwithstanding) as engineers. The state of academic science is what it is for a number of reasons. Climatology is unexceptional except in having to deliver some very disconcerting news. You may argue that the nature of the news is such that climatology becomes higher stakes and needs to be reorganized and formalized. I have a great deal of sympathy with that position, and in that regard among others I’m an outlier within the field. Note, though, that such endeavors are expensive and prone to failure. The ‘opus’ that exists, the response to a need for an organized presentation, is the IPCC WGI reports. For all its flaws, the IPCC consensus process and its reports are an interesting and useful achievement. The network of trust on which human progress is based is badly frayed these days. I don’t think Climate Audit has made matters any better, but I understand that trust can;t be manufactured on demand. All I can do is state that I have complete confidence in the intellectual competence and moral integrity of those leading figures in the field I have been privileged to work with… It’s a problem. People are demanding forms of “proof” that aren’t well suited to the problem area. Atmospheres are complicated and interesting beasts; atmosphere-ocean-ice systems (of which we have only one non-simulated instance) the more so. They aren’t unknowable, but predictions about large experiments on a specific system will always be contingent. I am not influential enough within the field to effectively push any fixes you might suggest, but it would be interesting to hear about them anyway, to take matters in a less confrontational way. I Suppose rather than sneering at what is wrong you make some suggestions as to how to set it right, what scale that would require, and who should pay for it. That said, I believe that the concept of an outside audit is sound and I advocate one for the field of economics, so I can’t consistently argue against one for climatology. I’d be interested in constructive ideas as to how we could improve our credibility if our understanding is sound, or test our understanding if it isn’t. In my view the UN IPCC report is simply a review of the literature, useful but unsystematic and unremarkable apart for the hype surrounding it. Medical science conducts reviews all the time, and they have found that some guiding principles of Evidence Based Practice are essential: [...]
Leave a Comment