Since 2006, in between promoting numeracy in education, and examples of simple statistics using topical issues from the theory of Anthropogenic Global Warming (AGW) to illustrate points, I asked the question “Have these models been validated?”, in blog posts and occasionally submissions to journals. This post summarizes these efforts.
CommentsSome time ago I had a brief discussion with Leif Svalgaard on ClimateAudit blog inspired by an exchange between Leif and David Archibald when the latter complained that Leif’s TSI reconstruction was “too flat”.
The sunspots exhibited cyclic variability in terms of the frequency of the cycles and that most thermostats work by pulse width modulation and some digital music with pulse frequency modulation. Both these work in a similar manner the thermal inertia of whatever the thermostat is controlling smooths the temperature variability and the pulse frequency modulation’s demodulator is a simple low pass filter often just a series resistor and shunt capacitor. In both these cases only the duty cycle or the frequency varies but not the amplitude. Below is a description of how this behaviour can be simulated with an electrical circuit emulator called ‘qucs’.
CommentsDetecting bias in research is not so difficult when you know what to look for. The conclusions are not justified by the data. Instead, the data may confirm, be consistent with, (or not inconsistent with) the conclusions. Working against this however are basic human motives on the part of the writer, to find novel and interesting approaches, find significant results when nothing is there, to be accepted by their colleagues, to get grants and be published.
CommentsIan Castles organized a review of the Drought Exceptional Circumstances Report by two Accredited Statisticians, who also review my first report on the skill of the climate models.
The statisticians find inadequate validation of the models of drought, as well as suboptimal regionalization in the DECR. They also find my analysis lacked force, and so I have done additional analysis in line with their suggestions.
The last few posts in the series have consisted of reviews of an unsuccessful submission to the Australian Meteorological Magazine (AMM), showing how contradictions between models and observations were suppressed from the conclusions of the DECR. These reviews cover similar ground from a different angle: the skill of the climate models in the DECR, failing to identify any real skill in the predictions of drought, and ways of showing variation between the model (increasing drought) and their real world observations (decreasing drought) at the climatic time scale.
Below are the abstracts:
Some comments on the Drought Exceptional Circumstances Report (DECR) and on Dr David Stockwell’s critique of it
K.R.W. Brewer1 and A.N. Other1
28 January, 20091. K.R.W. Brewer is an Accredited Statistician of the Statistical Society of Australia Inc. (SSAI) and a long term Visiting Fellow at the School of Finance and Applied Statistics within the College of Business and Economics at the Australian National University.
2. A.N. Other is a pseudonym for another Accredited Statistician of the SSAI who prefers to remain anonymous. Full responsibility for the content is taken by K.R.W. Brewer.Abstract
The Drought Exceptional Circumstances Report (DECR) was authored by a team drawn from the CSIRO and Australia’s Bureau of Meteorology, and was publicly released in July 2008. Almost immediately it became a source of controversy. This evaluation, both of the Report itself and of the critique of it written by Dr David Stockwell, finds good mixed with less than good in both. The DECR itself is criticized for its poor delineation of Regions within Australia, for the choices made of statistics to be constructed, for the manners of their construction, and for not getting the best out of the relevant available data. Dr Stockwell is criticized for his inappropriate choices of methodology and of time periods for analysis, and also for misunderstanding some parts of what the DECR’s authors had chosen to do. Nevertheless, both the Report itself and Dr Stockwell’s critique of it are welcome stimuli to further investigate a serious issue within the climate change debate.
Validation of Climate Effect Models: Response to Brewer and Other
David R.B. Stockwell
February 4, 2009Abstract
A review by independent Accredited Statisticians, Brewer and Other [KB09], suggested that some claims in the report “Tests of Regional Climate Model Validity in the Drought Exceptional Circumstances Report” [DS08] were premature. Additional tests suggested by KB09 support the claim made in the original report of “no credible basis for the claims of increasing frequency of Exceptional Circumstances declarations”. The contributions of KB09 and DS08 to the evaluation of skill of climate model simulations with, arguably, weakly validated idiosyncratic statistics are discussed. These include recommendations for greater rigor in evaluating the performance of climate effects simulations, such as those used in standardized forecasting practices [AG09].
One thing is clear, the climate models that all of these predictions rely on have not been validated to accepted standards. That is a major lapse on the part of the climatologists who nonetheless use the models to influence public opinion and action.
Contrast the quality and professionalism of the review by statisticians, with the error-ridden categorical reviews by climate scientists to the AMM article. The greater rigor of the statisticians is clearly evident.
CommentsA review by independent Accredited Statisticians, Brewer and Other [KB09], suggested that some claims in the report “Tests of Regional Climate Model Validity in the Drought Exceptional Circumstances Report” [DS08] were premature. Additional tests suggested by KB09 support the claim made in the original report of “no credible basis for the claims of increasing frequency of Exceptional Circumstances declarations”. The contributions of KB09 and DS08 to the evaluation of skill of climate model simulations with, arguably, weakly validated idiosyncratic statistics are discussed. These include recommendations for more rigor in evaluating the performance of climate effects simulations, such as those used in standardized forecasting practices [AG09]. Read the rest of this entry…
Comments1. K.R.W. Brewer is an Accredited Statistician of the Statistical Society of Australia Inc. (SSAI) and a long term Visiting Fellow at the School of Finance and Applied Statistics within the College of Business and Economics at the Australian National University.
2. A.N. Other is a pseudonym for another Accredited Statistician of the SSAI who prefers to remain anonymous. Full responsibility for the content is taken by K.R.W. Brewer.
The Drought Exceptional Circumstances Report (DECR) was authored by a team drawn from the CSIRO and Australia’s Bureau of Meteorology, and was publicly released in July 2008. Almost immediately it became a source of controversy. This evaluation, both of the Report itself and of the critique of it written by Dr David Stockwell, finds good mixed with less than good in both. The DECR itself is criticized for its poor delineation of Regions within Australia, for the choices made of statistics to be constructed, for the manners of their construction, and for not getting the best out of the relevant available data. Dr Stockwell is criticized for his inappropriate choices of methodology and of time periods for analysis, and also for misunderstanding some parts of what the DECR’s authors had chosen to do. Nevertheless, both the Report itself and Dr Stockwell’s critique of it are welcome stimuli to further investigate a serious issue within the climate change debate.
CommentsBelow are Peter Gallagher’s thoughts on the reviews of the submission to AMM. Contrast this with ac’s impressions that “To my reading the reviewer’s criticisms are reasonable and pertinent.” It goes to show, that reasonable and unrelated people can see things in different ways. Where is the resolvability of fact in the review process? Consensus?
CommentsHere are the second reviewers comments. This is quite short, so I have only a few remarks to make, mainly that I tried as much as possible to mimic the analysis used in the DECR, so any criticisms of the methodology are also criticisms of the report. Perhaps the reviewer would like to inform the authors of the DECR about their thoughts on that?
CommentsThe Financial Times recently reported on the Australian bushfires, linking them to increases in greenhouse gases. We take another look at the data in the DECR and find Australia is getting wetter not drier:
CommentsHere are my comments on the reviewers 1 comments on the manuscript submitted to AMM about the DECR. The summary of the manuscript stated:
CommentsThe 2008 Drought Exceptional Circumstances Report (DECR) makes a number of bold claims in its assessment of likely changes in the frequency and severity of severe rainfall deficiencies over the next 20-30 years. This review presents an analysis which brings into question whether these claims can be sustained by the data. Taking into account the poor performance of climate models, as evidenced by simulations of area of exceptionally low rainfall trending in the opposite direction to observations, a more valid interpretation of the data would be for drought frequency and severity in Australia to remain largely unchanged in the future, with no expectation of a change in the climatological basis for EC declarations.
Peter Gallagher reports that even while the coals are still warm, some are already blaming the Victorian fires on increases in greenhouse gases.
The following summarizes indications of decline in droughts in Australia from 1900 to the present, compiled from data provided with the Drought Exceptional Circumstances Report. Some of this information was provided in the submission the the Australian Meteorological Magazine (more about this tomorrow). Drought is defined as the percentage of area with rainfall lower than the 5th percentile. The areas are averaged over seven Australian regions.
Read the rest of this entry…
A number of familiar tests, often used to evaluate the performance of models: R2 correlation, Nash-Sutcliffe efficiency and similarity of trends and return period, were reported here, noting not much evidence of skill in the DECR models compared with observations at any of these. I also said what a better treatment might entail but left that for another time:
CommentsPosts over the next few weeks will be updates on the status of reviews myself and others have initiated of the Drought Exceptional Circumstances Report (DECR), by the CSIRO and Bureau of Meteorology (BoM).
CommentsA couple of days ago, Luke, a frequent commenter, sent in a number of links to a new Australian Government drought initiative. The Minister Tony Burke has appointed an Expert Panel to examine the social impacts of drought as part of its national review of exceptional circumstances (EC) funding, which argues for a major change, based on incentives rather than emergency aid. In a recent speech, Peter Kenny, chair of an expert panel looking at the social impact of drought said of the Drought Exceptional Circumstances Report (DECR):
CommentsOne of the tests of climate models predicting drought in my review of the Drought Exceptional Circumstances Report was the correlation of predicted area under drought with actual observed area under drought. Lazar criticized my inclusion of the R-Squared (r2) coefficient, an issue I didn’t follow up at the time. Read the rest of this entry…
CommentsGeoff Sherrington has been drawing attention to some changes in the legal language attached to various emails and reports associated with the CSIRO and the Climate Adaptation Flagship (CAF). Since I have been posting up emails in an attempt to hold people accountable, I have been looking into the legality. In the case of reports, to what degree are the authors accountable for the accuracy of the contents? (Disclaimer: This post makes no representations or warranties regarding merchantability, fitness for purpose or otherwise with respect to the following assessment.)
CommentsBelow is the email received a month ago from Dr Andrew Ash, Director of the Climate Adaptation Flagship, promising a formal response to issues raised about the Drought Exceptional Circumstances Report (DECR) concerning no apparent attempt at validation of the climate models for drought in the report, or evidence of skill at modeling drought. No reply has been forthcoming to date.
CommentsSupport was expressed for our efforts with the DECR from a reader of the Sydney Morning Herald below.
Comments“Climate change could devastate fishing industry: CSIRO” shouts the ABC news, as scientists predict the salmon, rock lobster and abalone industries, barramundi, prawn and mudcrab fisheries will be affected by changing rainfall patterns. In a welcome trend, the fishing industry have questioned the climate findings in the CSIRO report.
CommentsThe venue for more formal debate on controversial topics is the scientific journals. As part of my trek into the desert of drought predictions in Australia, I submitted a review of the Drought Exceptional Circumstances report (abstract below) two days ago to the Australian Meteorological Magazine. To date I have not received an acknowledgement of its receipt.
CommentsChanges in the exceptionally dry years (droughts) have been estimated in the Drought Exceptional Circumstances Report (DECR) in two ways: (a) a statistical modification of the observed rainfall data (Box 3); and (b) analysis of simulations from 13 climate models. Up until now I have been looking at the modeling in approach (b). Today I started to look at approach (a). As mean rainfall declines the probability of exceptionally low rainfall increases. This is graphed in Box 3 (see also Table 6).
The parameters used in this simple extrapolation exercise have curious inconsistencies with their source. The DECR report says:
CommentsAny claims or representations made by a business must be accurate and truthful. If a business has been dishonest, exaggerated the truth, or created a misleading impression, then there is a very broad provision in the Trade Practices Act to prohibit such conduct by a corporation.
For example, the ACCC webpage on misleading and deceptive conduct gives an example of a business predicting the health benefits of a therapeutic device or health product but having no proof that such benefits can be attained. Note that there is no need to show that the product has no benefit in fact, rather it is misleading to make a claim when there is no proof. In general:
CommentsFollowing up on the post from yesterday, I test the assumption underpinning the regional climate change work in Australia.
The most common approach has been to assess how well each of the available models simulates the present climate of the region (e.g. Dessai et al. 2005), on the assumption that the more accurately a model is able to reproduce key aspects of the regional climate, the more likely it is to provide reliable guidance for future changes in the region.
As far as I can see this is an untested assumption, and may be a case of ‘accident chasing’.
Comments