Show us your tests – Australian climate projections

My critique of models used in a major Australian drought study appeared in Energy and Environment last month (read Critique-of-DECR-EE here). It deals with validation of models (the subject of a recent post by Judith Curry), and regional model disagreement with rainfall observations (see post by Willis here).

The main purpose is summed up in the last sentence of the abstract:

The main conclusion and purpose of the paper is to provide a case study showing the need for more rigorous and explicit validation of climate models if they are to advise government policy.

It is well known that despite persistent attempts and claims in the press, general circulation models are virtually worthless at projecting changes in regional rainfall, the IPCC says so, and the Australian Academy of Science agrees. The most basic statistical tests in the paper demonstrate this: the simulated drought trends are statistically inconsistent with the trend of the observations, a simple mean value shows more skill that any of the models, and drought frequency has dropped below the 95%CL of the simulations (see Figure).

Rainfall has increased in tropical and subtropical areas of Australia since the 70’s, while some areas of the country, particularly major population centers to the south-east and south-west have experienced multi-year deficits of rainfall. Overall Australian rainfall is increasing.

The larger issue is how to acknowledge that there will always be worthless models, and the task of genuinely committed modellers to identify and eliminate these. It’s not convincing to argue that validation is too hard for climate models, or they are justified by physical realism, or use the calibrated eyeball approach. The study shows that the obvious testing regimes would have eliminated these drought models from contention — if performed.

While scientists are mainly interested in the relative skill of models, where statistical measures such as root mean square (RMS) are appropriate, decision-makers are (or should) be concerned with whether the models should be used at all (are fit-for-use). Because of this, model testing regimes for decision-makers must have the potential to completely reject some or all models if they do not rise above a predetermined standard, or benchmark.

There are a number of ways that benchmarking can be set up, which engineers or others in critical disciplines would be familiar with, usually involving a degree of independent inspection, documentation of expected standards, and so on. My study makes the case that climate science needs to start adopting more rigorous validation practises. Until they do, regional climate projections should not be taken seriously by decision-makers.

It is up to the customers of these studies to not rely on the say-so of the IPCC, the CSIRO and the BoM, and to ask “Show me your tests”, as would be expected with any economic, medical or engineering study where the costs of making the wrong decision are high. Their duty of care requires they are confident that all reasonable means have been taken to validate all of the models that support the key conclusions.

  • Katana

    “It is up to the customers of these studies to not rely on the say-so of the IPCC, the CSIRO and the BoM, and to ask “Show me your tests”, as would be expected with any economic, medical or engineering study where the costs of making the wrong decision are high.”

    Agreed, but, how do we educate them in this. Alternatively, how do we educate the public so they do not vote for those who ignore this standard for various reasons.

  • ricka

    Katana.

    In the United States, many legislators actually have backgrounds in important areas of regulation – such as finance, economics, law, health, etc.

    Perhaps the solution is to, over time, increase the number of statisticians in Congress.

    Also, perhaps more climatologists should run for office.

    Then, the public could be educated through the debate which occurs every two years, as the public is on many difficult topics.

    • Katana

      Ricka,

      Saying that legislators have experience with regulation is similar to saying they have experience with beating their spouses. MAYBE they have learned how not to cause major damage to the spouse and be effective, but, they shouldn’t be beating the spouse in the first place. Here in the US the Constitution and BoR limits the Federal Gubmint severely. Unfortunately the courts, legislative, and executive branches have determined that they know better and have reinterpreted the plain language to their preference of more power to “HELP” the people.

      Looking at the negative results of telecommunications, energy, food, drugs, health care, banks, insurance… regulation I would categorically state that they do NOT understand what they are regulating, or, only have selfish interests in mind. Again, they opt for more power to “HELP” the people in spite of continuing harm to the public.

      As far as Climatologists running for office, would you suggest Michael Mann or Roy Spencer as an ideal candidate? Again the issue comes down to people with agendas. I do not THINK Dr. Spencer has an agenda, but, I know little about him. Michael Mann may not have one, but, if he doesn’t is an absolute incompetent.

      Have you been following the court case in New Zealand over their local Temp record? Another great example of why Gubmint should NOT be doing this stuff.

      Was this an actual conspiracy? Who knows. Did many politicians jump into the middle of it for their own purposes? Undoubtedly.

  • Davids99us

    I think the decision-makers involved are educated in rigor, and are looking for a balanced assessment. But they are in the position, largely due to the political situation, forced to rely on sources like IPCC and CSIRO glossy pamphlets, that gloss over or ignore the uncertainty. It needs to be OK to apply the same standards as would be applied anywhere else, and not give climate models a free ride.

Bad Behavior has blocked 46962 access attempts in the last 7 days.