Absolutely agree that there are many more interesting issues such as detection limits of the methodology than ME vs R2. Most interesting are the basic assumptions, of which stationary fractional gaussian noise models are fascinating. Under these assumptions I have run verification protocols of MBH98, which indicated apparent skill for random recons by R2 but not by RE, and the protocol of just drawing random points for verification, where both RE and R2 indicated skill (wrongly). One could think up other ones, and try to identify protocols that did work in this case. I don’t know if I want to do that but it would be helpful for someone.

Absolutely agree that there are many more interesting issues such as detection limits of the methodology than ME vs R2. Most interesting are the basic assumptions, of which stationary fractional gaussian noise models are fascinating. Under these assumptions I have run verification protocols of MBH98, which indicated apparent skill for random recons by R2 but not by RE, and the protocol of just drawing random points for verification, where both RE and R2 indicated skill (wrongly). One could think up other ones, and try to identify protocols that did work in this case. I don’t know if I want to do that but it would be helpful for someone.

Thank you for the “Discover more …” button. By-and-large I am not very interested in blogs on climate because most of the postings are of the form “You [the previous person posting] haven’t taken X, Y or Z [fill in according to your own beliefs] and therefore you are at least wrong and more likely stupid.” However, having finished work for the day, I trolled through a couple pages of links largely confirming my prejudices. Sorry, but just because someone agrees with you doesn’t mean the agreement is sound. Anyway on the fifth page I found the a link to Roger Pielke’s (Sr.) site (Climate Science) and clicked over to “Reflections of a Climate Skeptic by Henk Tennekes”
http://climatesci.atmos.colostate.edu/2006/01/06/guest-weblog-reflections-of-a-climate-skeptic-henk-tennekes/

Prof. Tennekes’s essay is a nice statement of his skepticism which seems to be based on Popperian criteria and Tennekes’s own specialty, turbulence. But I comment here not on Tennekes rather on Pielke’s response (#18) to the query “Please define skillful.â€? Pielke gives the American Meteorological Society’s definition of skill:
“Skill: A statistical evaluation of the accuracy of forecasts or the effectiveness of detection techniques.�

That is bland enough for just about anyone. Pielke follows the bland definition with specific, but after reading those specifics I find I have no idea as to whether he would prefer an R2 (correlation squared), RE (reduction in error), CE (coefficient of efficiency) or maybe one of Prof. Theil’s statistics. Is this just non-technical commentary? Probably, but there is an implicit message here for all of us who investigate the empirical properties of various estimation and prediction schemes: the accuracy of a prediction [both forecasts and backcasts as in reconstructions] lies in the nature of the forecast. An R2 of 0.2 and an RE of -0.1 might be pretty good if we are trying to predict a 4 standard deviations outside our sample. The recent National Academy of Sciences conference on climate reconstruction asked the presenters if reconstructions for a 1000 years past were accurate within 0.5 degrees C. Let me ask: is this not a more interesting test than an R2 or RE?

Of course, this test does not come with its algorithm, which of course means a debate on the application, but why not add this test to the mix. There is the complication in modeling of the 1000 years. I believe that is addressed by dual verification testing on the verification period and the “unobservedâ€? period. That is simulate a 1000 years which should be broken into three parts: 1 to N1 — the unobserved, N1+1 to N2 — the verification, and N2+1 to N [the 1000] — the estimation. Then ask what skill (be measured in R2, RE, mean absolute deviations, or whatever) is needed in the verification period to assure a, say, 95% confidence of being within 0.5 degrees for a, again say, 30 year average 1000 years past?

And yes, I have started, but I’m lazy and probably won’t finish, plus I don’t code as fast as you “kidsâ€? do. So while your reconstruction bones simulation are good, I am suggesting a — there are presumably many more — way to add some muscle the model.

Thank you for the “Discover more …” button. By-and-large I am not very interested in blogs on climate because most of the postings are of the form “You [the previous person posting] haven’t taken X, Y or Z [fill in according to your own beliefs] and therefore you are at least wrong and more likely stupid.” However, having finished work for the day, I trolled through a couple pages of links largely confirming my prejudices. Sorry, but just because someone agrees with you doesn’t mean the agreement is sound. Anyway on the fifth page I found the a link to Roger Pielke’s (Sr.) site (Climate Science) and clicked over to “Reflections of a Climate Skeptic by Henk Tennekes”
http://climatesci.atmos.colostate.edu/2006/01/06/guest-weblog-reflections-of-a-climate-skeptic-henk-tennekes/

Prof. Tennekes’s essay is a nice statement of his skepticism which seems to be based on Popperian criteria and Tennekes’s own specialty, turbulence. But I comment here not on Tennekes rather on Pielke’s response (#18) to the query “Please define skillful.” Pielke gives the American Meteorological Society’s definition of skill:
“Skill: A statistical evaluation of the accuracy of forecasts or the effectiveness of detection techniques.”

That is bland enough for just about anyone. Pielke follows the bland definition with specific, but after reading those specifics I find I have no idea as to whether he would prefer an R2 (correlation squared), RE (reduction in error), CE (coefficient of efficiency) or maybe one of Prof. Theil’s statistics. Is this just non-technical commentary? Probably, but there is an implicit message here for all of us who investigate the empirical properties of various estimation and prediction schemes: the accuracy of a prediction [both forecasts and backcasts as in reconstructions] lies in the nature of the forecast. An R2 of 0.2 and an RE of -0.1 might be pretty good if we are trying to predict a 4 standard deviations outside our sample. The recent National Academy of Sciences conference on climate reconstruction asked the presenters if reconstructions for a 1000 years past were accurate within 0.5 degrees C. Let me ask: is this not a more interesting test than an R2 or RE?

Of course, this test does not come with its algorithm, which of course means a debate on the application, but why not add this test to the mix. There is the complication in modeling of the 1000 years. I believe that is addressed by dual verification testing on the verification period and the “unobserved” period. That is simulate a 1000 years which should be broken into three parts: 1 to N1 — the unobserved, N1+1 to N2 — the verification, and N2+1 to N [the 1000] — the estimation. Then ask what skill (be measured in R2, RE, mean absolute deviations, or whatever) is needed in the verification period to assure a, say, 95% confidence of being within 0.5 degrees for a, again say, 30 year average 1000 years past?

And yes, I have started, but I’m lazy and probably won’t finish, plus I don’t code as fast as you “kids” do. So while your reconstruction bones simulation are good, I am suggesting a — there are presumably many more — way to add some muscle the model.