Pardon me, but the “hockey stick” “blade” is forced by the … ohhh, “cherry picking”, shall we say? …. of “red noise” data series that correlate to the instrumental temperatures. Once you’ve done that, what you have is no longer random, but in fact highly correlated with the instrumental data record (which shows a sharp positive slope in the 20th century time frame; the “blade” region).

What happens away from that is that as this correlation gets attenuated (due to the weak temporal correlation of the “red noise”), the variance increases and the data settles to an “arbitrary” (to use your word) baseline. Selecting different parameters would result in a different baseline far away from the “blade” region. Which is not exactly what the MWP is either; it is an anomalous warmer period with lower temperatures on either side, or at least that’s what people are saying.

In addition, as expected for low-pass filtered random series (your “red noise”), your reconstructions undershoot the instrumental data at the early end, and overshoot the instrumental data at the late end. This feature is not present in the Mann reconstructions

If proxies contain a temperature signal, then they *are not* random red noise. Random red noise is the null hypothesis and proxies – as legitimate temp signal – is the alternative hypothesis. Since calibrating random red noise to insturmental data generates a hockey stick, then this is the de facto null.f calibrating random red noise to insturmental temperatures generates a “hockey stick” shaped series, then if you are trying to demonstrate that climate proxies *are not* random red noise, then

Well, yes, the over- and under-shoot are intrinsic to a short series of sharply increasing values (which is forced by the requirement that the “red noise” series you … ummm, “cherry pick” by insisting that they correlate strongly with the short instrumental temperature record. This forces a “blade”, and is unremarkable, because all we've done is essentially inject information into the “red noise” samples by insisting on that correlation to a “blade” in the instrumental series.The over- and under-shoot arise because of the use of “red noise” (another “cherry-picking” constraint of yours). “Red noise” isn't information-free: It is essentially low-pass-filtered, so it has a structure that is not completely random. Part of that structure is the lack of HF components that would allow a sharp bend near the ends of the instrumental data region, so we have the under- and overshoots. But the take-home point here is: By insisting on “red noise” (and with specific structural parameters), we may simply be saying that we find that there are similarities between the proxy data and other series that have a pronounced LF enancement (or equivalently HF attenuation). Which may be just because we've managed to “describe” the data (posssibly in an ad hoc fashion) through the careful selection of our supposedly “random” proxy values.What do the simulations look like with “white noise” (pure random data)? With “red noise” of differing 'saturation” (i.e., different attentuation slopes, cutoff frequencies, etc.)?

As I said above, “red noise” is not “random”. It, at the very least, contains “information” in the structure of the noise spectrum (I'll pass over the meta-discussion as to whether purely random noise is also 'non-random' in having the explicit structure [which is testable] of pure randomness….)Besides the bandpass characteristics of “red noise”, it may well have (extreme) LF attentuation as well; I suspect that all the numbers that Stockwell generated fell in a certain range (although his “Methods” section is not explicit on this), or there was normalisation, so that there was no DC bias (i.e., “component”) in these series either.The absence of a DC component (or LF components) forces the regression away from the calibration towards an “arbitrary” (to use Stockwell's word) baseline, giving you the “stick” portion of the “hockey stick”.We two things about the “red noise” series: First that they're highly correlated with the instrumental data for the instrumental data period, and second, that they are not correlated with the instrumental data outside this time frame (or more specifically, that any such correlation is more and more attenuated the farther away, although Stockwell doesn't report his parameters for the “red noise” generator, so we don't know how far). What we're saying is that we get a “hockey stick” where there is imperfect temporal correlation and where two different mechanisms pertain in two regions of the reconstruction. This is entirely consistent with physical changes in this timeframe as well, say, oh, COâ‚‚ staying pretty much stable up to the industrial period, and then taking a sharp increase as we start to burn more and more fossil fuel….

We could refer to it as trendless red noise, or a random walk if you prefer.Consider the null case which can be simulated. Trendless “red” psuedoproxies yield some series which correlate with insturmental data and some which do not. The calibration techniques of reconstructions select those series which by chance mirror the insturmental data. By definition of the simulation, these are obviously spurious correlations. The selected psuedoproxies can have no value to reconstruct past climate in spite of correlating with the insturmental series.If you add a signal to the psuedoproxies – both in the calibration period and the pre-insturmental period, the calibration selects both true and false predictors of the historical signal. Yet the process assumes 100% certainty in selecting true proxies and therefore the historic climate signal will be attenuated by the selected trendless red noise, yet the climate of the insturmental/calibration period is unaffected. The amount of attenuation of historic signal would depend on the SNR.

You say:”where two different mechanisms pertain in two regions of the reconstruction.” But the generator of the series is the same throughout. As I say, the series that go into the hockey stick are 20% of the total, that go in all directions in the calibration period:”About 20% (204) of those sequences had a positive slope and a very high significance P(> |t|) < 0.001 in a simple linear regression of values against temperatures.”You then say:”This is entirely consistent with [a theory of] physical changes in this timeframe” but my point is that this shape is forced by the selection methodology, and so cannot be used to support the theory, as that would be circular. Blind men are blind. You can't select a group of blind men and then use it to prove men are blind.

David, Jeff Id has a post up today discussing Mann 07 – comparing the “redness” of the psuedoproxies used in that paper with the actual proxies of Mann 08.http://noconsensus.wordpress.com/2010/07/19/9657/

Sorry about formatting; seems that HTML tag facility here is crude and there's no “preview”….

“How can you call the match spurious when you made sure as best you could that you would see a match?”This is a calibration excercise that uses correlations with insturmental to select predictors of historical climate. How can selected predictors in a psuedocase of trendless red noise (and therefore no connection with climate) be considered anything but spurious?I'm sorry, I don't see how the example in your second paragraph shows how my example of calibrating trendless red psuedoproxies (with actual paleo methods) are going to succesfully select psuedoproxies which mirror simulated real life pre-insturmental. IMHO you would have to change the calibration method, or change the nature of the psuedoproxies (the degree of persistence of autocorrelation), or both. In any event, characteristics of actual proxies used puts constraints on data simulations.Jeff Id has already shown that the calibration method used in Mann 08 is sensitive to the red noise characteristics of the actual proxies he selected from (Sorry David, I'm not as familiar with your work ) by calibrating Mann's proxies against fake temp signals.

I think we are not disagreeing and the following statements are notinconsistent:”you're saying that random data can account for what Mann saw, but Isay that Mann's data is consistent with the proxies having someintrinsic significance (which external research also shows to beplausible if not quite likely),”The way to look at it is, the proportion of random proxies vs thosewith 'intrinsic significance' is unknown as it has not been controlledfor. It may be impossible to control for it. Thats why you needproxies that are a priori temperature proxies without the selectionprocess. The jist of the nature exchange, (which I still cant find)is that if you regard the proxies as bona fide a priori then the 20century increase is significant, but if you regard it as coming from aselection process, then the confidence limits blow out beyond thepresent temperature.

In other words, when you make the “red noise” look like the proxies (in at least some respects), you get the same results.When you say “noise structure”, that's a bit of an oxymoron. What you're saying is that the structure of your data (be it [formerly] “noise” or “proxy”) is matched.I don't disagree that the “red noise” series have some randomness in them (I guess we can disagree as to how much). But I'd expect that you would agree that the proxies also have some 'noise' — some randomness — in them.The question in the end is whether it is the 'randomness' of your “red noise” series that produces the “hockey stick” or whether it is the constraints you place on your random noise (that is to say, the filtering you so, the structure you force, the 'information' you add).

“[Guest]: How can you call the match spurious when you made sure as best you could that you would see a match?”[Layman Lurker]: This is a calibration excercise that uses correlations with insturmental to select predictors of historical climate. How can selected predictors in a psuedocase of trendless red noise (and therefore no connection with climate) be considered anything but spurious?”Because you've tossed all the ones that don't calibrate. It you had kept them all, you'd look at them and say that the small subset of those that do 'match' were spurious. Once you've deliberately chucked the rest as not 'matching', the ones that do 'match' are not “spurious” at all, but in fact are exactly what you were looking for.”I'm sorry, I don't see how the example in your second paragraph shows how my example of calibrating trendless red psuedoproxies (with actual paleo methods) are going to succesfully select psuedoproxies which mirror simulated real life pre-insturmental.”Pardon me, but I'm having a bit of a time taking you seriously when you continually misspell “insturmental” and “psuedoproxies”. Jut a little nit, but it suggests perhaps less familiarity with the subject than might be expected.In answer to your question, whether or not the “red noise” series 'mirror' the real-life instrumental data depends on what that real-life instrumental data is, doesn't it? If the real-life instrumental data approximates a flat-line, no trend regime prior to the calibration period, then enough random series with no DC component (i.e. normalised to some “arbitrary” baseline mean) will average out to about the same thing: A flat line (with perhaps a deviation from that down towards the actual instrumental data at the beginning of the instrumental data calibration region due to autocorrelation (low-pass) characteristics of the “red noise”. There's your “stick” portion of the “hockey stick”. This is what happens when you have series with no “trend” (at least far away from the calibration region). But if you have real proxies with no such trend away from the calibration region (say, due [at least in part] to relatively constant COâ‚‚ levels), you will get the same thing (but perhaps with less variance).IIRC, Mann et al. did Monte Carlo simulations to show “skill” of the proxies, comparing with “red noise” sequences:[Mann et al., PNAS, 2008]: Results from the early and late validation experiments were then averaged for the purpose of estimating skill metrics and uncertainties. So-called “reduction of error” (RE) and “coefficient of efficiency” (CE) skill scores for the decadal reconstructions were used as metrics of validation skill as in past work (20, 32). Because of its established deficiencies as a diagnostic of reconstruction skill (32, 42), the squared correlation coefficient r2 was not used for skill evaluation. Statistical significance of RE and CE scores were estimated by Monte Carlo simulations based on the null hypothesis of first-order autoregressive “red noise” (20, 32). Only those reconstructions that passed validation at the P = 0.05 (i.e., 95% significance) level based on both metrics were retained.Jeff Id has already shown that the calibration method used in Mann 08 is sensitive to the red noise characteristics of the actual proxies he selected from (Sorry David, I'm not as familiar with your work ) by calibrating Mann's proxies against fake temp signals.Cites for this?

“The jist of the nature exchange, (which I still cant find)is that if you regard the proxies as bona fide a priori then the 20 century increase is significant, but if you regard it as coming from a selection process, then the confidence limits blow out beyond the present temperature.”The 20th century increase is a given here (it's what's used to “calibrate” or “validate” (or both) the proxies for periods where the instrumental temperature records are not available. That is to say, the “blade” is there pretty much any way you want to look at it (unless you want to dispute the accuracy of the 20th century instrumental data, but that has nothing to do with the proxies).Whether this increase is unusual is a different question.It wouldn't be unusual if what we're seeing in the instrumental record is consistent with longer term trends.But that leads one to ask why the pseudoproxies are constructed with an apparent baseline for the data, that assume that the data over the period far from the validation approximate (or average to) some “arbitrary” value. Why? Why not sequences that show LF/extremely-LF components that simply follow the slope of the 20th century instrumental data (and that don't show the bent “hockey stick”? The variability away from the calibration period should increase with “random” noise, I agree, but why does it settle out (with higher variance) at something akin to 'average' values for the calibration period? I think it's because of restrictions on the range imposed on the generation of the “red noise”, otherwise weak [and getting weaker with distance but still non-zero] effects of the autocorrelation requirement [also an "ad hoc" restriction] would tend to produce plots that simply continued the 'trend' of the calibration period, which in fact your sample graph shows in any case for some short distance, but which seems to be overcome over longer distances with regression towards the “arbitrary” baseline. Putting in such range restrictions as seems to have been the case is another ad hoc choice and in fact teh insertion of “information” into the “random” data: To wit, that there is some long-term baseline. Once you've assumed that with your random data generation, you will have the “hockey stick”, but that's kind of like assuming your conclusions which is that conditions in the far past were pretty stable (no changing “baseline”) and comparable to today's. You take that (“stable” conditions outside the calibration period) and the forced “blade” to correspond to the instrumental temperatures, and your conclusion is unsurprisingly that we had stable temperatures a ways back, but a recent dramatic uptick. And this is true whether or not temperatures actually were stable, simply by your choice of assumptions in generating the “random” data.

The signal to noise ratio of red noise is zero. It contains no information. An AM broadcast has a signal to noise ratio greater than zero. It contains information. The trick is to extract the signal from the noise without distorting it. The argument is that the statistical methods used to reconstruct the past climate can produce what looks like a signal when there is no signal present and worse, attenuate any signal that is present.If you think that red noise series will correlate more frequently than white noise, why not do the math? It wouldn't be difficult in R. I don't have any interest in doing it because I'm reasonably certain you're wrong. It's up to you to do something other than make unsupported assertions.

Obviously white noise would have very few simulations selected in thecalibration period at the same level of significance. That is whatthe whole concern about spurious trends is over, so its hardly novel.The main reason that my graph is more uniform on the handle of thestick is that its composed of 1000,s of proxies. Real studies haveused far less proxies and are correspondingly more variable. Theoriginal Mann paper applied PCA incorrectly, but dating uncertaintyany other things depress the red component, as has been welldocumented.