I have been interested in building intution for LTP also, with posts like Scale Invariance for Dummies. The difference between LTP and AR(1) say is easily seen on a plot of log standard deviation vs log aggregation (i.e. daily, weekly, yearly averaging say). LTP has this constant increasing relationship of variance to scale, but AR(1) (Markov or STP) the variance ‘fades’ at the higher aggregations or longer time scales. This shows clearly the danger in finding apparent ‘trends’ that are really noise at longer scales.

I would be interested in what you think of the arguments for LTP at a basic level. For something like financial series that ‘clear’ daily, one could imagine AR(1) applying, but for natural series the timing of measurements is arbitrary (above yearly). So the argument is that AR(1) is a function of the observers arvbitrary choice of time scale. Natural series can be imagined to be like AR(1) simultaneously at all time scales, a model approximated by fractional differencing. Random Numbers Predict Future Temperatures. Thus there is an argument underpinning LTP similar to special relativity, i.e. Newtonain physics is based on an absolute inertial frame, Special relativity drops that assumption. AR(1) is based on an absolute time scale, but dropping that assumption produces LTP behaviour. The evidence is that this is seen in all natural phenomena.

It terms of detecting LTP, another visual approach is the partial autocorrelation I showed in Options for ACF in R which shows you the possible order of the lags. What do you think of partial ACF?

Thanks again Martin.

Let me offer a little experiment. Generate a series with LTP noise. Now run a regression of it on a constant and linear trend. Check the D-W stat. It should generally be in the danger zone — well below 2.0. Now run the regression with the series on a constant, the linear time trend and the lagged value of the series. This should remove almost all of your first order autocorrelation and give a commensurately pretty D-W stat.

Take the residuals from that regression and run them on the trend and their lagged values up to, say, 10 to 20 lags. Then do a Wald test of the constraint that all the coefficients of the lagged residuals are zero. This is sort of rough version of the Breusch-Godfrey Serial Correlation LaGrangian Multiplier Test. Run the regression on the squared residuals without the exogenous and you have the basic construct of the ARCM LM test. For each there is test statistic that is asymptotically Chi Squared. I am certain that these tests are somewhere in one of the R packages.

Anyway the purpose of this comment is note for many of the variables that you and others have expressed interest in the autocorrelations, e.g. temperature series, there is often more than first order autocorrelation, and it might be wise to use test statistics that a little more general. (Note D&W came up with a D-W stat for higher order autocorrelations but I forgot just what it was. Also there are things more hip than the B-G and ARCH LM tests, but I wasn’t impressed enough to bother to implement them into my regression routines.)

Also for cases where one is interested in LTP per se, the Andrew Lo (of Campbell, Lo and MacKinlay fame) paper “Long-Term Memory in Stock Prices,” Econometrica, Vol 59, No. 5, Sep 1991 shows how to test for a function of a quasi Hurst exponent that accounts for the short term autocorrelation function. (Lo and MacKinlay’s book “A Non-Random Walk Down Wall Street” shows the test in a somewhat more digestible form.) The trick Lo uses is part of several tests or adjusted estimates of standard errors including the Newey-West estimates (which correct for autocorrelation and heteroskedasticity — well to a degree — in the S.E.s of the regression coefficients). Anyone who wants to assert long term persistence — or correlation or memory or what have you — needs to have a test that distinguishes from the “short term” correlation effects and the long term correlation affects. Lo gives one.