I finally turned over a few stones in Mann’s EIV reconstructions, little suspecting the perfection that awaited me in the cave using the simple and unassuming alter ego shglfulihad_smxx.
The figure below compares the SH reconstruction to the smoothed (SH) iHAD instrumental version. From the frail instruments of speleothems, bristlecone ring widths and upside-down sediments, Mann has achieved something divine – a “perfect” reconstruction.
Presumably out of an excess of modesty, Mann did not claim the perfect verification statistics that he was ‘entitled’, only claiming an “average” verification r2 of 0.28. But let’s not hide this light under a bushel. [Note: Jean S observes below that the calculations are done separately for calibration and verification periods. As I interpret his comment (together with my own take on Mannian RegEM), the method yields essentially 'perfect' reconstructions during the calibration period - no overfitting involved, of course - with verification stats dropping sharply in the verification period due, of course, to overfitting: a phenomenon that we discussed in a May 2006 post on VZ pseudoproxies.]
This was the first stone that I turned over in the EIV cave/ Perhaps more perfection awaits us deeper within the cave? Perhaps even richer treasures.
If you wish to confirm the perfection for yourself, here are scripts that download perfection from Mann’s FTP site. First download the shglfulihad reconstruction:
url=”http://www.meteo.psu.edu/~mann/supplements/MultiproxyMeans07/data/reconstructions/eiv”
x=ts(scan(file.path(url,”shglfulihad_smxx”)),start=0)
Next the instrumental target:
url=”http://www.meteo.psu.edu/~mann/supplements/MultiproxyMeans07/data/instrument/iHAD_SH_reform”
target=read.table(url);target=ts(target[,2],start=1850)
Now do Mannian smoothing on the instrumental target:
library(signal);source(“http://data.climateaudit.org/scripts/mann.2008/utilities.txt”)
url=”http://www.meteo.psu.edu/~mann/supplements/MultiproxyMeans07/data/instrument/iHAD_SH_reform”
target=read.table(url);target=ts(target[,2],start=1850)
cutfreq=0.1; ipts=10 #ipts set as 10 in Mann lowpass
bf=butter(ipts,2*cutfreq,”low”); npad=1/(2*cutfreq); #from
smooth= mannsmooth(target,M=npad,bwf=bf)
Now plot perfection:
test=ts.union(target,smooth,x)
ts.plot(test[,1:2],col=c(“grey80″,1,2),xlim=c(1850,2000))
title(“shglfulihad”)
lines(c(time(test)),test[,3],col=2)
points(c(time(test)),test[,2],pch=19,cex=.07)
legend(“topleft”,fill=c(“grey80″,1,2),legend=c(“Instrumental”,”Instr Smooth”,”‘Reconstruction'”),cex=.8)
Oh yes, here’s the rest of the ‘perfect’ reconstruction.
Figure 2 – this is the same data as above, but plotted over the entire record.
