I’ve now ported my emulation of Schneider’s RegEM PTTLS to R and benchmarked it against Jeff’s Matlab as shown below. I caution readers that this is just an algorithm. There are other ways of doing regressions and infills. The apparent convergence to three PCs noted by Roman is still pending as a highly interesting phenomenon. There are other RegEM flavors in Schneider’s algorithm, but I haven’t bothered trying to emulate these as it’s the PTTLS version that seems to be in play right now.
The method below retains coefficients from the final iteration, which I plan to use for further analysis. Here’s a short benchmark of the regem_pttls function.
First load functions
source(“http://data.climateaudit.org/scripts/regem/regem.functions.txt”)
Then load the data (also with a collation of Matlab intermediates)
#this uses BAS scrape for surface and reverse-engineered Steig recon_aws for AWS. Two surface series need to be deducted (Marion, Gough)
#version here is not quite file compatible with Steig. See Nic L on Dec 2002 problem
download.file(“http://data.climateaudit.org/data/steig/Data.tab”,”Data.tab”,mode=”wb”)
load(“Data.tab”)
download.file(“http://data.climateaudit.org/data/steig/Info.tab”,”Info.tab”,mode=”wb”)
load(“Info.tab”)
download.file(“http://data.climateaudit.org/data/steig/recon_aws.tab”,”recon_aws.tab”,mode=”wb”)
load(“recon_aws.tab”)
dimnames(recon_aws)[[2]]=Info$aws_grid$id
download.file(“http://data.climateaudit.org/data/steig/jeff.tab”,”temp.dat”,mode=”wb”);load(“temp.dat”) #matlab compilation#surface
surf=Data$surface
surf=window(surf,start=1957,end=c(2006,12)) #ends in 2006 per Steig
surf = surf[,-26] #deletes column 26 – Marion
surf = surf[,-17] #deletes column 17 – Gough
dim(surf) #600 42
sanom=as.matrix(surf)
for (i in 1:42) sanom[,i]=anom(surf[,i],reference=1957:2007)
#Steig probably used different reference period and this can be experimented with later
(ny=ncol(sanom)) #42#AWS reverse engineered (rather than READER scrape)
anoms =window(recon_aws, start = 1980);dim(anoms) #324 63
dat_aws = window(Data$aws[,colnames(recon_aws)],start=1980,end=c(2006,12));
dim(dat_aws) #324 63
anoms[is.na(dat_aws)] =NA
#apply(anoms,2,mean,na.rm=T) #some are zero, some aren’t
(nx=ncol(anoms)) #63# combine anomalies
anomalies=ts.union(anoms,sanom)
dimnames(anomalies)[[2]]=c( dimnames(anoms)[[2]],dimnames(sanom)[[2]])
dim(anomalies) #600 105
Now do the iteration through the function below for 5 iterations (it takes a while (: )
regemtest=regem_pttls(X=jeff$X,maxiter=5,regpar=3)
Now do a quick comparison to Matlab results:
test=unlist(sapply(regemtest,function (A) A$dXmis) )
data.frame(test=test,jeff=jeff$dXmis[1:length(test),3])
Match perfectly. Wasn’t that easy.
# test jeff
#1 1.25811556 1.25800
#2 0.46735524 0.46740
#3 0.31128378 0.31130
#4 0.18168994 0.18170
#5 0.11207077 0.11210
#6 0.08017931 0.08018
To do a run on (say) surface stations by themselves with regpar=2, just do this
regem.surface=regem_pttls(X=jeff$X[,64:105],maxiter=5,regpar=2)
To get station-by-station RegEM trends (whatever they may prove to be):
trend=function(x) lm(x~ I((1:length(x))/12))$coef[2]
trend.ant= apply(regem.surface[[21]]$X,2,trend)
