A variance campaign that failed

they ought at least be allowed to state why they didn’t do anything and also to explain the process by which they didn’t do anything.

First blush

One of the nice things about R is that new statistical techniques fall into it.  One such is the glasso (related to the statistical lasso) which converts degenerate variance matrices into positive definite ones.

Once I was in my local R session, all I had to do to try out glasso was:

> install.packages('glasso')
> require(glasso)
Loading required package: glasso
> ls('package:glasso')
[1] "glasso"     "glassopath"
> ?glasso

So four commands:

• Install the package — in general, you will be asked to select a nearby mirror for the CRAN repository.
• Load the package into the current session
• Look at what objects are in the package (and visible) — in this case two objects.
• Get help on the glasso function

The help says that the first argument should be the degenerate matrix that is to be rehabilitated.  But it also wants a second argument saying how much (and how) to shrink.  The help file is not lying in that regard:

> wrongo <- glasso(sp5.sampvar06)
Error in is.matrix(rho) : 'rho' is missing

The plan

Since there is a free parameter (actually a matrix, but a single number is allowed) that needs to be selected, I devised a plan:

• Use glasso with a range of values for rho on the sample variance for 2006 for the returns of almost all of the S&P 500.
• See which value of rho was best when tested out of sample in 2007 on a set of random portfolios — a la “The quality of variance matrix estimation”.
• Use the best value of rho on data from 2010, and use a different set of random portfolios to compare the quality of the glasso  in 2011 with Ledoit-Wolf shrinkage and a statistical factor model.

The reality

So now comes time to pick values of rho.  Hint: it has to be non-negative.  That’s the only hint I could glean from the help file.  And reading is for ninnies — it’s time to start hacking.  What number would you pick?

> sp5.sampvar06[1:4,1:4]
             MMM          ACE          ABT          ANF
MMM 1.314376e-04 3.050325e-05 2.225360e-05 4.302205e-05
ACE 3.050325e-05 1.379232e-04 4.379703e-05 5.564900e-05
ABT 2.225360e-05 4.379703e-05 1.091088e-04 2.977900e-05
ANF 4.302205e-05 5.564900e-05 2.977900e-05 4.320315e-04
> stillwrongo <- glasso(sp5.sampvar06, 1)
> stillwrongo$w[1:4,1:4]
         [,1]     [,2]     [,3]     [,4]
[1,] 1.000131 0.000000 0.000000 0.000000
[2,] 0.000000 1.000138 0.000000 0.000000
[3,] 0.000000 0.000000 1.000109 0.000000
[4,] 0.000000 0.000000 0.000000 1.000432

So rho equal 1 doesn’t look too promising.  But we do have a hint that maybe rho should be smaller than the typical value.  What would you try next?

I tried 1e-6.  And I waited.

And waited.

And waited.

After roughly an hour it finished.  That was, of course, before I thought that timing the computation would be a good thing — it took less than a second with rho equal 1.

Next was:

> system.time(gl06.1en7 <- glasso(sp5.sampvar06, 1e-7)$w)
    user   system  elapsed
10685.30     7.77 12457.60

That is, a leisurely lunch provided an insufficient interval (the timing numbers are seconds).  Maybe it’s like making chocolate — the longer it takes, the better it is.

Results

The plan has been mislaid by now, but we can still look at the results of random portfolios in 2007.

> require(PortfolioProbe)
> # generate random portfolios
> rp06.mw3 <- random.portfolio(1e4, prices=sp5.close[250,],
+    gross=1e6, long.only=TRUE, max.weight=.03,
+    port.size=c(90,100))
>
> # 2007 realized volatility of the random portfolios
> vol.rp06.mw3 <- sqrt(252) * apply(valuation(rp06.mw3,
+    prices=sp5.close[251:501,], returns='log'), 2, sd)
>
> # ex-ante variance from glasso
> predvar.gl06.1en7 <- unlist(randport.eval(rp06.mw3,
+    keep="var.values",
+    additional.args=list(variance=gl06.1en7)))

The correlation between the predicted volatility and realized volatility is 0.595 for both the glasso estimates.  The correlation for the Ledoit-Wolf estimate is 0.576.  So the glasso estimates give a better result.

Summary

• R is a good thing.
• glasso possibly has promise but we need some more of Moore’s law or algorithmic innovation before it will be of practical use.
• Thanks to the person who brought glasso to my attention.

Epilogue

We had flanked the farmhouse. We had made our first military movement and it was a success.

They were fine horsemen and good revolver shots, but their favorite arm was the lasso.

from “The Private History of a Campaign that Failed” by Mark Twain

Subscribe to the Portfolio Probe blog by Email