A function for testing auto.arima() This is the third, and the last from me on this particular topic for a while, in a series of posts on fitting time series models and comparing time series to eachother, originally inspired by a question on the Cross-Validated Q&A site. In ... [Read more...]

Comparing two timeseries-generating blackboxes In my last post I talked about how this question on Cross-Validated got me interested. Basically the challenge is to compare two data generating models to see if they are essentially the same. Since then I’ve noticed that this problem comes up in a number ... [Read more...]

Comparing two timeseries-generating blackboxes This question on Cross-Validated got me interested. I gave a fairly inadequate answer and want to explore a few of the issues. Actually, I have a plan for an effective technique which is what I think the original post was asking for, but I need to ... [Read more...]

Inequality measures Part of my motivation for importing the New Zealand Income Survey(NZIS) simulated unit record file provided by Statistics New Zealand was to explore the characteristics of various measures of inequality. In particular, I’m interested in what happens to the sampling distributions as sample size changes of ... [Read more...]

Something missing In my last post I developed a new scale transformation for R using the approach and platform from the {ggplot2} and {scales}. I implemented a method proposed in 1980 by John and Draper that does some of the job of a logarithmic transform in reducing the dominance on the ... [Read more...]

A better transformation than my better transformation In an earlier post I put forward the idea of a modulus power transform - basically the square root (or other similar power transformation) of the absolute value of a variable like income, followed by restoring the sign to it. The idea is ... [Read more...]

Where to start to start? I was recently asked by a colleague manager from another organisation what direction they could give to a staff member interested in building skills in the whole “big data” thing. A search of the web shows hundreds if not thousands of sites and blog posts ... [Read more...]

I wasn’t happy with my visualisation of individual incomes from the New Zealand income survey. Because it used a logarithmic scale to improve readability, in effect all zero and negative values are excluded from the data. Whenever I throw out data, my tail goes bushy… there has to be ... [Read more...]

The quest for income microdata For a separate project, I've been looking for source data on income and wealth inequality. Not aggregate data like Gini coefficients or the percentage of income earned by the bottom 20% or top 1%, but the sources used to calculate those things. Because it's sensitve personal financial ... [Read more...]

Probabilities of winning with a given rating Backgammon clubs and on-line forums use a modified form of the Elo rating system to keep track of how well individuals have played and draw inferences about their underlying strength. The higher the rating, the stronger the player. Players with higher ratings are ... [Read more...]

New data listing for New Zealand Wellington’s PrototypeAlex has created a new GitHub repository aiming to list data about New Zealand. My first reaction was “hmm, will this be an improvement?” After all, we already have data.govt.nz which is meant to be the definitive aggregator of government ... [Read more...]