New Zealand’s Family Violence Death Review Committee recently released their Fifth Report Data: January 2009 to December 2015, some grim reading on what it describes rightly as “the devastation of family violence”. I was interested by Figure 3 o... [Read more...]

Pooling the polls with Bayesian statistics In an important 2005 article in the Australian Journal of Political Science, Simon Jackman set out a statistically-based approach to pooling polls in an election campaign. He describes the sensible intuitive... [Read more...]

The Integrated Data Infrastructure (IDI) is an amazing research tool One of New Zealand’s most important data assets is the integrated data infrastructure: “The Integrated Data Infrastructure (IDI) is a large research database containing micro... [Read more...]

Global choropleth maps Today I’m writing about using country-level choropleth maps at a global level, using example data showing nations’ military spending as a percentage of GDP. There are a few specific issues to work through as a result of the... [Read more...]

Continuing my examination of the individual level voting behaviour from the New Zealand Election Study, I wanted to look at the way individuals swap between parties, and between “did not vote” and a party, from one election to another. How much an... [Read more...]

Last week I posted some analysis of individual voting behaviour in New Zealand’s 2014 general election. In that post, I used logistic regression in four different models to predict the probability of an individual giving party vote to each of the fo... [Read more...]

Someone asked on Twitter about the characteristics of New Zealand First voters. While some crude conclusions can be drawn from examining votes by location cast and then comparing that with census data, we really need individual level data to answer the... [Read more...]

Continuing my exploring methods for spatial visualisation of data in R, today I’m looking at linked micromaps. Micromaps are a way of showing statistical graphics for a small subset of regions at a time, with a small map indicating which regions are... [Read more...]

Just a short note to say that I’ve finished creating an experimental map of New Zealand by the 66 Territorial Authorities (districts and cities), with area expanded or shrunk to be proportional to population at the 2013 census. This is in addition t... [Read more...]

I’ve been trying to catch up with mapping functionality in R and its extended ecoverse, so might do a few posts on this in the next month or so. First up (more or less by accident) is cartograms - maps where some other numeric variable is substitut... [Read more...]

The story so far Last week I looked at a few related methods of investigating causality with observational data, where the treatment is expected to be received by observational units (people or firms)… in a way that is structurally related to ... [Read more...]

This post jots down some playing around with the pros, cons and limits of propensity score matching or weighting for causal social science research. Intro to propensity score matching One is often faced with an analytical question about causality and effect sizes when the only data around is from a ... [Read more...]

Over the weekend I released a new webpage, connected to this blog, with forecasts for the New Zealand 2017 General Election. The aim is to go beyond poll aggregation to something that takes the uncertainty of the future into account, as well as relati... [Read more...]

This post is one of a series leading up to a purely data-driven probabilistic prediction model for the New Zealand general election in 2017. No punditry will be indulged in (if only to avoid complications with my weekday role as an apolitical public s... [Read more...]

I was once in a meeting discussing a time series modelling and forecasting challenge where it was suggested that “the beauty of regression is you just add in more variables and more lags of variables and try the combinations until you get something that fits really well”. Well, no, it ... [Read more...]

Polling data and other goodies ready for download A new version, 0.3.0, of the nzelect R package is now available on CRAN. historical polling data from 2002 to February 2017, sourced from Wikipedia some small functions to help convert voting numb... [Read more...]

At the OZCOTS (Australian Conference on Teaching Statistics) in late 2016 George Cobb gave a great talk entitled “Ask not what data science can do for the Humanities. Ask rather, what the Humanities can do for data science.” “George Cobb of M... [Read more...]

In the chaos of the last month or so of United States of America governance, one item that grabbed my attention was the claim by President Trump that 80% of appeals decided by the Ninth Circuit Court of Appeal are overturned by the Supreme Court of the... [Read more...]

Moving around sparse matrices of text data - the limitations of as.h2o This post is the resolution of a challenge I first wrote about in late 2016, moving large sparse data from an R environment onto an H2O cluster for machine learning purposes. In th... [Read more...]

There’s rightly been a lot of attention paid to USA President Trump’s first speech as President at the time of his inauguration. The speech broke with tradition by extending the acrimonious atmosphere of the election campaign into the Presidency. ... [Read more...]