Data that’s been only partially observed I’ve been updating my skills in fitting models to truncated data and was pleased to find that, like so much else in statistics, it’s easier than it used to be. First, some definitions: censored data ... [Read more...]

Branko Milanovic asked on Twitter: Idea for a paper: “Homogamy” on Twitter. Do people with more followers follow people with more followers? I don’t have time to write a paper but I was sufficiently interested to want to blog about it. Th... [Read more...]

I last blogged about books in September 2016 and it’s still in my 10 most popular posts. I thought I’d update it with some of the data-relevant books I read in 2017. Mostly these are books that were published before 2017 - it’s just that it was... [Read more...]

Official statistical classifications can be big, complex things. For example, the complete version of the “International Standard Industrial Classification of All Economic Activities, Rev.4” comes as either a 300 page PDF or a 1.3GB Microsoft Acce... [Read more...]

A puzzle In my last blog post, I gave a hypothetical test exercise for candidates in a recruitment process, involving the International Visitor Survey (IVS) from the Ministry of Business, Innovation and Employment (MBIE). I once managed the team resp... [Read more...]

A management challenge Between 2011 and 2017 I selected somewhere between 20 and 30 staff and contractors, for New Zealand public sector roles with titles like Analyst, Senior Analyst and Principal Analyst. Alternative names for these roles could h... [Read more...]

I’ve been writing on this blog less frequently in the past few months. Mostly this is because I’ve been working on a very intensive and engaging professional project that is much more hands-on (ie coding) than I’ve been at work for a while. So ... [Read more...]

Seasonality of plague deaths In the course of my ramblings through history, I recently came across Mark R. Welford and Brian H. Bossak Validation of Inverse Seasonal Peak Mortality in Medieval Plagues, Including the Black Death, in Comparison to Moder... [Read more...]

Traffic fatalities in the news Fatalities from traffic crashes have been in the news in New Zealand recently, for tragic reasons. For once, data beyond the short term have come into the debate. Sam Warburton has written two good articles on the topi... [Read more...]

Election results are in The New Zealand Electoral Commission have released the final count of votes and seats from the 2017 election. Moving in the direction most sensible observers expected, the National Party lost two seats compared to the “on-th... [Read more...]

More nzelect New Zealand election data on CRAN Version 0.4.0 of my nzelect R package is now on CRAN. The key changes from version 0.3.0 are: election results by voting place are now available back to 2002 (was just 2014) polling place locations... [Read more...]

House effects in state space models of voting intention “House effects” is the term commonly used in models of voting intention to refer to the different statistical biases of different pollsters. In my Model B state space model of the New Zealan... [Read more...]

R Markdown in corporate settings I’ve been busy recently writing a paper at work using R Markdown, the wonderful tool provided by the folks at RStudio “to weave together narrative text and code to produce elegantly formatted output”. I don’t ... [Read more...]

The long view There’s been noise in the public debate on the forthcoming New Zealand election about the value of voting intention polls. There’s often not enough attention to the long view on these things, so here is a graphic of 15 years of poll... [Read more...]

A new cross tab tool I recently put up a simple web app, built with R Shiny, to let users explore the relationship between party vote in the 2014 New Zealand general election and a range of demographic and attitudinal questions in the New Zealand Elec... [Read more...]

Income inequality data Ideally the Gini coefficient to estimate inequality is based on original household survey data with hundreds or thousands of data points. Often this data isn’t available due to access restrictions from privacy or other concer... [Read more...]

Calculating Gini coefficients Stats NZ release a series of working papers, and a recent one caught my eye because of my interest in inequality statistics (disclaimer - I am working at Stats NZ at the moment, but on completely different things). Workin... [Read more...]

I recently read the high quality book Global Inequality by Branko Milanovic. When reading this sort of thing, I often find I can increase my engagement with a topic by playing around with the data myself. In this day and age, this is much more likely... [Read more...]

Motivation This thread on Twitter prompted some questions for me about who actually turns up to vote in New Zealand’s elections. With limited time, I can’t answer many of the important questions raised in that thread and the article it refers to ... [Read more...]

After I wrote a couple of weeks back about state-space modelling of the Australian 2007 federal election, I received some very helpful feedback from Bob Carpenter, a Columbia University research scientist and one of the core developers of the Stan probabilistic language. Some of the amendments I made in response to ... [Read more...]