Some Options for Testing Tables

Contingency tables are a very good way to summarize discrete data.  They are quite easy to construct and reasonably easy to understand. However, there are many nuances with tables and care should be taken when making conclusions related to the data. Here are just a few thoughts on the topic.

Dealing with sparse data

On one end of the spectrum there is a struggle to deal with the mass amounts of available data and trying to make sense of data in the petabyte (and larger) range. At the other end of the spectrum lacking sufficient data has its own problems.

Collapsing row or column levels

Sometimes when working with discrete data if certain factor levels lack adequate data then it may be possible to combine the factor levels together. This may be done fairly easily with 4 and 5-scale Likert scales and, in fact, happens quite frequently. Taking this approach may allow for sufficient data to make conclusions without violating underlying assumption.  The following tables of some random data show how cells can be collapse such that the basic assumptions are met.

Sanity Checks

Though it’s not a formal test just doing some sanity checks and some other sensitivity analysis is a good way to check the stability of the test. If one can take a single observation and move it to a different cell and that changes the decision then one should reevaluate the criteria for making a conclusion.

Using a basic chi square test (though other statistical tests would help with this problem, including the chi square correction for continuity) gives a p-value of .0363 for the following table of some made-up data and would be considered significant at .  However, by simply moving one observation from the Cold/Fast group to the Cold/Slow group the p-value of .1575 is no longer significant at .  The volatility of the data is suspect and decisions should be taken with caution.

Sparse Data Tests

There are many tests to handle many different categorical data situations. Listed here are a few of the common approaches.

Chi Square With Yates Correction
Sometimes decreasing the chi square statistics (and increasing the p-value) is sufficient for the specific case of a 2 x 2 table.  In R, for example, this is applied by default.

Fisher’s Exact Test

This is often the immediate fall back for 2 x 2 tables. This test is based on the hypergeometric distribution. However, one important rule for this test is that it is conditioned on the marginal totals. An example counter to this rule is to take a random sample of 15 people. Suppose 5 are male and 10 are female. Here the chi square starts to break down. But the other problem is that Fishers Exact Test calls for the marginals to be fixed and that is not the case. If another random sample of 15 people is selected we could get a different number of males and females.

Fisher’s Exact Test was developed in a time (1934) when a computer just wasn’t available to play around with really complex hypergeometric distributions. A 2 x 2 table was really the only feasible sized table. Consequently, Fisher’s Exact Test was designed for 2 x 2 tables but can be used on any m x n sized table.

So why not always use Fisher’s Exact Test? At some point the two begin to converge and using the exact test may just be too exact. Alan Agresti and Brent Coull write an article here (pdf) that discusses this topic in the context of interval estimation.

Barnard Test
Is similar to the Fisher Test but this test overcomes the problem of conditioning on the marginal.  Like many tests this applies only to a 2 x 2 table.

McNemar Exact Test
McNemar’s test is used when the data are correlated. For example, a matched pairs design where there is a before and after treatment or when a question is asked on a repeat survey.  Like Fishers Test this provide options for smaller sample sizes.

These are only a few of the options available and great care should be taken in any analysis but smaller sample sizes add a bit more complexity. There are many other options including logistic regression as well as other non-parametric tests that are available for small tables.

Examples

Here is some R code that shows some of the tests I described: Chi Square, Fishers Exact Test, Barnards Test, and McNemars Exact Test.

[sourcecode language=”css”]<br /> x1 = matrix(c(7, 1, 2, 4),<br /> nrow = 2,<br /> dimnames =<br /> list(c(&#8220;Hot&#8221;, &#8220;Cold&#8221;),<br /> c(&#8220;Fast&#8221;, &#8220;Slow&#8221;)))<br /> x2 = matrix(c(7, 2, 2, 3),<br /> nrow = 2,<br /> dimnames =<br /> list(c(&#8220;Hot&#8221;, &#8220;Cold&#8221;),<br /> c(&#8220;Fast&#8221;, &#8220;Slow&#8221;)))</p> <p>plot.table(x1)</p> <p>chisq.test(x1, correct=FALSE)<br /> chisq.test(x2, correct=FALSE)</p> <p>fisher.test(x1, alternative=&#8221;two.sided&#8221;, conf.int=TRUE, conf.level=0.95)<br /> fisher.test(x2, alternative=&#8221;two.sided&#8221;, conf.int=TRUE, conf.level=0.95)<br /> x3 = matrix(c(50, 5, 3, 15),<br /> nrow = 2,<br /> dimnames =<br /> list(&#8220;Replication 1&#8243; = c(&#8220;Hot&#8221;, &#8220;Cold&#8221;),<br /> &#8220;Replication 2&#8243; = c(&#8220;Hot&#8221;, &#8220;Cold&#8221;)))</p> <p>mcnemar.test(x3, correct=TRUE)<br /> library(Barnard)<br /> barnardw.test(x1[1,1],x[1,2],x[2,1],x[2,2])<br /> library(ade4)<br /> table.cont(x1, csi = 2,<br /> col.labels = colnames(x1),<br /> clabel.r = 1.5, clabel.c = 1.5)</p> <p>[/sourcecode]