Decline Effect in Linguistics?

Josef Fruehwald writes:

In the past few years, the empirical foundations of the social sciences, especially Psychology, have been coming under increased scrutiny and criticism. For example, there was the New Yorker piece from 2010 called “The Truth Wears Off” about the “decline effect,” or how the effect size of a phenomenon appears to decrease over time. . . .

I [Fruehwald] am a linguist. Do the problems facing psychology face me? To really answer that, I first have to decide which explanation for the decline effect I think is most likely, and I think Andrew Gelman’s proposal is a good candidate:

The short story is that if you screen for statistical significance when estimating small effects, you will necessarily overestimate the magnitudes of effects, sometimes by a huge amount.

I’ve put together some R code to demonstrate this point. Let’s say I’m looking at two populations, and unknown to me as a researcher, there is a small difference between the two, even though they’re highly overlapping. Next, let’s say I randomly sample 10 people from each population . . .

[simulation results follow, including some graphs]

I [Fruehwald] think how much I ought to worry about the decline effect in my research, and linguistic research in general, is inversely proportional to the size of the effects we’re trying to chase down. If the true size of the effects we’re investigating are large, then our tests are more likely to be well powered, and we are less likely to experience Type M errors.

And in general, I don’t think the field has exhausted all of our sledgehammer effects. For example, Sprouse and Almeida (2012) [pdf] successfully replicated somewhere around 98% of the syntactic judgments from the syntax textbook Core Syntax (Adger 2003) using experimental methods (a pretty good replication rate if you ask me), and in general, the estimated effect sizes were very large.

However, there is one phenomenon that I’ve looked at that I think has been following a decline effect pattern: the exponential pattern in /t d/ deletion. . . .

I’m curious what the linguists in the audience think, especially about the last point (for which Fruehwald supplies a bunch of data that can be found at his linked post).

The post Decline Effect in Linguistics? appeared first on Statistical Modeling, Causal Inference, and Social Science.