[This article was first published on Econometrics by Simulation, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.
In the following post I will map out some item information functions for item response theory (IRT) models using the common 3 parameter logistic model for binary responses. The model takes three parameters (obviously) which relate to the item features easily named a, b, and c. a refers to item discrimination or ability to tell between subjects with high ability and low ability while b is connected with item difficulty and c is commonly known as the guessing parameter. This is a value which represents the minimum probability a subject should have of getting the answer correct if the subject knows nothing.
The 3PL takes the form of a logistic model. The probability of observing a correct outcome given a subject with ability theta is defined as follows
$$P(u=1|\theta,a,b,c)=c+\frac{1-c}{1+\exp(-Da(\theta-b))}$$
See below to see this probability mapped out. This post is not so much about the probability of correct responses using the 3PL but also the information functions associated with the 3PL model.
Information is a tricky concept as it seems strange that there is a functions that produces information. In a sense the information is a measure of how much certainty we know about a specific parameter. In IRT we are primarily concerned with how much information we know about the ability scores of our test takers. Typical computerized adaptive tests (CAT) incorporate algorithms that select items which are meant to maximize postential information or some other criteria related to information. Information for a test item with regards to subject ability estimates theta is defined by Lord (1980) as
$$I_{\theta}=a^2 *\frac{(pi-c)^2}{(1-c)^2}\frac{Q(\theta)}{P(\theta)}$$
with Q defined as (1-P. In order to make informed analysis of our test takers we must also know how much information we have with regards to our item parameters. Stocking (1990) derives the item parameter information functions for the parameters a,b, and c
$$I_a = \frac{D^2}{(1-c)^2}(\theta-b)^2(P-c)^2 \frac{Q}{P}$$
$$I_b = \frac{D^2a^2}{(1-c)^2}(P-c)^2 \frac{Q}{P}$$
$$I_c = \frac{1}{(1-c)^2}\frac{Q}{P}$$
Now let us see how they look. I will not explain the components of each graph as such things have been explained by many others. Reading the wikipedia article on Item Response Theory and the Stocking 1990 paper “Specifying Optimum Examinees for Item Parameter Estimation in Item Response Theory” will give the curious fairly detailed sources to learn from.
Okay, okay, I will add some commentary. With regards to item information about subject ability it is clear that we want to target items at the point closest to where the subject has a equal or near equal probability of answering correctly or incorrectly. The more questions we drop on that location the more likely the response pattern will be something like 1,0,1,1,0,0,0,1 which will tell use that the subject’s ability is at or very close to that level. As we move away from that point the information we have with regards to subject ability begins to decline.
Notice how items which have a higher a parameter not only have steeper information functions but also considerably larger values (in terms of magnitude) which is significant. For this reason computerized adaptive tests must be carefully controlled through various exposure mechanisms so as to prevent those items which have the most discrimination from always being given.
Note how the information with regards to the b parameter and theta is maximized at the same point when c=0 (which is b=theta). When c>0 the optimal point for getting information for theta is offset to the right but so it is for theta (though not depicted in a graph). Thus regardless of which information you are seeking, item difficulty or test taker ability, you face a similar optimization when delivering items.
This is in stark contrast to item parameter a which has information minimized when b=theta. Information with regards to a parameter is best garnished when the question is taken by a test taker with considerably higher or lower ability (when c=0) and only with considerably higher ability (when c=.15). This is in contrast yet again with the information regarding the c parameter which is optimized when a question is taken by a subject who has considerably lower ability than the item difficulty.
R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.
The 3PL takes the form of a logistic model. The probability of observing a correct outcome given a subject with ability theta is defined as follows
$$P(u=1|\theta,a,b,c)=c+\frac{1-c}{1+\exp(-Da(\theta-b))}$$
See below to see this probability mapped out. This post is not so much about the probability of correct responses using the 3PL but also the information functions associated with the 3PL model.
Information is a tricky concept as it seems strange that there is a functions that produces information. In a sense the information is a measure of how much certainty we know about a specific parameter. In IRT we are primarily concerned with how much information we know about the ability scores of our test takers. Typical computerized adaptive tests (CAT) incorporate algorithms that select items which are meant to maximize postential information or some other criteria related to information. Information for a test item with regards to subject ability estimates theta is defined by Lord (1980) as
$$I_{\theta}=a^2 *\frac{(pi-c)^2}{(1-c)^2}\frac{Q(\theta)}{P(\theta)}$$
with Q defined as (1-P. In order to make informed analysis of our test takers we must also know how much information we have with regards to our item parameters. Stocking (1990) derives the item parameter information functions for the parameters a,b, and c
$$I_a = \frac{D^2}{(1-c)^2}(\theta-b)^2(P-c)^2 \frac{Q}{P}$$
$$I_b = \frac{D^2a^2}{(1-c)^2}(P-c)^2 \frac{Q}{P}$$
$$I_c = \frac{1}{(1-c)^2}\frac{Q}{P}$$
Now let us see how they look. I will not explain the components of each graph as such things have been explained by many others. Reading the wikipedia article on Item Response Theory and the Stocking 1990 paper “Specifying Optimum Examinees for Item Parameter Estimation in Item Response Theory” will give the curious fairly detailed sources to learn from.
Okay, okay, I will add some commentary. With regards to item information about subject ability it is clear that we want to target items at the point closest to where the subject has a equal or near equal probability of answering correctly or incorrectly. The more questions we drop on that location the more likely the response pattern will be something like 1,0,1,1,0,0,0,1 which will tell use that the subject’s ability is at or very close to that level. As we move away from that point the information we have with regards to subject ability begins to decline.
Notice how items which have a higher a parameter not only have steeper information functions but also considerably larger values (in terms of magnitude) which is significant. For this reason computerized adaptive tests must be carefully controlled through various exposure mechanisms so as to prevent those items which have the most discrimination from always being given.
Note how the information with regards to the b parameter and theta is maximized at the same point when c=0 (which is b=theta). When c>0 the optimal point for getting information for theta is offset to the right but so it is for theta (though not depicted in a graph). Thus regardless of which information you are seeking, item difficulty or test taker ability, you face a similar optimization when delivering items.
This is in stark contrast to item parameter a which has information minimized when b=theta. Information with regards to a parameter is best garnished when the question is taken by a test taker with considerably higher or lower ability (when c=0) and only with considerably higher ability (when c=.15). This is in contrast yet again with the information regarding the c parameter which is optimized when a question is taken by a subject who has considerably lower ability than the item difficulty.
Pr <- function(theta, it, D=1) {
a <- it[,1]
b <- it[,2]
c <- it[,3]
e <- exp(D * a * (theta - b))
c + (1 - c) * e/(1 + e)
}
### Set parameters to be mapped over
theta <- seq(-3.5,3.5,.1)
require(ggplot2)
### Plotting Item response functions
cfix=0
afix=1
presp <- data.frame(theta=rep(theta,3),
prob=c(Pr(theta, cbind(afix, -2, c=cfix)),
Pr(theta, cbind(afix, 0, c=cfix)),
Pr(theta, cbind(afix, 2, c=cfix))),
b=rep(c(-2,0,2),each=length(theta)))
ggplot(presp,aes(x=theta, y=prob, group=b, colour = b))+
geom_line(size=2) +
ggtitle(paste0("Information response function when c=",cfix, " and a=", afix)) +
xlab(expression(theta))
### Plotting Item response functions
cfix=.2
afix=2
presp <- data.frame(theta=rep(theta,3),
prob=c(Pr(theta, cbind(afix, -2, c=cfix)),
Pr(theta, cbind(afix, 0, c=cfix)),
Pr(theta, cbind(afix, 2, c=cfix))),
b=rep(c(-2,0,2),each=length(theta)))
ggplot(presp,aes(x=theta, y=prob, group=b, colour = b))+
geom_line(size=2) +
ggtitle(paste0("Information response function when c=",cfix, " and a=", afix)) +
xlab(expression(theta))
# Item information with regards to the estimate of person
# is a well known equation.
Itheta <- function(theta=theta, a=1,b=0,c=0, D=1) {
pi <- Pr(theta=theta, it=cbind(a,b,c),D)
a^2 * (pi-c)^2 * (1-pi) / ((1-c)^2*pi)
}
# Calculate the information about each item parameter
Ia <- function(theta=theta, a=1,b=0,c=0, D=1) {
pi <- Pr(theta=theta, it=cbind(a,b,c),D)
D^2/(1-c)^2*(theta-b)^2*(pi-c)^2*(1-pi)/pi
}
Ib <- function(theta=0, a=1,b=0,c=0, D=1) {
pi <- Pr(theta=theta, it=cbind(a,b,c),D)
(D*a*(pi-c)/(1-c))^2*(1-pi)/pi
}
Ic <- function(theta=0, a=1,b=0,c=0, D=1) {
pi <- Pr(theta=theta, it=cbind(a,b,c),D)
1/(1-c)^2*(1-pi)/pi
}
# Let's see these item information functions mapped out
### ---- Map out item information for latent ability theta
cfix=0
afix=1
ainfo <- data.frame(theta=rep(theta,3),
information=c(Itheta(theta, b=-2, a=afix, c=cfix),
Itheta(theta, b=0 , a=afix,c=cfix),
Itheta(theta, b=2 , a=afix,c=cfix)),
b=rep(c(-2,0,2),each=length(theta)))
ggplot(ainfo,aes(x=theta, y=information, group=b, colour = b))+
geom_line(size=2) +
ggtitle(paste0("Information function for a when c=",cfix, " and a=", afix)) +
xlab(expression(theta))
### ---- Map out item information for latent ability theta
cfix=0
afix=2
ainfo <- data.frame(theta=rep(theta,3),
information=c(Itheta(theta, b=-2, a=afix, c=cfix),
Itheta(theta, b=0 , a=afix,c=cfix),
Itheta(theta, b=2 , a=afix,c=cfix)),
b=rep(c(-2,0,2),each=length(theta)))
ggplot(ainfo,aes(x=theta, y=information, group=b, colour = b))+
geom_line(size=2) +
ggtitle(paste0("Information function for a when c=",cfix, " and a=", afix)) +
xlab(expression(theta))
### ---- Map out item information for latent ability theta
cfix=.5
afix=1
ainfo <- data.frame(theta=rep(theta,3),
information=c(Itheta(theta, b=-2, a=afix, c=cfix),
Itheta(theta, b=0 , a=afix,c=cfix),
Itheta(theta, b=2 , a=afix,c=cfix)),
b=rep(c(-2,0,2),each=length(theta)))
ggplot(ainfo,aes(x=theta, y=information, group=b, colour = b))+
geom_line(size=2) +
ggtitle(paste0("Information function for a when c=",cfix, " and a=", afix)) +
xlab(expression(theta))
### ---- Map out item information for parameter a
# c=0
cfix=0
ainfo <- data.frame(theta=rep(theta,3),
information=c(Ia(theta, b=-2, c=cfix),
Ia(theta, b=0 , c=cfix),
Ia(theta, b=2 , c=cfix)),
b=rep(c(-2,0,2),each=length(theta)))
ggplot(ainfo,aes(x=theta, y=information, group=b, colour = b))+
geom_line(size=2) +
ggtitle(paste0("Information function for a when c=",cfix)) +
xlab(expression(theta))
### ---- Map out item information for parameter a
# c=0.15
cfix=.15
ainfo <- data.frame(theta=rep(theta,3),
information=c(Ia(theta, b=-2, c=cfix),
Ia(theta, b=0 , c=cfix),
Ia(theta, b=2 , c=cfix)),
b=rep(c(-2,0,2),each=length(theta)))
ggplot(ainfo,aes(x=theta, y=information, group=b, colour = b))+
geom_line(size=2) +
ggtitle(paste0("Information function for parameter a when c=",cfix)) +
xlab(expression(theta))
### ---- Map out item information for parameter b
# c=0
cfix=0
binfo <- data.frame(theta=rep(theta,3),
information=c(Ib(theta, b=-2, c=cfix),
Ib(theta, b=0 , c=cfix),
Ib(theta, b=2 , c=cfix)),
b=rep(c(-2,0,2),each=length(theta)))
ggplot(binfo,aes(x=theta, y=information, group=b, colour = b))+
geom_line(size=2) +
ggtitle(paste0("Information function for parameter b when c=",cfix)) +
xlab(expression(theta))
### ---- Map out item information for parameter b
# c=0.15
cfix=.15
binfo <- data.frame(theta=rep(theta,3),
information=c(Ib(theta, b=-2, c=cfix),
Ib(theta, b=0 , c=cfix),
Ib(theta, b=2 , c=cfix)),
b=rep(c(-2,0,2),each=length(theta)))
ggplot(binfo,aes(x=theta, y=information, group=b, colour = b))+
geom_line(size=2) +
ggtitle(paste0("Information function for parameter b when c=",cfix)) +
xlab(expression(theta))
### ---- Map out item information for parameter c
# a=1, b=0
afix=1
bfix=0
cinfo <- data.frame(theta=rep(theta,3),
information=c(Ic(theta, c=.15 , a=afix),
Ic(theta, c=.3 , a=afix),
Ic(theta, c=.45 , a=afix)),
c=rep(c(.15,.3,.45),each=length(theta)))
ggplot(cinfo,aes(x=theta, y=information, group=c, colour = c))+
geom_line(size=2) +
ggtitle(
paste0("Information function for parameter c when a=",afix, " and b=", bfix)) +
xlab(expression(theta))
### ---- Map out item information for parameter c
# a=2, b=0
afix=2
bfix=0
cinfo <- data.frame(theta=rep(theta,3),
information=c(Ic(theta, c=.15 , a=afix),
Ic(theta, c=.3 , a=afix),
Ic(theta, c=.45 , a=afix)),
c=rep(c(.15,.3,.45),each=length(theta)))
ggplot(cinfo,aes(x=theta, y=information, group=c, colour = c))+
geom_line(size=2) +
ggtitle(
paste0("Information function for parameter c when a=",afix, " and b=", bfix)) +
xlab(expression(theta))
To leave a comment for the author, please follow the link and comment on their blog: Econometrics by Simulation.
R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.