Game of Chefs is an Israeli reality cooking competition show, where chefs compete for the title of “Israel’s most talented chef”. The show has four stages: blind auditions, training camp, kitchen battles, and finals. Here I will examine the 4th season’s (Game of Chefs: Confectioner) inter-rater reliability in the blind auditions, using some simple R code.

The Format

df <- read.csv('https://github.com/mattansb/blogCode/raw/master/2018_10_18%20Inter-Rater%20Reliability/IJR.csv')
head(df)

##   Moshik Assaf Erez Yossi
## 1      1     1    1     1
## 2      1     0    0     0
## 3      1     1    1     0
## 4      1     0    0     0
## 5      0     1    1     1
## 6      1     1    1     1

We will need the following packages:

library(dplyr) # for manipulating the data
library(psych) # for computing Choen's Kappa
library(corrr) # for manipulating matrices

We can now use the psych package to compute Cohen's Kappa coefficient for inter-rater agreement for categorical items, such as the fail/pass categories we have here. \$\kappa\$ ranges from -1 (full disagreement) through 0 (no pattern of agreement) to +1 (full agreement). Normally, \$\kappa\$ is computed between two raters, and for the case of more than two raters, the mean \$\kappa\$ across all pair-wise raters is used.

cohen.kappa(df)

## Cohen Kappa (below the diagonal) and Weighted Kappa (above the diagonal) 
## For confidence intervals and detail print with all=TRUE
##        Moshik Assaf  Erez  Yossi
## Moshik   1.00  0.34  0.17   0.29
## Assaf    0.34  1.00  0.17   0.68
## Erez     0.17  0.17  1.00  -0.16
## Yossi    0.29  0.68 -0.16   1.00
## 
## Average Cohen kappa for all raters  0.25
## Average weighted kappa for all raters  0.25

We can see that overall \$\kappa=0.25\$ - surprisingly low for what might be expected from a group of pristine, accomplished, professional chefs and bakers in a blind taste test.

When examining the pair-wise coefficients, we can also see that Erez seems to be in lowest agreement with each of the other judges (and even in a slight disagreement with Yossi!). This might be because Erez is new on the show (this is his first season as judge), but it might also be because of the four judges, he is the only one who is actually a confectioner (the other 3 are restaurant chefs).

df <- mutate(df,
             PASS = as.numeric(rowSums(df) >= 3))
head(df)

##   Moshik Assaf Erez Yossi PASS
## 1      1     1    1     1    1
## 2      1     0    0     0    0
## 3      1     1    1     0    1
## 4      1     0    0     0    0
## 5      0     1    1     1    1
## 6      1     1    1     1    1

We can now use the wonderful new corrr package, which is intended for exploring correlations, but can also generally be used to manipulate any symmetric matrix in a tidy-fashion.

cohen.kappa(df)$cohen.kappa %>%
  as_cordf() %>%
  focus(PASS)

## # A tibble: 4 x 2
##   rowname  PASS
##   <chr>   <dbl>
## 1 Moshik  0.619
## 2 Assaf   0.746
## 3 Erez    0.159
## 4 Yossi   0.676

Perhaps unsurprisingly (to people familiar with previous seasons of the show), it seems that Assaf's judgment of a dish is a good indicator of whether or not a candidate will continue on to the next stage. Also, once again we see that Erez is barely aligned with the other judges' total decision.

Conclusion

Every man to his taste...

Even among the experts there is little agreement on what is fine cuisine and what is not worth a doggy bag. Having said that, if you still have your heart set on competing in Game of Chefs, it seems that you should at least appeal to Assaf's palate.

Bon Appétit!