Scenario 1: Independent samples with 2 known variances

For the first scenario, suppose the data below. Moreover, suppose that the two samples are independent, that the variances \(\sigma^2 = 1\) in both populations and that we would like to test whether the two populations are different.

So we have:

• 5 observations in each sample: \(n_1 = n_2 = 5\)
• mean of sample 1: \(\bar{x}_1 = 0.02\)
• mean of sample 2: \(\bar{x}_2 = 0.06\)
• variances of both populations: \(\sigma^2_1 = \sigma^2_2 = 1\)

Following the 4 steps of hypothesis testing we have:

1. \(H_0: \mu_1 = \mu_2\) and \(H_1: \mu_1 – \mu_2 \ne 0\). (\(\ne\) because we want to test whether the two means are different, we do not impose a direction in the test.)
2. Test statistic: \[z_{obs} = \frac{(\bar{x}_1 – \bar{x}_2) – (\mu_1 – \mu_2)}{\sqrt{\frac{\sigma^2_1}{n_1} + \frac{\sigma^2_2}{n_2}}} = \frac{0.02-0.06-0}{0.632} = -0.063\]
3. Critical value: \(\pm z_{\alpha / 2} = \pm z_{0.025} = \pm 1.96\) (see a guide on how to read statistical tables if you struggle to find the critical value)
4. Conclusion: The rejection regions are thus from \(-\infty\) to -1.96 and from 1.96 to \(+\infty\). The test statistic is outside the rejection regions so we do not reject the null hypothesis \(H_0\). In terms of the initial question: At the 5% significance level, we do not reject the hypothesis that the two populations are the same, or there is no sufficient evidence in the data to conclude that the two populations considered are different.

Scenario 2: Independent samples with 2 equal but unknown variances

For the second scenario, suppose the data below. Moreover, suppose that the two samples are independent, that the variances in both populations are unknown but equal (\(\sigma^2_1 = \sigma^2_1\)) and that we would like to test whether population 1 is larger than population 2.

So we have:

• 6 observations in sample 1: \(n_1 = 6\)
• 5 observations in sample 2: \(n_2 = 5\)
• mean of sample 1: \(\bar{x}_1 = 1.247\)
• mean of sample 2: \(\bar{x}_2 = 0.1\)
• variance of sample 1: \(s^2_1 = 0.303\)
• variance of sample 2: \(s^2_1 = 0.315\)

Following the 4 steps of hypothesis testing we have:

1. \(H_0: \mu_1 = \mu_2\) and \(H_1: \mu_1 – \mu_2 > 0\). (> because we want to test if the mean of the first population is larger than the mean of the second population.)
2. Test statistic: \[t_{obs} = \frac{(\bar{x}_1 – \bar{x}_2) – (\mu_1 – \mu_2)}{s_p\sqrt{\frac{1}{n_1} + \frac{1}{n_2}}}\] where \[s_p = \sqrt{\frac{(n_1-1)s^2_1+ (n_2 – 1)s^2_2}{n_1 + n_2 – 2}} = 0.555\] so \[t_{obs} = \frac{1.247-0.1-0}{0.555 * 0.606} = 3.411\] (Note that as it is assumed the variances of the two populations are equal, a pooled (common) variance, denoted \(s_p\), is computed.)
3. Critical value: \(t_{\alpha, n_1 + n_2 – 2} = t_{0.05, 9} = 1.833\)
4. Conclusion: The rejection region is thus from 1.833 to \(+\infty\) (there is only one rejection region because it is a one-sided test). The test statistic lies within the rejection region so we reject the null hypothesis \(H_0\). In terms of the initial question: At the 5% significance level, we conclude that the population 1 is larger than the population 2.

Scenario 3: Independent samples with 2 unequal and unknown variances

For the third scenario, suppose the data below. Moreover, suppose that the two samples are independent, that the variances in both populations are unknown and unequal (\(\sigma^2_1 \ne \sigma^2_1\)) and that we would like to test whether population 1 is smaller than population 2.

So we have:

• 5 observations in sample 1: \(n_1 = 5\)
• 6 observations in sample 2: \(n_2 = 6\)
• mean of sample 1: \(\bar{x}_1 = 0.42\)
• mean of sample 2: \(\bar{x}_2 = 1.247\)
• variance of sample 1: \(s^2_1 = 0.107\)
• variance of sample 2: \(s^2_1 = 0.303\)

Following the 4 steps of hypothesis testing we have:

1. \(H_0: \mu_1 = \mu_2\) and \(H_1: \mu_1 – \mu_2 < 0\). (< because we want to test if the mean of the first population is smaller than the mean of the second population.)
2. Test statistic: \[t_{obs} = \frac{(\bar{x}_1 – \bar{x}_2) – (\mu_1 – \mu_2)}{\sqrt{\frac{s^2_1}{n_1} + \frac{s^2_2}{n_2}}} = \frac{0.42-1.247-0}{0.268} = -3.084\]
3. Critical value: \(-t_{\alpha, \upsilon}\) where \[\upsilon = \frac{\bigg(\frac{s^2_1}{n_1} + \frac{s^2_2}{n_2} \bigg)^2}{\frac{\bigg(\frac{s^2_1}{n_1}\bigg)^2}{n_1 – 1} + \frac{\bigg(\frac{s^2_2}{n_2}\bigg)^2}{n_2 – 1}} = 8.28\] so \[-t_{0.05, 8.28} = -1.851\] The degrees of freedom 8.28 does not exist in the standard Student distribution table, so simply take 8, or compute it in R with qt(p = 0.05, df = 8.28).
4. Conclusion: The rejection region is thus from \(-\infty\) to -1.851. The test statistic lies within the rejection region so we reject the null hypothesis \(H_0\). In terms of the initial question: At the 5% significance level, we conclude that the population 1 is smaller than the population 2.

Scenario 4: Paired samples where the variance of the differences is known

Student’s t-test with paired samples are a bit different than with independent samples, they are actually more similar to one sample Student’s t-test. Here is how it works. We actually compute the difference between the two samples for each pair of observations, and then we work on these differences as if we were doing a one sample Student’s t-test by computing the test statistic on these differences.

In case it is not clear, here is the fourth scenario as an illustration. Suppose the data below. Moreover, suppose that the two samples are dependent (matched), that the variance of the differences in the population is known and equal to 1 (\(\sigma^2_D = 1\)) and that we would like to test whether the difference in the population is different than 0.

The first thing to do is to compute the differences for all pairs of observations:

So we have:

• number of pairs: \(n = 5\)
• mean of the difference: \(\bar{D} = 0.04\)
• variance of the difference in the population: \(\sigma^2_D = 1\)
• standard deviation of the difference in the population: \(\sigma_D = 1\)

Following the 4 steps of hypothesis testing we have:

1. \(H_0: \mu_D = 0\) and \(H_1: \mu_D \ne 0\)
2. Test statistic: \[z_{obs} = \frac{\bar{D} – \mu_0}{\frac{\sigma_D}{\sqrt{n}}} = \frac{0.04-0}{0.447} = 0.089\] (This formula is exactly the same than for one sample Student’s t-test with a known variance, except that we work on the mean of the differences.)
3. Critical value: \(\pm z_{\alpha/2} = \pm z_{0.025} = \pm 1.96\)
4. Conclusion: The rejection regions are thus from \(-\infty\) to -1.96 and from 1.96 to \(+\infty\). The test statistic is outside the rejection regions so we do not reject the null hypothesis \(H_0\). In terms of the initial question: At the 5% significance level, we do not reject the hypothesis that the difference in the two populations is equal to 0.

Scenario 5: Paired samples where the variance of the differences is unknown

For the fifth and final scenario, suppose the data below. Moreover, suppose that the two samples are dependent (matched), that the variance of the differences in the population is unknown and that we would like to test whether a treatment is effective in increasing running capabilities (the higher the value, the better in terms of running capabilities).

The first thing to do is to compute the differences for all pairs of observations:

So we have:

• number of pairs: \(n = 5\)
• mean of the difference: \(\bar{D} = 8\)
• variance of the difference in the sample: \(s^2_D = 16\)
• standard deviation of the difference in the sample: \(s_D = 4\)

Following the 4 steps of hypothesis testing we have:

1. \(H_0: \mu_D = 0\) and \(H_1: \mu_D > 0\) (> because we would like to test whether the treatment is effective, so whether the treatment has a positive impact on the running capabilities.)
2. Test statistic: \[t_{obs} = \frac{\bar{D} – \mu_0}{\frac{s_D}{\sqrt{n}}} = \frac{8-0}{1.789} = 4.472\] (This formula is exactly the same than for one sample Student’s t-test with an unknown variance, except that we work on the mean of the differences.)
3. Critical value: \(t_{\alpha, n-1} = t_{0.05, 4} = 2.132\) (n is the number of pairs, not the number of observations!)
4. Conclusion: The rejection regions are thus from 2.132 to \(+\infty\). The test statistic lies within the rejection region so we reject the null hypothesis \(H_0\). In terms of the initial question: At the 5% significance level, we conclude that the treatment has a positive impact on the running capabilities.

This concludes how to perform the different versions of the Student’s t-test for two samples by hand. In the next sections, we detail how to perform the exact same tests in R.

Scenario 1: Independent samples with 2 known variances

For the first scenario, suppose the data below. Moreover, suppose that the two samples are independent, that the variances \(\sigma^2 = 1\) in both populations and that we would like to test whether the two populations are different.

dat1 <- data.frame(
  sample1 = c(0.9, -0.8, 0.1, -0.3, 0.2),
  sample2 = c(0.8, -0.9, -0.1, 0.4, 0.1)
)
dat1
##   sample1 sample2
## 1     0.9     0.8
## 2    -0.8    -0.9
## 3     0.1    -0.1
## 4    -0.3     0.4
## 5     0.2     0.1
dat_ggplot <- data.frame(
  value = c(0.9, -0.8, 0.1, -0.3, 0.2, 0.8, -0.9, -0.1, 0.4, 0.1),
  sample = c(rep("1", 5), rep("2", 5))
)

library(ggplot2)

ggplot(dat_ggplot) +
  aes(x = sample, y = value) +
  geom_boxplot() +
  theme_minimal()

Note that you can use the {esquisse} RStudio addin if you want to draw a boxplot with the package {ggplot2} without writing the code yourself. If you prefer the default graphics, use the boxplot() function:

boxplot(value ~ sample,
  data = dat_ggplot
)

The two boxes seem to overlap which illustrate that the two samples are quite similar, so we tend to believe that we will not be able to reject the null hypothesis that the two populations are similar. However, only a formal statistical test will confirm this belief.

Since there is no function in R to perform a t-test with known variances, here is one with arguments accepting the two samples (x and y), the two variances of the populations (V1 and V2), the difference in means under the null hypothesis (m0, default is 0), the significance level (alpha, default is 0.05) and the alternative (alternative, one of "two.sided" (default), "less" or "greater"):

t.test_knownvar <- function(x, y, V1, V2, m0 = 0, alpha = 0.05, alternative = "two.sided") {
  M1 <- mean(x)
  M2 <- mean(y)
  n1 <- length(x)
  n2 <- length(y)
  sigma1 <- sqrt(V1)
  sigma2 <- sqrt(V2)
  S <- sqrt((V1 / n1) + (V2 / n2))
  statistic <- (M1 - M2 - m0) / S
  p <- if (alternative == "two.sided") {
    2 * pnorm(abs(statistic), lower.tail = FALSE)
  } else if (alternative == "less") {
    pnorm(statistic, lower.tail = TRUE)
  } else {
    pnorm(statistic, lower.tail = FALSE)
  }
  LCL <- (M1 - M2 - S * qnorm(1 - alpha / 2))
  UCL <- (M1 - M2 + S * qnorm(1 - alpha / 2))
  value <- list(mean1 = M1, mean2 = M2, m0 = m0, sigma1 = sigma1, sigma2 = sigma2, S = S, statistic = statistic, p.value = p, LCL = LCL, UCL = UCL, alternative = alternative)
  # print(sprintf("P-value = %g",p))
  # print(sprintf("Lower %.2f%% Confidence Limit = %g",
  #               alpha, LCL))
  # print(sprintf("Upper %.2f%% Confidence Limit = %g",
  #               alpha, UCL))
  return(value)
}

test <- t.test_knownvar(dat1$sample1, dat1$sample2,
  V1 = 1, V2 = 1
)
test
## $mean1
## [1] 0.02
## 
## $mean2
## [1] 0.06
## 
## $m0
## [1] 0
## 
## $sigma1
## [1] 1
## 
## $sigma2
## [1] 1
## 
## $S
## [1] 0.6324555
## 
## $statistic
## [1] -0.06324555
## 
## $p.value
## [1] 0.949571
## 
## $LCL
## [1] -1.27959
## 
## $UCL
## [1] 1.19959
## 
## $alternative
## [1] "two.sided"

The output above recaps all the information needed to perform the test: the test statistic, the p-value, the alternative used, the two sample means and the two variances of the populations (compare these results found in R with the results found by hand).

The p-value can be extracted as usual:

test$p.value
## [1] 0.949571

The p-value is 0.95 so at the 5% significance level we do not reject the null hypothesis of equal means. There is no sufficient evidence in the data to reject the hypothesis that the two means in the populations are similar. This result confirms what we found by hand.

Scenario 2: Independent samples with 2 equal but unknown variances

For the second scenario, suppose the data below. Moreover, suppose that the two samples are independent, that the variances in both populations are unknown but equal (\(\sigma^2_1 = \sigma^2_1\)) and that we would like to test whether population 1 is larger than population 2.

dat2 <- data.frame(
  sample1 = c(1.78, 1.5, 0.9, 0.6, 0.8, 1.9),
  sample2 = c(0.8, -0.7, -0.1, 0.4, 0.1, NA)
)
dat2
##   sample1 sample2
## 1    1.78     0.8
## 2    1.50    -0.7
## 3    0.90    -0.1
## 4    0.60     0.4
## 5    0.80     0.1
## 6    1.90      NA
dat_ggplot <- data.frame(
  value = c(1.78, 1.5, 0.9, 0.6, 0.8, 1.9, 0.8, -0.7, -0.1, 0.4, 0.1),
  sample = c(rep("1", 6), rep("2", 5))
)

ggplot(dat_ggplot) +
  aes(x = sample, y = value) +
  geom_boxplot() +
  theme_minimal()

Unlike the previous scenario, the two boxes do not overlap which illustrates that the two samples are different from each other. From this boxplot, we can expect the test to reject the null hypothesis of equal means in the populations. Nonetheless, only a formal statistical test will confirm this expectation.

There is a function in R, and it is simply the t.test() function. This version of the test is actually the “standard” Student’s t-test for two samples. Note that it is assumed that the variances of the two populations are equal so we need to specify it in the function with the argument var.equal = TRUE (the default is FALSE) and the alternative hypothesis is \(H_1: \mu_1 - \mu_2 > 0\) so we need to add the argument alternative = "greater" as well:

test <- t.test(dat2$sample1, dat2$sample2,
  var.equal = TRUE, alternative = "greater"
)
test
## 
##  Two Sample t-test
## 
## data:  dat2$sample1 and dat2$sample2
## t = 3.4113, df = 9, p-value = 0.003867
## alternative hypothesis: true difference in means is greater than 0
## 95 percent confidence interval:
##  0.5304908       Inf
## sample estimates:
## mean of x mean of y 
##  1.246667  0.100000

The output above recaps all the information needed to perform the test: the name of the test, the test statistic, the degrees of freedom, the p-value, the alternative used and the two sample means (compare these results found in R with the results found by hand).

The p-value can be extracted as usual:

test$p.value
## [1] 0.003866756

The p-value is 0.004 so at the 5% significance level we reject the null hypothesis of equal means. This result confirms what we found by hand.

Unlike the first scenario, the p-value in this scenario is below 5% so we reject the null hypothesis. At the 5% significance level, we can conclude that the population 1 is larger than the population 2.

If your data is formatted in the long format (which is even better), simply use the ~. For instance, imagine the exact same data presented like this:

dat2bis <- data.frame(
  value = c(1.78, 1.5, 0.9, 0.6, 0.8, 1.9, 0.8, -0.7, -0.1, 0.4, 0.1),
  sample = c(rep("1", 6), rep("2", 5))
)
dat2bis
##    value sample
## 1   1.78      1
## 2   1.50      1
## 3   0.90      1
## 4   0.60      1
## 5   0.80      1
## 6   1.90      1
## 7   0.80      2
## 8  -0.70      2
## 9  -0.10      2
## 10  0.40      2
## 11  0.10      2

Here is how to perform the Student’s t-test in R with long data:

test <- t.test(value ~ sample,
  data = dat2bis,
  var.equal = TRUE,
  alternative = "greater"
)
test
## 
##  Two Sample t-test
## 
## data:  value by sample
## t = 3.4113, df = 9, p-value = 0.003867
## alternative hypothesis: true difference in means is greater than 0
## 95 percent confidence interval:
##  0.5304908       Inf
## sample estimates:
## mean in group 1 mean in group 2 
##        1.246667        0.100000
test$p.value
## [1] 0.003866756

The results are exactly the same.

Scenario 3: Independent samples with 2 unequal and unknown variances

For the third scenario, suppose the data below. Moreover, suppose that the two samples are independent, that the variances in both populations are unknown and unequal (\(\sigma^2_1 \ne \sigma^2_1\)) and that we would like to test whether population 1 is smaller than population 2.

dat3 <- data.frame(
  value = c(0.8, 0.7, 0.1, 0.4, 0.1, 1.78, 1.5, 0.9, 0.6, 0.8, 1.9),
  sample = c(rep("1", 5), rep("2", 6))
)
dat3
##    value sample
## 1   0.80      1
## 2   0.70      1
## 3   0.10      1
## 4   0.40      1
## 5   0.10      1
## 6   1.78      2
## 7   1.50      2
## 8   0.90      2
## 9   0.60      2
## 10  0.80      2
## 11  1.90      2
ggplot(dat3) +
  aes(x = sample, y = value) +
  geom_boxplot() +
  theme_minimal()

There is a function in R for this version of the test as well, and it is simply the t.test() function with the var.equal = FALSE argument. FALSE is the default option for the var.equal argument so you actually do not need to specify it. This version of the test is actually the Welch test, used when the variances of the populations are unknown and unequal. To test if two variances are equal, you can use the Levene’s test (leveneTest(dat3$value, dat3$sample) from the {car} package). Note that the alternative hypothesis is \(H_1: \mu_1 - \mu_2 < 0\) so we need to add the argument alternative = "less" as well:

test <- t.test(value ~ sample,
  data = dat3,
  var.equal = FALSE,
  alternative = "less"
)
test
## 
##  Welch Two Sample t-test
## 
## data:  value by sample
## t = -3.0841, df = 8.2796, p-value = 0.007206
## alternative hypothesis: true difference in means is less than 0
## 95 percent confidence interval:
##        -Inf -0.3304098
## sample estimates:
## mean in group 1 mean in group 2 
##        0.420000        1.246667

The output above recaps all the information needed to perform the test (compare these results found in R with the results found by hand).

The p-value can be extracted as usual:

test$p.value
## [1] 0.00720603

The p-value is 0.007 so at the 5% significance level we reject the null hypothesis of equal means, meaning that we can conclude that the population 1 is smaller than the population 2. This result confirms what we found by hand.

Scenario 4: Paired samples where the variance of the differences is known

For the fourth scenario, suppose the data below. Moreover, suppose that the two samples are dependent (matched), that the variance of the differences in the population is known and equal to 1 (\(\sigma^2_D = 1\)) and that we would like to test whether the difference in the population is different than 0.

dat4 <- data.frame(
  before = c(0.9, -0.8, 0.1, -0.3, 0.2),
  after = c(0.8, -0.9, -0.1, 0.4, 0.1)
)
dat4
##   before after
## 1    0.9   0.8
## 2   -0.8  -0.9
## 3    0.1  -0.1
## 4   -0.3   0.4
## 5    0.2   0.1
dat4$difference <- dat4$after - dat4$before

ggplot(dat4) +
  aes(y = difference) +
  geom_boxplot() +
  theme_minimal()

Since there is no function in R to perform a t-test with paired samples where the variance of the differences is known, here is one with arguments accepting the differences between the two samples (x), the variance of the differences in the population (V), the mean of the differences under the null hypothesis (m0, default is 0), the significance level (alpha, default is 0.05) and the alternative (alternative, one of "two.sided" (default), "less" or "greater"):

t.test_pairedknownvar <- function(x, V, m0 = 0, alpha = 0.05, alternative = "two.sided") {
  M <- mean(x)
  n <- length(x)
  sigma <- sqrt(V)
  S <- sqrt(V / n)
  statistic <- (M - m0) / S
  p <- if (alternative == "two.sided") {
    2 * pnorm(abs(statistic), lower.tail = FALSE)
  } else if (alternative == "less") {
    pnorm(statistic, lower.tail = TRUE)
  } else {
    pnorm(statistic, lower.tail = FALSE)
  }
  LCL <- (M - S * qnorm(1 - alpha / 2))
  UCL <- (M + S * qnorm(1 - alpha / 2))
  value <- list(mean = M, m0 = m0, sigma = sigma, statistic = statistic, p.value = p, LCL = LCL, UCL = UCL, alternative = alternative)
  # print(sprintf("P-value = %g",p))
  # print(sprintf("Lower %.2f%% Confidence Limit = %g",
  #               alpha, LCL))
  # print(sprintf("Upper %.2f%% Confidence Limit = %g",
  #               alpha, UCL))
  return(value)
}

test <- t.test_pairedknownvar(dat4$after - dat4$before,
  V = 1
)
test
## $mean
## [1] 0.04
## 
## $m0
## [1] 0
## 
## $sigma
## [1] 1
## 
## $statistic
## [1] 0.08944272
## 
## $p.value
## [1] 0.9287301
## 
## $LCL
## [1] -0.8365225
## 
## $UCL
## [1] 0.9165225
## 
## $alternative
## [1] "two.sided"

The output above recaps all the information needed to perform the test (compare these results found in R with the results found by hand).

The p-value can be extracted as usual:

test$p.value
## [1] 0.9287301

The p-value is 0.929 so at the 5% significance level we do not reject the null hypothesis of the mean of the differences being equal to 0. There is no sufficient evidence in the data to reject the hypothesis that the difference in the two populations is equal to 0. This result confirms what we found by hand.

Scenario 5: Paired samples where the variance of the differences is unknown

For the fifth and final scenario, suppose the data below. Moreover, suppose that the two samples are dependent (matched), that the variance of the differences in the population is unknown and that we would like to test whether a treatment is effective in increasing running capabilities (the higher the value, the better in terms of running capabilities).

dat5 <- data.frame(
  before = c(9, 8, 1, 3, 2),
  after = c(16, 11, 15, 12, 9)
)
dat5
##   before after
## 1      9    16
## 2      8    11
## 3      1    15
## 4      3    12
## 5      2     9
dat5$difference <- dat5$after - dat5$before

ggplot(dat5) +
  aes(y = difference) +
  geom_boxplot() +
  theme_minimal()

There is a function in R for this version of the test, and it is simply the t.test() function with the paired = TRUE argument. This version of the test is actually the standard version of the Student’s t-test with paired samples. Note that the alternative hypothesis is \(H_1: \mu_D > 0\) so we need to add the argument alternative = "greater" as well:

test <- t.test(dat5$after, dat5$before,
  alternative = "greater",
  paired = TRUE
)
test
## 
##  Paired t-test
## 
## data:  dat5$after and dat5$before
## t = 4.4721, df = 4, p-value = 0.005528
## alternative hypothesis: true difference in means is greater than 0
## 95 percent confidence interval:
##  4.186437      Inf
## sample estimates:
## mean of the differences 
##                       8

Note that we wrote after and then before in this order. If you write before and then after, make sure to change the alternative to alternative = "less".

If your data is in the long format, use the ~:

dat5 <- data.frame(
  value = c(9, 8, 1, 3, 2, 16, 11, 15, 12, 9),
  time = c(rep("before", 5), rep("after", 5))
)
dat5
##    value   time
## 1      9 before
## 2      8 before
## 3      1 before
## 4      3 before
## 5      2 before
## 6     16  after
## 7     11  after
## 8     15  after
## 9     12  after
## 10     9  after
test <- t.test(value ~ time,
  data = dat5,
  alternative = "greater",
  paired = TRUE
)
test
## 
##  Paired t-test
## 
## data:  value by time
## t = 4.4721, df = 4, p-value = 0.005528
## alternative hypothesis: true difference in means is greater than 0
## 95 percent confidence interval:
##  4.186437      Inf
## sample estimates:
## mean of the differences 
##                       8

The output above recaps all the information needed to perform the test (compare these results found in R with the results found by hand).

The p-value can be extracted as usual:

test$p.value
## [1] 0.005528247

The p-value is 0.006 so at the 5% significance level we reject the null hypothesis of the mean of the differences being equal to 0, meaning that we can conclude that the treatment is effective in increasing the running capabilities. This result confirms what we found by hand.