Submission Correctness Testing in DataCamp DataCamp (Part I)
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.
This blog post will walk you through some basic examples of building Submission Correctness Tests (SCTs) so that you can begin to create your own interactive courses on DataCamp. Together we will walk through six basic functions and see how they work. You will be able to see how to write SCTs in the R code chunks and follow along in the interactive exercises to see their output. We will be using the same exercise for each example, but you will notice some changes along the way as we improve our SCT code in the background! The correct answers are already filled in for you, but try to mess around with them to see how the SCT performs.
First, let’s introduce the exercise we will be working with today. We have created an exercise below with three separate instructions. We are asking the student to do the following:
-
Create a vector called
my_vectcontaining 1 through 5. -
Take the mean of
my_vectand assign it to a variable namedmy_mean. Use themean()function. -
Use
my_meanto compute the sum of squares ofmy_vect. Use thesum()function.
# Create a vector called my_vect containing 1 through 5 my_vect <- c(1, 2, 3, 4, 5) # Take the mean and assign to my_mean my_mean <- mean(my_vect) # Use my_mean to compute the sum of squares of my_vect sum((my_vect - my_mean)^2)
The code chuck above represents the solution to the exercise. Now we need to build some tests to make sure our students are entering the correct response. Let’s make some SCTs!
The first function that every exercise should have is called the success_msg(). Within this function, you will write the message that you want to appear when the student gets the problem correct. Remember our problem from above? We’ve added a success message behind the scenes.
# sct code
success_msg("Nice work! Success messages are a great way to keep users engaged even after the exercise is over. Try to encourage them to play around in the console to really grasp the concepts you're trying to teach!")
Now try the exercise below. First, click submit answer without changing anything in the code. You should see a success message. After, try purposefully getting the answer wrong. What happens?
# pec # Create a vector called my_vect containing 1 through 5 my_vect <- c(1, 2, 3, 4, 5) # Take the mean and assign to my_mean my_mean <- mean(my_vect) # Use my_mean to compute the sum of squares of my_vect sum((my_vect - my_mean)^2) # Create a vector called my_vect containing 1 through 5 my_vect <- c(1, 2, 3, 4, 5) # Take the mean and assign to my_mean my_mean <- mean(my_vect) # Use my_mean to compute the sum of squares of my_vect sum((my_vect - my_mean)^2) success_msg("Nice work! Success messages are a great way to keep users engaged even after the exercise is over. Try to encourage them to play around in the console to really grasp the concepts you're trying to teach!")
Notice that no matter what you enter in the script, when you click submit answer it will pass the test. Why? Because we haven’t added any other SCTs and without any other tests to pass, it passes by default. Of course, this alone isn’t very helpful, so let’s learn a couple more functions!
test_error() is another great function that should be included in every single SCT section that you build. It simply tests the R script to ensure that the code contains no syntax errors.
We have the same problem from before, but we’ve added a test_error() function behind the scenes. Try to purposefully create a syntax error here and see how test_error is able to catch it.
# sct code
test_error()
success_msg("Nice job! `test_error` and `success_msg` are two functions that should be present in every SCT you build.")
# pec # Create a vector called my_vect containing 1 through 5 my_vect <- c(1, 2, 3, 4, 5) # Take the mean and assign to my_mean my_mean <- mean(my_vect) # Use my_mean to compute the sum of squares of my_vect sum((my_vect - my_mean)^2) # Create a vector called my_vect containing 1 through 5 my_vect <- c(1, 2, 3, 4, 5) # Take the mean and assign to my_mean my_mean <- mean(my_vect) # Use my_mean to compute the sum of squares of my_vect sum((my_vect - my_mean)^2) test_error() success_msg("Nice job! `test_error` and `success_msg` are two functions that should be present in every SCT you build.")
Okay, now we’ve got two functions in our SCT for this exercise, but we need to add some more. We want to make sure that the student is actually entering the correct answer to the question. To start off, we can use the most strict SCT function - test_student_typed(). This function will only pass if the student’s entry is exactly the same as the entry in the solution script. We’ve added a test_student_typed() function to test the solution for each of our instructions.
# sct code
# first instruction
test_student_typed("my_vect <- c(1,2,3,4,5)", not_typed_msg = "Something is wrong with `my_vect`. Take another look at the instruction.")
# second instruction
test_student_typed("my_mean <- mean(my_vect)", not_typed_msg = "Something is wrong with `my_mean`.")
# third instruction
test_student_typed("sum((my_vect - my_mean)^2)", not_typed_msg = "Take a look at your code for the third instruction. To calculate the sum of squares you will need to take the sum of the square of the difference between `my_vect` and `my_mean`.")
# General
test_error()
success_msg("`test_student_typed` can be useful in certain situations, but it should be avoided in general since there are often multiple ways of doing the same thing in R. A student might be doing the problem correctly, but this SCT will tell them they are wrong unless they enter it exactly as you have.")
While this function is useful at times, it can cause some problems. Anyone familiar with R knows that there are often a number of ways of doing the same thing. For example, instead of creating your vector in the first instruction like 1,2,3,4,5, try entering 1:5 which is also correct.
# pec # Create a vector called my_vect containing 1 through 5 my_vect <- c(1, 2, 3, 4, 5) # Take the mean and assign to my_mean my_mean <- mean(my_vect) # Use my_mean to compute the sum of squares of my_vect sum((my_vect - my_mean)^2) # Create a vector called my_vect containing 1 through 5 my_vect <- c(1, 2, 3, 4, 5) # Take the mean and assign to my_mean my_mean <- mean(my_vect) # Use my_mean to compute the sum of squares of my_vect sum((my_vect - my_mean)^2) # first instruction test_student_typed("my_vect <- c(1,2,3,4,5)", not_typed_msg = "Something is wrong with `my_vect`. Take another look at the instruction.") # second instruction test_student_typed("my_mean <- mean(my_vect)", not_typed_msg = "Something is wrong with `my_mean`.") # third instruction test_student_typed("sum((my_vect - my_mean)^2)", not_typed_msg = "Take a look at your code for the third instruction. To calculate the sum of squares you will need to take the sum of the square of the difference between `my_vect` and `my_mean`.") # General test_error() success_msg("`test_student_typed` can be useful in certain situations, but it should be avoided in general since there are often multiple ways of doing the same thing in R. A student might be doing the problem correctly, but this SCT will tell them they are wrong unless they enter it exactly as you have.")
Do you understand why this could be a problem? While test_student_typed() can be useful, it’s best to use it sparingly as it can cause some unnecessary frustration on behalf of the student. Since there are a couple of ways to solve this exercise, test_student_typed() is probably not the best function to use.
Let’s try a different function called test_function(). This tests to make sure that the student is using a particular function. This is helpful when teaching them a new function in your course to ensure that they are actually using it. In addition to the function name, you can also test to make sure that there the student has specified certain arguments. We have added an SCT for each instruction in the exercise using test_function().
Notice how test_function() also has an argument incorrect_msg =. Here we can enter a message that will apprear if this test fails. Keep in mind that if a submission fails the first instruction, the error messages for the second and third will not appear.
# sct code
# first instruction
test_function("c", incorrect_msg = "Something is wrong with `my_vect`. Did you create a vector using `c`?.")
# second instruction
test_function("mean", incorrect_msg = "Something is wrong with `my_mean`. Did you use the `mean` function?")
# third instruction
test_function("sum", incorrect_msg = "Take a look at your code for the third instruction. To calculate the sum of squares you will need to take the sum of the square of the difference between `my_vect` and `my_mean`. You should use the `sum` function.")
# General
test_error()
success_msg("`test_function` can be very helpful, but again it might be a little too rigid in some cases. For example, in the second instuction, what if a student calculated the sum by adding the values in the vector and dividing by 5? This is still correct, but since we were looking for the `mean()` function, it will tell them that it is wrong.")
In this example, test_function() doesn’t catch the error. While it does test whether or not we have used the appropriate function, it does not test whether the actual output of this function is the output that we are looking for from our students. For example, try entering 6:10 in the first instruction instead of 1,2,3,4,5.
# pec # Create a vector called my_vect containing 1 through 5 my_vect <- c(1, 2, 3, 4, 5) # Take the mean and assign to my_mean my_mean <- mean(my_vect) # Use my_mean to compute the sum of squares of my_vect sum((my_vect - my_mean)^2) # Create a vector called my_vect containing 1 through 5 my_vect <- c(1, 2, 3, 4, 5) # Take the mean and assign to my_mean my_mean <- mean(my_vect) # Use my_mean to compute the sum of squares of my_vect sum((my_vect - my_mean)^2) # first instruction test_function("c", incorrect_msg = "Something is wrong with `my_vect`. Take another look at the instruction.") # second instruction test_function("mean", incorrect_msg = "Something is wrong with `my_mean`.") # third instruction test_function("sum", incorrect_msg = "Take a look at your code for the third instruction. To calculate the sum of squares you will need to take the sum of the square of the difference between `my_vect` and `my_mean`.") # General test_error() success_msg("`test_function` can be very helpful, but again it might be a little too rigid in some cases. For example, in the second instuction, what if a student calculated the sum by adding the values in the vector and dividing by 5? This is still correct, but since we were looking for the `mean()` function, it will tell them that it is wrong.")
- Set
my_vecttoc(6:10) - Avoid the
mean()function
See how test_function() failed to pick up that the values in our vector were incorrect? It merely looked to make sure we were calling c(). You need to be careful when using test_function() to avoid circumstances like this. Keep in mind, you can also test for specific arguments within test_function() - this makes it very powerful!
Another option we can use is called test_object()! test_object() will check whether the value of a variable matches the value of the variable in the solution script. This provides some more flexibility in the exercise. It is a great function to use if you are only concerned with the correct output and not necessarily with how the student got there. Again, the exercise has been updated to reflect the new SCT function. The first two instructions are now being tested with test_object().
# sct code
# first instruction
test_object("my_vect", incorrect_msg = "Something is wrong with `my_vect`. Make sure you've assigned the correct value to the variable.")
# second instruction
test_object("my_mean", incorrect_msg = "Something is wrong with `my_mean`. Make sure you've assigned the correct value to the variable.")
# third instruction
test_function("sum", incorrect_msg = "Take a look at your code for the third instruction. To calculate the sum of squares you will need to take the sum of the square of the difference between `my_vect` and `my_mean`.")
# General
test_error()
success_msg("Great! `test_object` worked great for instructions 1 & 2. It give the student flexibilty in how they go about solving the problem, but it still checks the value of the variable they create to ensure that is is correct. But what about the third instruction?")
Remember how last time test_function() failed to notice that we had switched 1,2,3,4,5 to 6:10? Let’s see if we can fool test_object() - do the same thing here!
# pec # Create a vector called my_vect containing 1 through 5 my_vect <- c(1, 2, 3, 4, 5) # Take the mean and assign to my_mean my_mean <- mean(my_vect) # Use my_mean to compute the sum of squares of my_vect sum((my_vect - my_mean)^2) # Create a vector called my_vect containing 1 through 5 my_vect <- c(1, 2, 3, 4, 5) # Take the mean and assign to my_mean my_mean <- mean(my_vect) # Use my_mean to compute the sum of squares of my_vect sum((my_vect - my_mean)^2) # first instruction test_object("my_vect", incorrect_msg = "Something is wrong with `my_vect`. Take another look at the instruction.") # second instruction test_object("my_mean", incorrect_msg = "Something is wrong with `my_mean`. The easiest way to complete this instruction is to use the `mean()` function with `my_vect`.") # third instruction test_function("sum", incorrect_msg = "Take a look at your code for the third instruction. To calculate the sum of squares you will need to take the sum of the square of the difference between `my_vect` and `my_mean`.") # General test_error() success_msg("Great! `test_object` worked great for instructions 1 & 2. It give the student flexibilty in how they go about solving the problem, but it still checks the value of the variable they create to ensure that is is correct. But what about the third instruction?")
This is great, but what about our third instruction? Since we didn’t instruct the user to assign the sum of squares to a variable, we don’t have an “object” to use test_object() on. This is where test_output_contains() comes in. This function evaluates the first argument and sees if the value of the expression is contained in the output of the student’s code. Let’s see how this works below.
# sct code
# first instruction
test_object("my_vect", incorrect_msg = "Something is wrong with `my_vect`. Take another look at the instruction.")
# second instruction
test_object("my_mean", incorrect_msg = "Something is wrong with `my_mean`. The easiest way to complete this instruction is to use the `mean()` function with `my_vect`.")
# third instruction
test_output_contains("sum((my_vect - my_mean)^2)", incorrect_msg = "Take a look at your code for the third instruction. To calculate the sum of squares you will need to take the sum of the square of the difference between `my_vect` and `my_mean`.")
# General
test_error()
success_msg("Good job! You've learned some very important and useful functions in this course. Writing SCT's can get complex, but mastering the functions covered in this exercise will get you started so that you can begin creating your own content.")
Let’s try to fool test_output_contains(). Add + 1 at the end of sum((my_vect - my_mean)^2).
# pec # Create a vector called my_vect containing 1 through 5 my_vect <- c(1, 2, 3, 4, 5) # Take the mean and assign to my_mean my_mean <- mean(my_vect) # Use my_mean to compute the sum of squares of my_vect sum((my_vect - my_mean)^2) # Create a vector called my_vect containing 1 through 5 my_vect <- c(1, 2, 3, 4, 5) # Take the mean and assign to my_mean my_mean <- mean(my_vect) # Use my_mean to compute the sum of squares of my_vect sum((my_vect - my_mean)^2) # first instruction test_object("my_vect", incorrect_msg = "Something is wrong with `my_vect`. Take another look at the instruction.") # second instruction test_object("my_mean", incorrect_msg = "Something is wrong with `my_mean`. The easiest way to complete this instruction is to use the `mean()` function with `my_vect`.") # third instruction test_output_contains("sum((my_vect - my_mean)^2)", incorrect_msg = "Take a look at your code for the third instruction. To calculate the sum of squares you will need to take the sum of the square of the difference between `my_vect` and `my_mean`.") # General test_error() success_msg("Good job! We have a pretty good SCT here. Let's try combining everything we learned to make one final SCT in the next exercise.")
+ 1 to the last instruction so that it no longer matches the solution script.Looks like our SCT worked! We changed the value produced by sum((my_vect - my_mean)^2) by adding + 1 to it. When it printed in the console, our SCT couldn’t find it and it failed the test.
Of course, you can always use multiple functions on each exercise. In fact, as your exercises get more complicated you may need to in order to test the solutions appropriately. Let’s pretend that we just introduced the mean() and sum() function and you want to be sure that a student uses these in their response. Of course, you should specify this in the instructions.
# sct code
# first instruction
test_object("my_vect", incorrect_msg = "Something is wrong with `my_vect`. Take another look at the instruction.")
# second instruction
test_object("my_mean", incorrect_msg = "Something is wrong with `my_mean`. The easiest way to complete this instruction is to use the `mean()` function with `my_vect`.")
test_function("mean")
# third instruction
test_output_contains("sum((my_vect - my_mean)^2)", incorrect_msg = "Take a look at your code for the third instruction. To calculate the sum of squares you will need to take the sum of the square of the difference between `my_vect` and `my_mean`.")
test_function("sum")
# General
test_error()
success_msg("Good job! You've learned some very important and useful functions in this course. Writing SCT's can get complex, but mastering the functions covered in this exercise will get you started so that you can begin creating your own content.")
This looks pretty good. Go ahead and see if you can break it!
# pec # Create a vector called my_vect containing 1 through 5 my_vect <- c(1, 2, 3, 4, 5) # Take the mean and assign to my_mean my_mean <- mean(my_vect) # Use my_mean to compute the sum of squares of my_vect sum((my_vect - my_mean)^2) # Create a vector called my_vect containing 1 through 5 my_vect <- c(1, 2, 3, 4, 5) # Take the mean and assign to my_mean my_mean <- mean(my_vect) # Use my_mean to compute the sum of squares of my_vect sum((my_vect - my_mean)^2) # first instruction test_object("my_vect", incorrect_msg = "Something is wrong with `my_vect`. Take another look at the instruction.") # second instruction test_object("my_mean", incorrect_msg = "Something is wrong with `my_mean`. The easiest way to complete this instruction is to use the `mean()` function with `my_vect`.") test_function("mean") # third instruction test_output_contains("sum((my_vect - my_mean)^2)", incorrect_msg = "Take a look at your code for the third instruction. To calculate the sum of squares you will need to take the sum of the square of the difference between `my_vect` and `my_mean`.") test_function("sum") # General test_error() success_msg("Nice! See how layering your functions can help make them more robust? Remember: R will run this code in order, so consider the order in which you place the functions.")
- Take the sum of the vector and divide by 5
- Do not use the sum function - calculate these manually and print the result
Any luck? Didn’t think so…Together we have built a pretty strong SCT that tests the value of the objects, tests the results printed in the consoles, and tests the functions they used to make sure they’ve done it the way we intended. Nice work!
R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.