The specflow and cucumber.io for R. Enabling non-coders to interpret test reports for R-packages, moreover allowing non-coders to create test cases. A step towards simple r package validation.

Table of contents

• Why RTest?
• What’s special about RTest?
• An example of a test implementation with RTest.
• Further Reading

Why RTest?

Testing in R seems simple. Start by using usethis::test_name("name") and off you go by coding your tests in testthatwith functions like expect_equal. You can find a lot of tutorials online, there is even a whole book on “Testing R Code”. Sadly, this is not the way I can go. As I mentioned a few times, I work in a strongly regulated environment. Inside such environments your tests are not only checked by coders, but also by people who cannot code. Some of your tests will even be written by people who cannot code. Something like specflow or cucumber would really help them to write such tests. But those do not exist in R. Additionally, those people cannot read command line test reports. You can train them to do it, but we decided it is easier to provide us and them with a pretty environment for testing, called RTest.

If you want to know more about the reasons for developing such an environment, you can read the article: Why do we need human readable test for a programming language.

What’s special about RTest?

To explain which features we put into RTestI will start describing of a basic testing workflow.

1 Testing code starts with writing code. Your R-package will contain functions, classes and methods. These shall be tested.

2 Writing the tests now mostly includes calls like this:

my_function(x,y){sums_up(x,y) return(z)}
x=3
y=4
z=7
stopifnot(my_function(x,y)==z)

It is easy to see, that your test will break if your function my_function cannot sum up two values. You will create a bunch of such tests and store them in a separate folder of your package, normally called tests .

3 Afterwards you can run all such tests. You can include a script in the tests folder or use testthat and runtestthat::test_dir() .

4 If one of your test fails the script will stop and tell you which test failed in the console. This describes the 4 steps shown in the figure below.

What’s now special about RTest are two major steps.

1. The definition of the tests
2. The reporting of the test execution

For the definition of the tests we decided for XML. Why XML? XML is not just easier to read then pure R-Code, it comes with a feature, that is called XSD; “XML schema definition”. Each XML test case we create can immediately be checked against a schema designed by the developer. It can also be checked against our very own Rtest.xsd. This means the tester can double check the created test cases before even executing them. This saves us a lot of time and gives a fixed structure to all test cases.

The reporting was implemented in HTML. This is due to the many features HTML comes with for reporting. It allows coloring of test results, linking to test cases and including images. The main difference for the reporting in HTML between RTest and testthat is that RTest reports every test that shall be executed, not only the failed ones. The test report will also include the value created by the function call and the one given as a reference. The reader can see if the comparison really went right. By this the test report contains way more information than the testthat console log.

An example of a test implementation with RTest

Please note the whole example is stored in a github gist. Please star the gist if you like this example.

1. Given a function that sums up two columns:

my_function <- function(
  data = data.frame(x = c(1,2), y = c(1,2))){
    stopifnot(dim(data)[2] == 2)
    data[, "sum"] <- apply(data, 1, 
     function(x){sum(x)})
    return(data)
} 

---

2. We want to have one successful and one non successful test. Both will have three parts in the XML file:

<params><reference><testspec>

params accounts for input parameters

reference for the output data.frame

testspec for whether the test shall run silently and what the tolerance is

For the successful test our test would look like this:

<my_function test-desc="Test data.frame">
  <params>
    <RTestData_input_data param="data" name="test01" />
  </params>
  <reference>
    <col-defs>
      <coldef name="x" type="numeric" /><coldef name="y" type="numeric" /><coldef name="sum" type="numeric" />
    </col-defs>
    <row>
      <cell>1</cell><cell>2</cell><cell>3</cell>
    </row>
    <row>
      <cell>1</cell><cell>2</cell><cell>3</cell>
    </row>
  </reference>
  <testspec>
    <execution execution-type="silent" />
    <return-value compare-type="equal" diff-type="absolute"
      tolerance="0.001" />
  </testspec>
</my_function>

view rawRTest_part.xml hosted with ❤ by GitHub

You can immediately see one special feature of RTest. It allows to use data sets for multiple tests, we store those data sets in the input-data tag.This saves space in the file. The dataset test01 will be used here. Moreover a test description can be given for each test. For each data.frame stored in XML the types of the columns can be given in col-defs . Here those are all numeric.

Theinput-datais now given here:

<input-data>
<data.frame name="test01">
  <col-defs>
    <coldef name="x" type="numeric" /><coldef name="y" type="numeric" />
  </col-defs>
  <row>
	<cell>1</cell><cell>2</cell>
  </row>
  <row>
    <cell>1</cell><cell>2</cell>
  </row>
</data.frame>
</input-data>

view rawRTest_input_data.xml hosted with ❤ by GitHub

It’s a data frame with the x column just carrying 1 and the y column just carrying 2. The test shall create a data.frame with the sum column being 3 in each row.

We can easily let the test fail by changing the reference tag and instead of having just 3 in the sum column we can add a 3.5 to let the test fail. The whole test case can be found inside the github gist with 90 rows.

---

3. The execution of the test case is just one line of code. You shall have your working directory in the directory with the XML file and my_function shall be defined in the global environment.

RTest.execute(getwd(),"RTest_medium.xml")

---

4. The test report now contains one successful and one failed test. Both will be visualized:

additional information is given on all tests. For the test that failed we caused it by setting the sum to be 3.5 instead of 3. It’s reported at the end of the table:

Moreover the Report contains information on the environment where the test ran:

That’s it. Now you can test any package with RTest.