Experiment

We will consider different scenarios of number of warnings thrown to tease out the relationship between speed of the function and number of warnings thrown in a function with and without cli::cli_warn(). We will also check how the benchmark scales with the number of warnings in the function.

Based on this description, we will need the following packages: cli for its warning function, and bench for measuring the run times, dplyr for manipulating the results, and ggplot2 for plotting.

Let’s define a function that takes an argument n for the number of warnings to throw and pkg for the type of package to use.

warn_Ntimes <- function(n, pkg) {
  warning_msg <- "x must be an integer"
  switch(pkg,
    base = for (i in seq_len(n)) {
      warning(warning_msg)
    },
    cli = for (i in seq_len(n)) {
      cli_warn(warning_msg)
    }
  )
}

Let’s test our function to see if it works as expected.

warn_Ntimes(3, "base")

Warning in warn_Ntimes(3, "base"): x must be an integer

Warning in warn_Ntimes(3, "base"): x must be an integer

Warning in warn_Ntimes(3, "base"): x must be an integer

warn_Ntimes(3, "cli")

Warning: x must be an integer

Warning: x must be an integer
x must be an integer

Now, we’ll consider scenarios where a function throws 1, 5, 10, 15, 20, and 100 warnings using base R and cli.

# Number of warnings to throw
warnings <- c(1, 5, 10, 15, 20, 100)

Let’s run benchmarks over the different scenarios and store results in a data.frame.

bnmark_res <- press(
  warnings = c(1, 5, 10, 15, 20, 100),
    mark(
      cli = warn_Ntimes(warnings, "cli"),
      base = warn_Ntimes(warnings, "base")
    )
  )

Results

It’s time to explore the results of the data generated. Let’s first make a boxplot of the run times for the different scenarios.

Drum roll please…

autoplot(
  bnmark_res,
  type = "boxplot"
  ) +
  labs(
    x = "Package",
    caption = "Functions throwing various numbers of warnings as indicated in the facet label"
  )

Loading required namespace: tidyr

Now, let’s see how the benchmarks scale with the number of warnings thrown.

bnmark_res$pkg <- attr(bnmark_res$expression, "description")

ggplot(bnmark_res) +
  geom_line(
    aes(
      x = warnings,
      y = median,
      color = pkg
    ),
    linewidth = 2
  ) +
  labs(
    x = "Number of warnigns",
    y = "Median execution time",
    color = "Package"
  )

As we can see, cli is consistently slower than base R for the same number of warnings thrown. The median times also follow a similar non-linear trajectory. Benchmarking experiments with other cli functions have revealed similar non-linear relationships between its speed and the number of times it invokes those functions ². However, those exercises were not compared with equivalent functions in base R. The relative comparison here is useful for decision-making.

The developers of cli have also conducted benchmarks of the ansi_*() family of functions in cli in comparison to base R and the fansi package. They find that cli is consistently slower than base R, which corroborates the results of our experiment here. These benchmarks are available in the cli documentation ³.

So, should we be worried about the speed of cli? Well, it depends on the context. The “R Packages” book by Hadley Wickham and Jenny Bryan suggests approaching such a decision from a holistic, balanced, and quantitative approach(Wickham and Bryan 2023). We’ll leave the reader to make their own decision based on their use case.

Most developers might argue that this is an optimisation overkill⁴. However, it is important to consider speed differences in context. In the case of simple printing, the speed difference is negligible yet disruptive and somewhat painful. However, in the grand scheme of things, this might be nothing compared with much slower processes that need more attention. In those cases, the developer might want to consider other optimisation strategies such as profiling ⁵. The essence of this experiment is to demonstrate the utility of benchmarking in making quick design decisions.

Conclusion

In designing R package infrastructure with dependencies, it might sometimes be necessary to check if they don’t slow down existing processes. Here, we have demonstrated how benchmarking is one way to achieve this for a process involving condition signalling. We show how a simple decision to use cli::cli_warn() to handle warnings could come at the cost of a tiny loss in speed, which is worth considering in its context.

The demonstration here can be extended to other dependency adoption decisions for input checking, loops, object manipulations, and so forth. We recommend benchmarking as a way to help developers make quick design decisions. However, we also recommend that developers consider the context of the optimisation in interpreting the results.

Other R packages for benchmarking

• microbenchmark: an R package for comparing the execution time of R expressions.
• rbenchmark: an R package for benchmarking R code.
• tictok: an R package to time R functions
• touchstone: an R package for benchmarking of pull requests with statistical confidence.