Backtesting

The ts_backtesting function provides you the ability to train, test and evaluate multiple models with the use of backtesting approach. This allows automating the forecasting process by running a ‘horse race’ between different models or approaches while testing them over multiple periods to evaluate their performance over time. The example below demonstrated the use of the function to forecast the monthly consumption of natural gas in the US for the next five years (or 60 months). By default, the function is testing seven different models, using expended window over six periods:

library(TSstudio)
data("USgas")

ts_info(USgas)
##  The USgas series is a ts object with 1 variable and 223 observations
##  Frequency: 12 
##  Start time: 2000 1 
##  End time: 2018 7
usgas_backtesting <- ts_backtesting(USgas,
                                 periods = 6, # Set the number of periods in the backtesting
                                 window_size = 12, # Set the length of the testing set
                                 h = 60, # Set the horizon of the final forecast
                                 plot = FALSE,
                                 error = "MAPE" # Set the error matrix
                                 )
##    Model_Name  avgMAPE    sdMAPE  avgRMSE   sdRMSE
## 1  auto.arima 4.988333 0.6685931 163.5050 15.56474
## 2      hybrid 6.785000 1.0958056 205.0417 27.75507
## 3       tbats 6.861667 0.8633288 194.5400 19.22430
## 4      nnetar 7.226667 0.8768048 260.7700 23.34733
## 5 HoltWinters 8.163333 1.4684232 232.0233 30.69781
## 6        bsts 9.325000 3.3563716 258.3433 75.55861
## 7         ets 9.350000 2.7671140 247.5117 72.37548

By default, the model which performed the best in the testing sets, according to the error criteria (RMSE or MAPE), will be selected by the function to forecast the series. In the case of the USgas series, as you can see in the leaderboard above, the auto.arima model achieved the best results and therefore, the function will select this model to forecast the future values:

# Plotting the results
usgas_backtesting$summary_plot

In addition, the output of this function includes all the output from the trained models and their forecasts. For instance, you can check the residuals of the selected model:

check_res(usgas_backtesting$Models_Final$auto.arima)

Or pull the forecast of the hybrid model:

plot_forecast(usgas_backtesting$Forecast_Final$hybrid)

A short video of this function is available here

The ts_seasonal function

The ts_seasonal function is now supported data frame objects (data.frame, data.table, and tbl), in addition to the time series objects (ts, xts, and zoo). The function has three modes, which can define with the type argument:

• normal - subsetting and plotting the series by its full cycle (or year), this allows identifying if there is a repeated pattern in the series from year to year

ts_seasonal(USgas, type = "normal")