Creating progress bars with foreach parallel processing

by Andrie de Vries

In my previous post, I demonstrated how to get some status of running jobs on a parallel back end. However, I stopped short of actually demonstrating progress bars. In this post I demonstrate how to do this.

The StackOverflow question How do you create a progress bar when using the “foreach()” function in R? () asks the question how to create progress bars with foreach.

This question attracted some helpful answers, but ultimately didn’t answer the question.  In particular, Steve Weston said this approach is flawed, since

It won't work well with doParallel because doParallel only calls the combine function after all of the results have been returned, since it is implemented by calling the parallel clusterApplyLB function. This technique only with works well with backends that call the combine function on-the-fly, like doRedis, doMPI, doNWS, and (defunct?) doSMP.

My experiments

However, this set me thinking.  Could it be that one could trigger a tcltk progress bar in each worker?

Thus I started to experiment, and came up with the following code.  On my windows machine, using doParallel, this seems to work reliably (but I could not get this approach to work on Linux using doMC).

When you run this code, you should see synchronous progress bars, one for each worker:

Conclusion

Although it takes some work, and there isn’t a single solution for all cases, it is possible to get progress bars for at least some parallel back ends.

In this post I demonstrated how to create progress bars using doParallel on Windows, but failed to replicate on Linux.

Can you do better?