How We Curate Our Monthly Newsletter

How to keep up with rOpenSci? We agree that we’re doing so much good work that it’s hard. ? More seriously, we’ve been curating and sharing a news digest with our community for years because we believe it to be useful. Over time its structure and infrastructure have evolved. In this post we’ll share how we currently prepare content for the newsletter and send it to subscribers’ mailboxes, as automatically as possible.

🔗 Short history of our newsletter

rOpenSci has had a newsletter since 2014¹ For the longest time the newsletter has been living at a Jekyll website at a subdomain; whose entries were also sent as email to subscribers. It was published mostly once every two weeks.² The preparation of each post was helped with a Makefile, and some manual manipulation and prose writing.

Later, the newsletter got complemented by posts on our main blog called “two months in two minutes” that were digests of the digests.³

For a few months now we have settled on:

• A monthly post on our main blog, tagged with the “newsletter” tag;
• That is also sent as an email to our newsletter subscribers.

🔗 Newsletter content

Find the June newsletter. The newsletter structure is (? for automatically generated content, ✍️ for manually curated sections):

• ✍️rOpenSci HQ with announcement about events, new projects, etc.
• ? Software with new packages and package updates.
• ? Software Peer Review with a snapshot of current or recent submissions.
• ? On the blog with short summaries of latest blog posts.
• ? Citations of our tools.
• ? Use cases of our tools.
• ✍️Call for maintainers when packages need a new CEO.
• ✍️Package development corner with tips for package developers.

🔗 Tech stack and tips

🔗 Newsletters as blog posts

Having the newsletter available as entries online somewhere is very handy for linking to them from e.g. Twitter. Furthermore, we really like our website (redesigned last summer ✨) so it felt logical to have the newsletter live in it, especially as it meant one could hop from the newsletter to other website pages much more easily. Having the newsletter issues as blog posts was the most natural way to integrate them here, but we’ve also:

• Created a newsletter page that features its sign-up form, and previous newsletter-tagged posts;
• Added a direct link to the newsletter page via an email icon on the newsletter tag page, which makes it even easier to find how to subscribe.

🔗 Newsletters as R Markdown parameterized reports

Now, how do we go from “information out there” to a formatted blog post? We use an R Markdown document that needs the date of the latest newsletter as parameter. That document is stored as an Hugo archetype so we can create a newsletter via the blogdown New Post addin in RStudio. It’s filled with chunks producing text.

Compared to the previous way we created newsletters, our new method relies more heavily on data accessed directly from GitHub:

• New packages are packages that were not in rOpenSci registry at the time of the last newsletter. To find rOpenSci registry at the time of the last newsletter, we query GitHub commits.
• The citations data that we curate is also stored on GitHub and we use similar history logic to identify “new” citations.
• Names of reviewers of new packages come from DESCRIPTION when they’re listed there, otherwise we manually add a snippet of code to the R Markdown document to add their names to the package presentation.
• Package updates are GitHub releases in the repositories of packages listed in the rOpenSci registry. While this adds some more work to maintainers (as opposed to e.g. “only” releasing their package on CRAN), it’s a one small workflow step that can be helped by usethis. We were already recommending the use of GitHub releases in our dev guide.
• Software review processes are summarized according to the issue labels. While these used to be applied by editors, we’ve been using our review bot more and more so that the label data has become better.

The use cases data comes from our forum via Discourse API whose client discgolf is maintained by rOpenSci Scott Chamberlain. The same JSON data feeds our use cases page.

A random tip is that we use english::english() to transcribe numbers to English. E.g. as.character(english::english(2021)) gives “two thousand and twenty-one”. An alternative (that we had not heard of when creating the R Markdown document) is xfun::numbers_to_words().

We still have some progress to make e.g. instead of using a DESCRIPTION file to store dependencies, we should probably switch to using renv. Let’s see, we may wait for something to break before making the change!

🔗 Newsletters as products crafted with human care

The newsletter has a few manually curated sections. To populate this, we use information from the website (comm calls), our reading (was an interesting answer or question sent to the R-package-devel mailing list?), etc. To not forget ideas or to bring them to the attention of the newsletter curator we store them as issues in a GitHub repository.

Furthermore, the manually curated sections are reviewed by at least one other person. The posts don’t get a full blown review but this helps strengthen them.

🔗 Newsletters as emails: hello Sendgrid, bye MailChimp!

We used to send our newsletter via MailChimp, however we’ve been hearing problematic things about the company itself so asked around for recommendations (thanks a ton to everyone who chimed in!) and settled on SendGrid. SendGrid offers both email and marketing services. We were already using email services for the CRAN checks API and our forum. We now also use SendGrid marketing services, for sending our newsletter.⁴

How do we go about that? We extract the HTML corresponding to the post from our full content RSS feed (set up for R Bloggers). We use that feed as all relative links (except for images) have been made absolute at that point. We copy-paste it into a text editor and make a few tweaks:

• make the images src an absolute link,
• remove all heading anchors,
• add a no tracking attribute to links as the global SendGrid account option didn’t work for that.⁵

These tweaks are listed in our newsletter checklist but we might transform them to code using xml2 soon-ish.

We then copy this HTML as a code block in SendGrid visual editor. Regarding the email appearance, we added some blocks (e.g. the unsubscribe one at the bottom), tweaked a few styling rules (e.g. the color of text, background, links, to match our website). To send a new newsletter we duplicate a previous one. Folks more experienced with SendGrid might create a “template” for that, presumably.

We make sure the subject and preheader (what MailChimp calls preview, the email bit recipients might see near the subject in their inbox) are updated, then send a few test emails, and then more or less nervously send the newsletter to its more than 2,000 subscribers.

Voilà, at that point all that remains is tweeting about the newsletter, and hoping it gets read at least a bit. A problem we encountered with our first SendGrid campaign is the rendering of the on some devices, which we will investigate by next time.

Note that SendGrid sign-up forms offer much less flexibility than MailChimp similar features, so we had to add an iframe to our website to incorporate sign-up, where we used to have a more integrated form whose JS code pinged a dedicated MailChimp URL.

🔗 Conclusion

In this post we presented our workflow for curating and sending our monthly newsletter. Our newsletter is published as a monthly post on our blog, and sent as an email to subscribers via SendGrid. We prepare its content with a mix of automatically generated and manually curated sections, followed by a review for the latter. The automatic generation of some content depends on upstream manual and automatic work for gathering data (e.g. our packages registry, citations monitoring, encouraging users to report use cases). We hope the newsletter helps our community stay informed of our work. If you have any suggestion for our newsletter, don’t hesitate to leave a comment below! And to subscribe, head over to the newsletter page!

1. Reading that first issue is amusing as it announces the beginning of comm calls, that are now a well established part of rOpenSci activities and have a very polished flow. ↩︎

2. That is why it was called rOpenSci biweekly, although not everyone agrees on what frequency biweekly means (twice a week or once every two weeks?). ↩︎

3. The newsletter section about blog posts mentioned these digests. ↩︎

4. Transferring contacts merely means exporting subscribed contacts out of MailChimp and then importing them into SendGrid. As we assume most people unsubscribe from a very recent newsletter, we should not miss unsubscribe events. Note that to add contacts by CSV to SendGrid make sure to remove any space in the column headers (MailChimp CSV export is not usable for SendGrid without these tweaks). ↩︎

5. We do track opens, via a 1px square image, which you can turn off by not letting your email service load external images by default. ↩︎