Running GitHub Actions Sequentially

The issue

Ever since the generally available release of GitHub Actions in November (2019), it seems like many R packages developed on GitHub have switched from Travis CI or another continuous integration service to now using GitHub Actions (GHA). It seems like a great service to have integrated as closely as possible to your codebase, but the product is still under active development. There is a dedicated tag in the GitHub Support Community to ask questions and browse answers. However, in a few different questions (here, here, and here) it seems like folks are still grappling with GitHub’s design decision to execute all workflows and jobs in a repository in parallel, myself included.

As background, it is important to note here the basic lingo of GitHub Actions. A repository can contain one or more ‘workflows’ defined by YAML files located in the .github/workflows folder in the top level of the repository. Each ‘workflow’ can contain one or more ‘jobs’ that execute a series of ‘steps’. By default, all steps in a single job execute sequentially. If you’re trying to limit the number of parallel ‘jobs’ then you you can set a limit of 1 for the workflow by setting max-parallel: 1 within the jobs.strategy element of the workflow YAML. However, the issue with multiple jobs in a single workflow is that if one job fails out of 10 jobs in your workflow, then you’ll have to re-run all 10 jobs for the workflow status to be a success. For this reason, I have decided to split jobs across different workflows. This way, I can re-run one individually, if needed. I will refer to ‘jobs’ and ‘workflows’ interchangeably throughout this article because you can have workflows that execute one job and one job only.

Why do I need sequential workflows/jobs?

The reason I need this functionality is that my workflows interact with a 3rd-party service (Salesforce) and one workflow might affect the results of another workflow if accessing the service simultaneously. I also want to prevent other workflows from executing if I can’t get the first one to succeed since the issue could occur across all workflows. This allows me to reduce the total number of API calls to Salesforce which are capped in a 24-hour period by catching issues early before other workflows execute.

An Example

You can setup sequential workflows using a repository_dispatch action in 4 easy steps:

1. Step 1 – Create a Personal Access Token (PAT)
2. Step 2 – Add the PAT as an actions secret in the repository
3. Step 3 – Add the repository_dispatch event to Workflow 1
4. Step 4 – Add the repository_dispatch event as trigger in Workflow 2 YAML

For context, a required element in every workflow the name of the GitHub event that triggers the workflow. For example, on: pull_request means “execute this workflow every time a pull request is opened”. If you want to run workflows sequentially, then you just need to issue a specific event type that lets the next workflow know when to begin. You could try to write your own solution that uses the GitHub Actions APIs to list the workflows, jobs, or check-suites and find out which ones have failed or not, but the easy alternative is to use a repository_dispatch event.

“You can use this endpoint to trigger a webhook event called repository_dispatch when you want activity that happens outside of GitHub to trigger a GitHub Actions workflow or GitHub App webhook.”

^{https://developer.github.com/v3/repos/#create-a-repository-dispatch-event}

Workflows aren’t aware of other workflows, so this event webhook is perfect to trigger, or “daisy-chain”, separate workflows. You could execute the event via a curl command from the shell in a new job step, but I recommend using the Repository Dispatch action that Peter Evans has released in the GitHub Marketplace, which makes it dirt simple to execute a repository dispatch event from a workflow.

- name: Trigger next workflow
  if: success()
  uses: peter-evans/repository-dispatch@v1
  with:
    token: ${{ secrets.REPO_GHA_PAT }}
    repository: ${{ github.repository }}
    event-type: trigger-workflow-2
    client-payload: '{"ref": "${{ github.ref }}", "sha": "${{ github.sha }}"}'

client-payload: '{"ref": "${{ github.ref }}", "sha": "${{ github.sha }}"}'

name: Workflow 2

on:
  repository_dispatch:
    types: [trigger-workflow-2]

    steps:
      - uses: actions/checkout@v2
        with:
          ref: ${{ github.event.client_payload.sha }}
      
      ... (other steps)
      
      - uses: r-lib/actions/setup-r@master
      
      - uses: r-lib/actions/setup-pandoc@master

name: Workflow 1

on:
  push:
    branches: main

Considerations

If you need your jobs to execute sequentially but you want them to all still run, even if some fail, then just change the if: statement mentioned above in Step 3 to if: always(). The only reason I did not do this is that even if the following workflow is successful, it will get triggered again when we re-run the failed workflow, which I didn’t want. In order to achieve that you may need to use some more advanced tricks/hacks to only execute if the next workflow has the same ref and sha and the latest run does not have a ‘completed’ status.

Another consideration is the cost to execute GitHub actions in private repositories. It’s true that many projects will not likely need to enforce sequential workflows because the tests, examples, checks or dependencies do not affect other workflows. However, GitHub Actions is only free for public respositories. You may have private repositories and want to limit the amount of processing time so you can stay within the free tier (less than 2,000 minutes per month). Sequential execution can prevent all the workflows from executing if upstream workflows fail.

References

You may find the following resources helpful when setting up on your own repository:

• Peter Evans’ GitHub Action to create a respository_dispatch API call

• Getting Started with GitHub Actions

• GitHub Actions REST API Documentation

• GitHub Support Community for Actions