News from R Submissions Working Group – Pilot 3 Successfully Reviewed by FDA

Blog contributed by Ning Leng, People and Product Leader, Roche-Genentech and Joel Laxamana, Principal Data Scientist, Roche-Genentech

The R Consortium is pleased to announce the successful completion of the Pilot 3 Submission which extended the work done in Pilots 1 and 2 by generating the ADaM datasets. The complete FDA response letter is available.  

The objective of the R Consortium R Submission Pilot 3 Project is to test the concept that an R-language based submission package for ADaMs and TLFs can meet the needs and the expectations of the FDA reviewers, including assessing code review and analyzing reproducibility. All submission materials and communications from this pilot are publicly available, with the aim of providing a working example for future R-language based FDA submissions. This is an FDA-industry collaboration through the non-profit organization R Consortium.

All submission materials can be found at: submissions-pilot3-adam-to-fda. This is the first publicly available R-based FDA submission package including R scripts to generate Analysis Data Models (ADaM) and Tables, Listings & Figures (TLFs).

Open-Source Collaboration in standardizing Clinical Trial analyses and Submissions. Broadening ways to bring treatments to Patients.

Pilot 3 Timeline

The initial submission was submitted through the eCTD gateway on Aug 28, 2023. FDA verbal responses were received from Jan-July 2024 during R submission working group meetings. Documentation of this initial feedback and response can be found at response-FDA-IR-pilot3.pdf. The updated submission package addressed reported issues and was re-submitted on Apr 19, 2024. The final response letter from FDA was received on Aug 8, 2024.

Pilot 3 Scope

The Pilot 3 test submission exemplifies an all-R submission package for ADaMs and TLFs, adhering to electronic Common Technical Document (eCTD) specifications. This comprehensive package not only includes ADaMs and TLGs, but it emulates a full study submission package including the source Study Data Tabulation Models (SDTMs) used to generate the Pilot 3 ADaMs. It also encompasses the installation and loading of the proprietary {pilot3utils} R package, various open-source R packages, R scripts for the Analysis Data Model (ADaM) datasets derived from Pilot 3, and Tables, Listings, and Figures (TLFs) from Pilot 1. In addition to other requisite eCTD components, the Pilot 3 package also includes the Analysis Data Reviewer’s Guide (ADRG) providing detailed steps leading to the execution of the analysis R scripts to re-produce the ADaMs and TLFs from a FDA reviewers perspective. These Pilot 3 submission materials are linked above.

Pilot 3 serves as a complement to Pilots 1 and 2, which demonstrated the feasibility of submitting TLF R scripts and R Shiny code, respectively. Furthermore, Pilot 3 successfully validated the submission of proprietary R packages in compressed file formats, serving as another alternative to {pkglite} or installation directly from github.

If you have any questions about this Pilot 3 submission, we would love to hear from you. Feel free to submit any questions you may have as a new issue in our Pilot 3 github repository or you may find any of the Pilot 3 team members in Pharmaverse slack.

Learnings from Pilots 1 through 3

In the three pilots, for various TLFs, the working group members intentionally created different tables in different formats using various open-source R packages. The FDA staff successfully accepted and reproduced the results generated from these different open-source packages. However, though not in scope of these Pilots, we want to share awareness that sponsors are responsible for selecting open-source packages that demonstrate sufficient reliability. Further information on this can be found in the R Validation Hub, formed in 2018 by the PSI AIMS Special Interest Group and supported by the R Consortium. It offers tools like {riskmetric} to quantify the “risk” of R packages and a user-friendly R Shiny app, {riskassessment}, to evaluate package reliability.

The majority of FDA staff feedback falls under the following themes :

1. Clear ADRG documentation on computing environment, package dependencies, and expected warnings
2. Clear documentation on data processing rules and statistical method implementation
3. Good statistical practice in confirmatory trials, such as avoiding the possibility of “p-hacking” 

For future submissions using open-source languages, it is recommended to give special attention to recommendation theme 1. Recommendation themes 2 and 3 are language-agnostic and should always be followed, regardless of the programming language used. All of these themes fall in line with the FDA’s Statistical Software Clarifying Statement.

Upcoming Pilots

As a next step, the R Consortium R Submission Working Group initiated submission pilot 4, to explore the use of novel technologies such as Linux containers and web assembly to bundle a Shiny application into a self-contained package, facilitating a smoother process of both transferring and executing the application.

The R Consortium R Submission Working Group

The R Consortium R Submissions Working Group is focused on improving practices for R-based clinical trial regulatory submissions.

To bring an experimental clinical product to market, electronic submission of data, computer programs, and relevant documentation is required by health authority agencies from different countries. In the past, submissions have been mainly based on the SAS language. 

In recent years, the use of open source languages, especially the R language, has become very popular in the pharmaceutical industry and research institutions. Although the health authorities accept submissions based on open source programming languages, sponsors may be hesitant to conduct submissions using open source languages due to a lack of working examples.

Therefore, the R Submissions Working Group aims at providing R-based submission examples and identifying potential gaps during submission of these example packages. All materials, including submission examples and communications, are publicly available on the R consortium Github page: https://github.com/RConsortium .

The R consortium R submission working group includes members from more than 10 pharmaceutical companies, as well as regulatory agencies. More details of the working group can be found at: https://rconsortium.github.io/submissions-wg/ .

The R consortium R submission working group is open to anyone who is interested in joining. If interested, please contact Joseph Rickert at director@r-consortium.org. 

Pilot 3 FDA Reviewers

FDA reviewers included  

Hye Soo Cho,  Paul Schuette, and Youn Kyeong Chang.

Pilot 3 Developers

The Pilot 3 development team included Joel Laxamana (Project Lead, Roche), Robert Devine (J&J) , Benjamin Straub (GSK) , Kangjie Zhang (Bayer) , Thomas Neitmann (Roche), Phanikumar Tata (Syneos), Steven Haesendonckx (J&J), Yutong Liu (Moderna), Lei Zhao (Roche), Nicole Jones (Merck), Benjamin Wang (Merck), Dadong Zhang (Illumina), Declan Hodges (GSK).

The post News from R Submissions Working Group – Pilot 3 Successfully Reviewed by FDA appeared first on R Consortium.