This post can also be found on the CSCCE blog.”

How do you measure the impact of a champions program? This was the central question of a working session at CZI’s Accelerating Open Science in Latin America workshop, convened by rOpenSci’s Community Manager Yani Bellini Saibene and attended by CSCCE’s Founder and Director, Lou Woodley.

Measuring the impact of any kind of community program presents a series of challenges:

• What is the impact that you’re hoping your program will have?

• Is the impact you hope the program will have something that can be measured?

• What types of instruments can be used to measure impact? (e.g., surveys, focus groups, etc.)

• How many times can you reasonably ask your participants to give feedback?

• How do you (or can you?) reliably follow up with participants months or even years after a program has concluded?

The session took the form of a facilitated discussion of these points that all of the 15 – 20 participants were invited to contribute to – with the goal of learning more about one another’s experiences and creating a list of useful resources. In this blog post, we summarize these conversations with a view to continuing them in the rOpenSci Slack, the CZI Slack, and CSCCE’s Slack workspace for STEM community managers.

What is a champions program?

Community champions are individuals who take on more responsibility for the success, sustainability, and/or running of a community (read more in CCSCE’s guidebook). Champions programs formalize these roles, and may offer champions incentives, opportunities to connect and access to training and other support. They also offer community managers a means of maintaining, growing, or evolving a community by partnering with champions to further the community’s mission.

Champions programs have a variety of names in STEM, including fellowships and ambassador programs. You can find out more about champions programs, including a collection of tip sheets for those interested in creating a program for their own community, on the CSCCE website.

What is “impact,” and how can you measure it?

Community managers running champions programs are invested in their success for a variety of reasons. But deciding how to evaluate a program, and define the data that will support that evaluation, isn’t particularly straightforward.

During the session, participants considered a range of nuances around evaluation, starting with understanding the meaning of “impact.” Champions programs ideally impact both the champions themselves and the community writ large. Asking participants to complete surveys or exit interviews is a great way to collect initial feedback on the former, but measuring longer-term community impacts is less straightforward.

A complicating factor for many community managers is assigning “value” to different types of activities, which is especially complicated when measuring outcomes that rely on “invisible labor.” For example, in open-source communities, impact is often measured by tracking code contributions, but the work that champions may do to organize a monthly community call or translate documentation is neither so obvious nor so quantifiable.

Some champions programs include a training and mentoring aspect, which adds roles such as mentors and instructors in addition to champions. Measuring the impact of the program on these individuals is useful, too.

Combining qualitative and quantitative metrics

Some solutions to these challenges included defining clear goals for the program and then designing evaluation tools to match; benchmarking champions’ expectations at the beginning of the program during an initial intake meeting; and using regular check-in meetings to ask about how the program is going as a way to learn more about less visible work. By combining “annecdata” with quantitative metrics (e.g., number of commits, attendance at training sessions, participation in a community forum, etc.), community managers can gain a more complete picture of the overall success of their program.

As you design your survey, only include questions that matter to your evaluation, and clearly communicate to your champions and other participants what you’ll be doing with any data you collect. It is also important to think through the metrics you’re collecting so that you aren’t always valuing the same type of contribution. Program participants can help identify and collect non-traditional contributions, especially in open software projects. And champions programs can be very good places to start valuing these contributions by using and displaying the metrics associated with them.

Resources

• rOpenSci Champions Program: https://ropensci.org/champions/

• Mozilla Open Leaders: https://foundation.mozilla.org/en/initiatives/mozilla-open-leaders/fork-open-leaders/

• CSCCE resources on community champions programs: https://www.cscce.org/resources/community-champions-programs/ including a series of 9 tipsheets and a guidebook describing the role of community champions.

• Channel in the CZI Slack: #opensci-measuring-impact

• Simple and Editable Logic Model Development Template (Google Doc, CC BY licensed)