In recent years, an increasing number of organizations have started to consider how to implement practical changes toward responsible academic assessment practices. However, reform is a process that takes time and resources, and there is no one-size-fits-all approach. In December 2020, to begin to address the increasing need for clear, structured examples of the processes behind responsible assessment reform, DORA, the European University Association (EUA), and SPARC Europe launched the case study repository “Reimagining academic assessment: stories of innovation and change.”
In 2021, three new case studies were added to the repository to offer the community additional examples of organizations in various stages of academic assessment reform. On December 7, 2021, DORA held a webinar featuring these organizations. The goal of this webinar was for participants to hear about the process of implementing reform, including: motivations, methodologies, and challenges encountered. Speakers included Laura Rovelli of the Latin American Forum for Research Assessment (FOLEC), Argentina; Johanna McEntyre of the European Molecular Biology Laboratory (EMBL), United Kingdom; and Richard Holliman of the Open University (OU), United Kingdom. The webinar was moderated by Haley Hazlett, DORA Program Manager, and Anna Hatch, DORA Program Director.
Reform in an international interdisciplinary consortium: FOLEC
FOLEC is an initiative of the Latin American Council of Social Sciences (CLACSO) that aims to facilitate the creation of a collective and improved Latin American and Caribbean standard of practice for academic research assessment. The motivation to create FOLEC stemmed from an internal desire from researchers in the region to create evaluation practices that were not centered on journal-based metrics and that made sense in their local contexts.
Rovelli explained that CLACSO’s structure, which spans across Latin America and the Caribbean, allows FOLEC to facilitate systems change on an international level. FOLEC does this by creating a space for knowledge exchange between CLACSO research member institutions and regional policymakers, and by working to translate and contextualize evaluation practices that are meaningful for the region. This dynamic enables FOLEC to promote a bottom-up, participatory and context-specific approach to implement change in research evaluation.
Rovelli noted that FOLEC has focused on developing resources to raise awareness and build expertise on research assessment practices in the region. In the first years of its operation, FOLEC developed a diagnosis of the region’s evaluation practices and shared its proposals for change. Moving forward, in 2021, FOLEC developed a series of advocacy papers on critical topics for systemic change in Latin America and the Caribbean:
- Current Information Systems (CRIS)
- Multilingualism & bibliodiversity
- National journals and their assessment in evaluation processes
On a systems level, FOLEC’s challenges include developing interoperable platforms that reflect the diversity of research outputs in the region, broadening the scope of scientific outcomes that are rewarded, and promoting the social diversification of evaluators. To address these challenges, foster international dialogue, build consensus, and raise awareness, FOLEC plans to launch a declaration of research assessment principles at the 9th Latin American and Caribbean Conference on Social Sciences in June 2022.
Reform in an intergovernmental research organization: EMBL
EMBL is an intergovernmental life sciences research organization whose mission includes research, training, and services. After signing DORA in 2018, EMBL established a Research Assessment Working Group that created four responsible research assessment recommendations to guide their reform efforts (see graphic below).
EMBL’s 4 Research Assessment Recommendations. Image Credit: Johanna McEntyre, EMBL-EBI
McEntyre explained that EMBL’s motivation for assessment reform came from a mix of internal and external forces. Internally, EMBL recognized the importance of codifying and providing examples of responsible assessment practices for the scientists trained at its sites. EMBL’s nine-year term limit for faculty members creates a unique training opportunity to equip alumni to recognize and support responsible evaluation practices as they progress through their academic careers. External influences for implementing change included the reform efforts of funding agencies like Wellcome.
While EMBL’s reform was initiated by senior leadership, there was also strong support and engagement from EMBL’s researchers and research support staff. For example, when the EMBL formed their DORA Working Group, they sought to compose a group that was representative of the different nationalities, sites, work cultures, and departments involved in the assessment process. After collecting feedback on what their staff understood as “hot areas” for interventions, the working group presented its findings to EMBL’s leadership and received buy-in to move forward. McEntyre said that this joint vision between leadership and staff has been key in promoting engagement from institutional leadership, researchers, and support staff, such as human resources.
In conclusion, McEntyre stressed that EMBL still has a long journey ahead. Still, she remains confident about the importance of reform, stressing that a clear communication strategy is critical to foster the conversation about research culture change.
Reform in a university: The OU
The OU is a distance learning university in the UK with a strong focus on knowledge exchange. According to Holliman, the OU’s motivation for research assessment reform was triggered when the Research Councils UK (now part of the UKRI) launched the Concordat for Engaging the Public with Research in 2013. The Concordat recognized the importance of public engagement to help maximize the social and economic impact of UK research. In response to this call to action, the OU carried out the “An open research university” project, which was designed to create conditions in which publicly engaged research could flourish.
Holliman explained that the 2015 project report identified the need to revise faculty promotion criteria, resulting in the creation of a working group to do so. With the National Co-ordinating Centre for Public Engagement’s “EDGE tool” as their conceptual framework, the working group used a combination of strategies to inform the development of new promotion criteria. For example, they interviewed senior leadership and research staff members to identify new ways to recognize research excellence. One key output of these efforts was the creation of an additional promotion route for faculty with strong track records of knowledge exchange in their scholarly work. The addition of this promotion route better allows for the recognition and reward of work that is publicly engaged, enabling faculty to submit promotion portfolios based on what best fits their expertise.
Here, Holliman noted that one of the challenges encountered during this process was the uncovering of an implicit hierarchy between different faculty priorities, with research on top, followed by teaching in the middle, and knowledge transfer at the bottom. To address this, he emphasized the importance of aligning values and goals using tools like the SPACE rubric.
Lessons learned
The speakers highlighted several important features of organizational change. First, they emphasized that signing DORA is the first step in a long and iterative process toward responsible research assessment. Second, there is no one-size-fits-all reform model. Each organization is working to implement change in its own context and has experienced unique challenges. Finally, a common thread throughout the discussion was that each one of these institutions is striving for the co-creation of a reform process that best fits their goals, purposes, and circumstances.
Amanda Akemi is DORA’s Policy Intern