In late 2020 and early 2021, DORA, the European University Association (EUA) and SPARC Europe created a case study repository and accompanying report to showcase a range of innovators in the area of research assessment. To complete this project, the organizations hosted a joint webinar on February 24, 2021, to present and discuss the main findings from the project with some of the organizations that contributed case studies.
An outsized focus on traditional metrics in academic career assessment can introduce unintended biases that limit institutional capacity to capture non-traditional, qualitative research contributions. Achievements like openness, transparency, teaching, quality assurance, and other valuable practices for scientific advancement are often ignored and less rewarded in favor of traditional metrics.
In an effort to more appropriately assign value to both traditional quantitative metrics and non-traditional metrics, academic career assessment practices are being reevaluated by institutions worldwide. Although more institutions are looking to change, information on the creation and execution of new practices is not readily available. Specifically, the webinar highlighted four dimensions of institutional change: What, Who, Why, and How.
What: what changed and key characteristics
EUA Policy and Project Officer Bregt Saenen noted that all of the case studies shared a similar goal of not necessarily abandoning quantitative publication metrics, but at least de-emphasizing bibliometrics while moving toward a more holistic approach. The final goal of this shift is to incentivize a broad range of academic activities and ultimately improve academic culture.
One major challenge institutions face is the lack of faculty awareness in terms of implementing non-traditional narrative evaluations. In addition to offering an application template to guide faculty through the application process, Nele Brecke emphasized the importance of taking faculty concerns seriously and facilitating an open dialogue between the faculty and institutional leadership. Marta Aymerich noted that the Open University of Catalonia (UOC) established a rubric with examples of scientific achievements and societal impact to help researchers better understand what was required of them.
Janne Pölönen from the Federation of Finnish Learned Societies pointed out that even when there is a strong commitment to reform, implementation at the institutional level remains complicated. Limited capacity is a major barrier to achieving long-term goals. Moving away from traditional metrics requires reliable information on other, non-traditional areas of academic assessment. Pölönen emphasized the importance of investing in the development of systems where researchers can report diverse activities. To address this challenge, the Federation of Finnish Learned Societies is developing national and international open science infrastructure dedicated to supporting research assessments.
Who: the stakeholders involved and driving the process
Coalition building with stakeholders is critical for gaining bottom-up, grass-roots momentum for research assessment reform.
The Recognition and Rewards Program in the Netherlands used coalition building to bring stakeholders together to create a position paper with concrete proposals for next steps to recognize and reward a wider range of academic work. According to Kim Huijpen, Recognition and Rewards in Academia works with university medical centers, research funders, and research institutes. The position paper articulated a unifying set of guiding principles that could be applied to facilitate institution-specific, bottom-up change.
Finn-Erik Johansen, University of Oslo, discussed the assembly of a national working group from universities around Norway. The working group generated a flexible guide of research assessment practices for individual institutions and funding agencies to implement. Johansen highlighted two key points: the importance of generating a national, flexible template that can fit the unique needs of each university, and the critical importance of working hand-in-hand with the Norwegian government funding agency.
Why: motivation for change
The third dimension discussed during the webinar was the reasoning behind why action was taken. Across the case studies, there were unique combinations of internal and external influences that drove motivation for change within institutions and national consortia.
Marta Aymerich spoke on systemic challenges associated with an individual university implementing academic assessment reform within the context of nationally or regionally established academic assessment practices. Aymerich explained that UOC postdoctoral researchers may be at a career disadvantage due to the heavy national and international focus on publications over holistic measurements of research success. Aymerich emphasized the need to implement change slowly, to allow for transformation at a national level and encourage consistency in research assessment practices for policy makers and research funders.
How: processes and dynamics for developing, implementing, and managing change
The primary aim of the case study repository is to understand how organizations implemented change. To this end, the case studies document the processes and dynamics of developing, implementing, and managing change. Across the case studies, the cooperation of both administrative or leadership and grass-roots faculty networks played an important role implementing change.
Rinze Benedictus and Frank Miedema, University Medical Center Utrecht, discussed the importance of using bottom-up and top-down approaches, and how the combined efforts of leadership and faculty supported iterative processes for policy refinement. Benedictus outlined two critical components for implementing change at the University Medical Center Utrecht: First, the role of the Dean, Miedema, in opening up institutional discussions about research assessment practices. Second, facilitating meetings, debates, workshops, and committees with researchers representing diverse fields and stages in professional development. Miedema emphasized that it is critical for leadership to repeatedly explain the higher purpose of reforming academic research assessment practices: accurately rewarding a broad range of academic outputs.
Moving forward, Saenen and DORA program director Anna Hatch emphasized the importance of including universities and organizations outside of Europe in future case studies. To this end, Neil Smyth, Associate Director of Research and Learning at the University of Nottingham Ningbo China, spoke in a recorded video on his organization’s commitment to carrying out the mission of DORA and the importance of equitable research assessment practices.
The collaboration among DORA, EUA, and SPARC Europe has yielded ten case studies of change in academic career assessment that can serve as both inspiration and a roadmap to other institutions looking to make similar change. This is just the start as the case study repository continues to grow to include a broader geographic range of organizations, institutions, and initiatives. A full recording of the webinar can be found here.
Haley Hazlett is DORA’s Policy Intern