As Alison Mudditt described in her Scholarly Kitchen post last month, the path to reforming research assessment has been met with significant challenges. We agree with her that culture change is often a slow process. However, as DORA demonstrates, it is possible to identify tangible progress on the path to large-scale research assessment reform.
Thanks to DORA there is an ongoing conversation to rethink research assessment and what we value in academia. By adding their name to the declaration, DORA signers are making public commitments and holding themselves accountable to their staff and the broader research community. In many cases, these institutions are taking real steps to implement DORA principles and drive research assessment reform.
For example, the DORA-inspired working group at the Universitat Oberta de Catalunya (UOC) met monthly for almost a year to develop a plan (still underway) for research assessment reform. The first step was to revise the criteria for hiring postdoctoral researchers by removing the Journal Impact Factor as an indicator of success and reclassifying “scientific output” to “main scientific achievements.”
In fact, working groups are a common first step for many institutions. The working group at Cardiff University in the UK developed an action plan and appointed a Responsible Research Assessment Officer to support its implementation efforts. In Finland, the University of Oulu established a working group to develop its policy for responsible researcher assessment and create a plan for implementation. So far, the university has identified a list of considerations, including equity, differences between disciplines, and how metrics can support qualitative assessment. In France, the CNRS is using a new system to evaluate individual researchers on DORA principles.
Some academic departments are positioned to make their own changes. For example, the Cell Biology Department at UT Southwestern Medical Center reinvented the traditional faculty search process by getting rid of the search committee to involve the entire faculty in the hiring process. Applicants for their part are directed to focus on the cover letter, which plays a large role in the initial review process. The candidates selected for Skype interviews receive questions ahead of time to identify more thoughtful candidates in addition to those who process information quickly. And each candidate has a designated faculty advocate who can speak to their strengths during hiring deliberations.
For more widespread change, a coalition of Dutch universities and funders is working together to advance research assessment so that emphasis is placed on the content of a contribution rather than the reputation of the venue where it is published. The Wellcome Trust recently released draft guidance for the institutions they fund to implement DORA principles, and this will undoubtedly have an impact on processes on the ground. The meeting that DORA co-organized with the Howard Hughes Medical Institute (HHMI) in October 2019, Driving Institutional Change for Research Assessment Reform, was also a source of input for the guidance. Change is not limited to DORA signatories. Ghent University in Belgium and the Quest Center for Transforming Biomedical Research at the Berlin Institute for Health have taken significant steps to improve research assessment practices.
Culture change may be slow, and although we have not reached our end goal, DORA is making meaningful progress to advance research assessment reform. We have a group of innovating institutions providing models of change, which serve as a guide for others to follow. Together, these steps forward will bring us closer to systems change for research assessment reform.
Anna Hatch is the Program Director for DORA