2019 in review: DORA’s list of the top 10 advances in research assessment

As 2019 winds down, the DORA steering committee and advisory board wanted to highlight the ways research assessment reform has advanced in the last year. From new data on assessment policies to the development of new tools, the scholarly community is taking action to improve research assessment in concrete ways. Below, we highlight our 10 favorites:

1. New research from the Scholarly Communications Lab at Simon Fraser University and their collaborators has mapped out how widely the Journal Impact Factor is used in review, promotion, and tenure decisions in the United States and Canada. Studies like this one help us to understand the scale of the challenge facing DORA, but also serve as a stimulus for institutions to innovate and improve their assessment policies.
2. One institution that has already taken steps to reform its research assessment practices is the Universitat Oberta de Catalunya, which has signed DORA and released an action plan and timeline to improve research assessment on campus. By the end of 2020, the university aims to propose indicators to measure the social impact of research. They also intend to expand the list of research outputs to assess the impact and transfer of research. The plan was developed by the university’s DORA working group and approved by the Research and Innovation Commission. While it was released at the end of last year, we wanted to recognize the university’s efforts throughout 2019 to implement widespread changes.
3. In September, the Research on Research Institute (RORI) was launched in London to advance more strategic, open, diverse and inclusive research. The institute is a partnership among Wellcome Trust, Digital Science, and the Universities of Sheffield and Leiden. An important aspect of their work will focus on measurement and evaluation, including broadening and diversifying criteria and indicators used in evaluation of research. More broadly, by providing a deeper understanding of the machinations of the research landscape, RORI aims to improve decision-making, career advancement and the culture of research.
4. The Hong Kong Principles for Assessing Researchers: Fostering Research Integrity were developed as part of the 6th World Conference on Research Integrity. Institutions and funders can adopt the principles to recognize scholars for activities that lead to trustworthy research. Similar to DORA, the Hong Kong principles emphasize the importance of considering the value of all contributions, outputs, and outcomes of scholarly work.
5. In May, the revised implementation guidance for Plan S established research assessment reform—in accordance with DORA—as a core principle. Members of cOAlition S are committing to revise their policies by January 2021 to assess research on its own merits rather than the venue where it is published.
6. DORA advisory board member Needhi Bhallla published a perspective summarizing several proven strategies to improve equity in faculty hiring. Some examples include evaluating and revising departmental review and promotion processes and developing a rubric to assess diversity statements at the beginning of a search. By adopting these practical measures, institutions also increase transparency and consistency of faculty searches.
7. Many organizations are rethinking how to structure job and grant applications to encourage robust but time-efficient review processes that are detached from journal impact factors. In October, the Royal Society released the Résumé for Researchers to recognize a wider range of scholarly outputs and accomplishments. The Dutch Research Council (NWO) is also piloting a narrative CV format for its Veni funding scheme and announced in December that it was going to do the same for the Vici round. Narratives permit researchers to contextualize their work and varied contributions in a way that numbers cannot.
8. In October, DORA and the Howard Hughes Medical Institute convened a diverse group of stakeholders to consider how to improve policy and practice by exploring different approaches to culture and systems change. The background readings and participant commentaries provide a better understanding of the opportunities and challenges research institutions face with research assessment reform. Toolkits to help institutions adopt new policies and practices will be made available in 2020.
9. The Latin American Council of Social Sciences (CLACSO) and the National Research and Technology Council of Mexico (CONACYT) organized a Forum on Research Evaluation in November to kick off a two-year working group to identify alternative evaluation procedures for social sciences in Latin America. Stay tuned for updates on their work.
10. A number of cross-sector initiatives are aiming to develop more positive visions of research culture, which is intimately connected to and impacted by research assessment practices. The Wellcome Trust, for instance, is looking to redefine ‘excellence’ by including consideration of how research is conducted. And the Royal Society is following up its conference on research culture by sharing examples of how different organizations are working to eliminate bullying and harassment; to improve equity, diversity, and inclusion; and to foster greater collaboration.

A lot can happen in a year. It is difficult to limit ourselves to just 10 advances. While the following did not fit in the top 10, we think they deserve recognition too:

• The number of DORA translations is still growing. It can now be read in 21 languages!
• Released in May, the Helsinki Initiative on Multilingualism in Scholarly Communication promotes language diversity in research assessment.
• Published in April, the feature article “How Journals and Publishers Can Help to Reform Research Assessment” In Science Editor includes a call to action with 10 specific actions for journals and publishers.
• In addition, there was a call to action in May for the expansion of indicators to describe a scholarly journal’s qualities.
• More organizations are supporting DORA than ever before. We now have 14 members contributing financially.
• The SCOPE process developed by the INORMS Research Evaluation Working Group was announced in December and can be used to help institutions create new policies and practices.