Introduction
Over the past year, it has become increasingly clear that research assessment reform is a systems challenge that requires collective action. Point interventions simply do not solve these types of complex challenges that involve multiple stakeholders. Because of this, we dedicated our efforts in 2020 on building a community of practice and finding new ways to support organizations seeking to improve the decision-making that impacts research careers.
Current events also influenced our approach this year and evolved our thinking about research assessment reform. The Covid-19 pandemic led to the abrupt global disruption of academic research, along with many other industries. For academics with limited access to research laboratories and other on-campus resources, work stalled. Without appropriate action, this disruption will have a profound effect on the advancement and promotion of the academic workforce, and it will likely disproportionately affect women and underrepresented and minoritized researchers. So in April DORA called on institutions to redefine their expectations and clearly communicate how evaluation procedures will be modified. In May, DORA organized a webinar with Rescuing Biomedical Research to better understand specific faculty concerns as a result of the pandemic.
Demonstrations for racial justice in the wake of the killing of George Floyd in the United States called attention to longstanding systemic racial inequities, including academic career advancement. As institutions work to implement the principles of DORA, there needs to be a broader representation of researchers in the design of research assessment practices that directly address the structural inequities in academia.
Examining our role in scholarly communication
DORA’s vision is to advance practical and robust approaches to research assessment globally and across all scholarly disciplines. With that in mind, the Steering Committee and Advisory Board developed a core set of objectives and approaches in the first half of 2020 to guide our work.
Objectives
- Raise awareness
- Facilitate implementation
- Catalyze change
- Improve equity
Approaches
- Community engagement
- Resource development
- Partnership
- Advising
- Convening
Clarifying the work DORA does and how it gets accomplished is not the only area where we sought to examine our purpose. DORA has observed and participated in many discussions about academic assessment that inevitably led to deeper conversations about equity and open scholarship. This prompted the Steering Committee and Advisory Board to examine the intersections between responsible researcher evaluation, open scholarship, and equity. Because these topics interact in many complex ways, DORA identified a set of principles where they align.
The concepts of (i) research assessment reform, (ii) open scholarship, and (iii) equality and inclusion cannot be treated separately. They interact strongly and in many complex ways – presented only in broad outline here – and are converging to create a research culture that is centered on people (practitioners and beneficiaries) and on values that embody the highest aspirations of a diverse world.
The intersections between DORA, open scholarship, and equity. 2020.
Building and supporting communities of practice
Solving systems challenges like research assessment reform require information sharing and coordinating actions across stakeholders and stakeholder groups. Research funders play a critical role in reimagining academic assessment, because their policies directly and indirectly influence the ones at academic institutions. So DORA launched two communities of practice in 2020 for public and private funders of research that meet quarterly. The goal of these groups is to increase communication and facilitate collective action for fair and responsible research assessment.
DORA also creates space for the broader academic community to discuss and learn about topics related to research assessment reform through webinars. In 2020, we invited speakers to deepen our understanding of the importance of recognizing academic outputs in diverse languages and the challenges presented by an evaluation system that favors English publications. More than 150 people joined us in April to hear librarians share their experiences working to create change and offer ideas about the roles libraries can play in research assessment reform. By dissecting academic hypercompetition in the United States, we learned how this phenomenon has led to systemic flaws embedded in the current biomedical research system.
In July, more than 100 people tuned in to hear advocates and policy makers from the Asia-Pacific talk about progress to improve research assessment in the region. We came to better understand the obstacles preventing action and opportunities to leverage change. DORA also sponsored a webinar for Open Access Week in October with UNESCO and partners in Latin America reflecting on the evolution of academic assessment and how current evaluation practices stand to impact the future of Open Access.
Creating resources to support institutional change
Since its release in 2013, the declaration has become a powerful tool to raise awareness about research assessment reform. The Steering Committee recognized, however, that more work was needed to translate that awareness into action. So over the past two years DORA has been engaging with the academic community to generate tangible action for research assessment reform. In 2020, DORA began developing resources to support academic institutions in navigating research assessment reform.
Changing how we evaluate research is difficult, but not impossible
Inspired by discussions at the 2019 meeting DORA co-sponsored with the Howard Hughes Medical Institute (HHMI) on driving institutional change for research assessment, DORA published a framework for action in August. It includes four broad goals for academic institutions: 1) understand obstacles that prevent change; 2) experiment with different ideas and approaches at all levels; 3) create a shared vision for research assessment when reviewing and revising policies and practices; and 4) communicate that vision on campus and externally to other organizations. We learned from the meeting that institutional change demands careful examination of policies, processes, and power structures. It also requires organizations to take stock of their values to see whether they are reflected in the decisions that impact research careers.
Ideas for Action
We also captured learnings from the meeting in a brief entitled “Rethinking Research Assessment: Ideas for Action.” It addresses five common myths about evaluation to motivate academic leaders to take action on research assessment reform and offers five design principles to encourage experimentation and development of better practices. The brief launched DORA’s partnership with Ruth Schmidt, Associate Professor at the Institute of Design of the Illinois Institute of Technology to develop tools that support individuals and organizations working to improve policies and practices.
Unintended Cognitive and Systems Biases
The second brief developed with Schmidt was released in the Fall and offers strategies to counteract the institutional implications of cognitive biases on academic assessment. While decision-making biases occur at a personal level, they have institutional implications. The brief highlights seven specific decision-making biases, offers examples of how each one manifests in academic assessment, and explains why it is problematic. It also examines the institutional implications of these biases and provides strategies for academic institutions to address them. Additional context for the brief was published in a series of blog posts. Universities are paying attention: the Office of the Provost at the University of Richmond added the brief to its list of faculty hiring resources.
Judgment and decision-making biases that impact how we weigh options and make choices have been shown to result in inequitable review, promotion, and hiring practices. While recognizing these biases at a personal level is important, creating new structural and institutional conditions to reduce bias can be even more valuable.
Rethinking Research Assessment: Unintended Cognitive and System Biases. 2020.
SPACE to Evolve Academic Assessment
In the Fall of 2020, DORA initiated a new community project with Schmidt to develop a means for institutions to gauge their ability to support academic assessment interventions and set them up for success. Our goal for the project was to support the development of new practices by helping institutions analyze the outcomes of their efforts. More than 70 individuals in 26 countries and 6 continents responded to our informal survey in August, and about 35 people joined us for 3 working sessions in September. From these activities, we heard it was important to look beyond individual interventions to improve assessment, because the success of these interventions depends on institutional conditions and capabilities. We were also reminded that institutional capabilities impact interventions, so it is important not only to gauge success but also to support interventions. These and other insights led us to create SPACE to Evolve Academic Assessment: a rubric for analyzing institutional conditions and progress indicators. The first draft of the rubric was developed in the last quarter of 2020. The final version was released in 2021 after an initial pilot phase with seven members of the academic community, including a college dean, policy advisor, research administrator, faculty member, and graduate student.
Resource library
Because knowledge-sharing is central to DORA’s vision of advancing fair and responsible research assessment, we released an updated website in December designed to help users easily identify and surface relevant content. Part of the updates included the creation of a resource library that combines and expands on DORA’s existing collection of good practices and resources. The library is searchable and is organized based on type and intended audience.
Reimagining Academic Assessment: stories of innovation and change
Another addition to the website was a repository of case studies documenting key elements of institutional change to improve academic career assessment, such as motivations, processes, timelines, new policies, and the types of people involved. The repository, Reimagining academic assessment: stories of innovation and change, was produced in partnership with the European University Association and SPARC Europe. At the time of launch, the repository included 10 structured case studies coming from 7 universities and 3 national consortia. Nine of the 10 cases are from Europe and one is from China. The case studies have shown us the importance of coalition-building to gain bottom-up support for change. We also learned that limited awareness and capacity for incentivizing and rewarding a broader range of academic activities were challenges that all the cases had to overcome. By sharing information about the creation of new policies and practices, we hope the case studies will serve as a source of inspiration for institutions seeking to review or improve academic career assessment.
Reaching out to the community
As mentioned earlier, a key part of DORA’s work this year was identifying ways to better support research assessment reform in the academic community. In addition to resource development, we introduced a bimonthly newsletter to increase knowledge sharing about research assessment reform. It offers information about new resources, upcoming events, and opportunities to get involved with DORA activities. DORA’s program director and members of the Steering Committee and Advisory Board collectively gave 20 invited presentations about fair and responsible research assessment this year in more than 9 countries.
While the number of new DORA translations being added to the website has slowed down, 3 new ones were added in 2020, bringing the total to 23. The Russian translation was also published as an open access article in the Russian journal Научный редактор и издатель, helping it to reach a wider audience. We thank the volunteers who write and review the translations!
Conference sessions are another way DORA reaches out to the broader academic community to raise awareness about fair and responsible research assessment. At the ESOF2020 Euroscience Open Forum meeting in September, DORA organized a session on addressing obstacles that prevent meaningful changes in research assessment by funders, academic institutions, and researchers. DORA also hosted a session at Cell Bio Virtual 2020 in December to examine emerging trends at the intersection of research publishing, open science, and academic reward and recognition systems.
Observing policy progress
Policy progress for research assessment reform continued to gain momentum in 2020. A new national policy on research assessment in China announced in February prohibits cash rewards for research papers and indicates that institutions can no longer exclusively hire or promote researchers based on their number of publications or citations. In June, Wellcome published guidance for research organizations on how to implement responsible and fair approaches for research assessment that are grounded in the DORA principles: Starting in January 2021, Wellcome-funded organizations will be asked to publicly commit to assessing research outputs and other research contributions based on their intrinsic merit and discouraging the inappropriate use of proxies or metrics.
Science Europe issued a new position statement and recommendations on research assessment processes in July that seeks to promote assessment of research quality that is effective, efficient, fair, and transparent. In the United States, DORA was mentioned three times in the public responses to The White House Office of Science, Technology, and Policy (OSTP) request for information on public access to peer-reviewed scholarly publications, data, and code resulting from federally funded research.
The Global Research Council organized a virtual conference on responsible research assessment in November to explore the research ecosystem as a whole, the stakeholders who influence and impact research assessment, and the roles funders play in assessment. In advance of the meeting, the Research on Research Institute in partnership with the Global Research Council, DORA, UK Research and Innovation (UKRI), and the National Research Foundation in South Africa published a white paper examining how the role of funders in responsible research assessment is evolving.
Funders are not the only organizations taking action. In March, DORA reflected on the progress being made at academic institutions to improve the ways decisions are made that impact careers. At the end of the year, the Steering Committee and Advisory Board published their annual list of new developments and essential reading for research assessment reform.
Looking ahead
The declaration’s signatories have shared ambitions to improve the ways in which researchers and the outputs of scholarly research are evaluated. More than 1,300 individuals and 290 organizations signed the declaration in 2020. We have learned that signatures do not automatically translate into fair and responsible research assessment practices. Culture change for research assessment requires collective action. We aim to build on our efforts in 2020 in improve knowledge-sharing and create resources to stimulate collective action by research funders and academic institutions.
DORA’s portfolio will expand in 2021 thanks to a generous $1.2M grant from the Arcadia – a charitable fund of Lisbet Rausing and Peter Baldwin supporting Tools to Advance Research Assessment (TARA). The aim of Project TARA is to identify, understand, and make visible the criteria and standards universities use to make hiring, promotion, and tenure decisions. This information will be used to create resources and practical guidance on the reform of research assessment for academic and scholarly institutions. Project TARA is a collaboration with Sarah de Rijcke, Professor in Science and Evaluation Studies and Director of the Centre for Science and Technology Studies (CWTS) at Leiden University, and Ruth Schmidt, Associate Professor at the Institute of Design at the Illinois Institute of Technology.
Through knowledge-sharing, resource development, and community engagement, DORA is working to support culture change and catalyze the development of fair and responsible research assessment policies and practices.