As we wind down our tenth anniversary celebration this May 2023, we’re taking a look back at our evolution from a statement of principles to a global initiative for responsible research assessment, and the wider issues in research assessment that DORA has sought to address.
Why a declaration on research assessment?
“When quantitative measures have an outsized impact on how people are rewarded, it can increase the temptation to focus on a narrow set of activities and reduce investment in other meaningful, but less rewarded, achievements.”
DORA brief Unintended Cognitive & Systems Biases
When the San Francisco Declaration on Research Assessment (DORA) was drafted in 2012, publishing in prestigious journals with a high Journal Impact Factor (JIF) was, and for many still remains, one of the primary mechanisms for judging the “impact” and “quality” of a researcher’s work. DORA was derived initially to address the oversized impact that the JIF, a journal-level metric, had when used erroneously to assess the quality of individual research articles. Though the JIF can be skewed and variable across disciplines, it remains a frequently used metric to inform decisions around hiring, promotion, tenure, and funding. The pervasive use of inappropriate proxy measures of quality, like the JIF, feed into a “publish or perish” cycle that devalues scholarly contributions like engagement with local communities; rigor and reproducibility; the generation of intellectual property and software; mentorship; academic service; and work to improve diversity, equity & inclusion. Though others had raised issues with the JIF, DORA was the first collaborative initiative to address it.
It was in this environment that a group of like-minded publishers and academics convened to address pervasive misuse of the JIF and the devaluing of other scholarly contributions in research assessment practices. The Impact Factor Discussion Group, which gathered for the first time at the 2012 Annual Meeting of the American Society for Cell Biology (ASCB) and European Molecular Biology Organization (EMBO), worked to draft DORA over several months. The declaration was eventually published in May 2013 with 155 individual signatories and 82 organizational signatories including research funders, publishers, and professional societies. While DORA offers 18 recommendations for improving research assessment practices for funding agencies, institutions, publishers, organizations that supply metrics, and researchers, it is perhaps best known for its first general stricture: “Do not use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions.”
After its publication in 2013, a group of volunteers set out to make the declaration a rallying point for the responsible research assessment movement, collecting individual and organizational signatures to demonstrate the widespread support and solidarity among the academic community. These efforts were driven by many of the founding members, including Bernd Pulverer, Stefano Bertuzzi, Mark Patterson, and Sandy Schmid, plus other advocates such as Stephen Curry and Cationa MacCallum. After five years of gathering a coalition of support, DORA had collected signatures from 500 organizations and 12,000 individuals. It was time to make a change and move towards catalyzing action in a more concrete fashion.
Putting advocacy into practice
In February 2018 it was announced that DORA would intensify its efforts through direct action made possible by funding and in-kind support from ASCB, Cancer Research UK, the Company of Biologists, eLife, EMBO, F1000, Hindawi, PLOS, and Wellcome. This was accompanied by the launch of the DORA website and hiring DORA’s first full time staff member, Anna Hatch, who went on to become DORA’s first Program Director and a key catalyst who propelled DORA into action. Curry was named Chair of DORA in October 2018.
“I think culture change takes root and grows most efficiently when communities are engaged. DORA’s evolution from a declaration of intent to a global initiative is emblematic of the power of culture change. By bringing different viewpoints and perspectives in the academic ecosystem together to facilitate knowledge sharing, and by building coalitions, such as the funder discussion group, DORA demonstrates that change has a catalytic effect.”
Anna Hatch, DORA Program Director from 2018-2022
Convening, coalition building and expanding scope
Obtaining the funding to hire skilled and dedicated staff to support DORA’s mission fulltime was integral to the organization growing into a global initiative. From 2018 to 2022, Hatch deftly ushered in a new era of DORA characterized not only by increased signatory numbers but also more outreach, advocacy, and coalition building within the broader academic community. This period also was a time when DORA consolidated its governance procedures and reached out beyond Europe and the Americas. In 2018, Hatch kicked off DORA’s five-year anniversary with a live-interview series featuring community members who were working to implement responsible research assessment at their institutions. In 2019, DORA also hosted a joint meeting with the Howard Hughes Medical Institute (HHMI) Driving Institutional Change for Research Assessment Reform that convened a group of scholarly actors from around the globe. The group explored practical approaches and strategies to reduce reliance on the JIF and inappropriate proxy measures of quality, to discuss how institutions might align their evaluation policies and practices with their values, and how to build a community to support these changes long-term. This was also complemented by a 2019 Participant Commentary blog series and a 2020 report Research Culture: Changing how we evaluate research is difficult, but not impossible.
Building on the momentum of the joint meeting it was clear that there was an appetite to drive change, but a lack of forums for community members to gather for productive discussions about how to implement reform in their sectors. To meet this need, in 2020 DORA launched its first ever Community of Practice, the Research Funder Discussion Group, for public and private research funders. Two parallel funder groups, one for the Asia-Pacific region and another for the Africa, Americas, and Europe regions, now meet virtually each quarter.
“DORA realized early on the importance of regional representation and that’s been key to serious engagement for example, in the Asia-Pacific.”
Ginny Barbour, DORA Vice-Chair
The funder discussion groups provide a space to discuss new assessment policies and practices. During the funder discussion group meetings, we increasingly heard that funders were interested in innovating – for example, through implementing narrative style CVs for funding applicants. Because the group was keen to avoid recreating pitfalls and biases associated with more traditional CVs, DORA and the Funding Organisations for Gender Equality Community of Practice (FORGEN CoP) co-organized two workshops for research funders. Based on the discussion from these workshops, a short report was published outlining key takeaways and recommended actions. To align on shared definitions and use-cases, DORA also co-organized two funder workshops with FORGEN CoP, UK Research and Innovation, and the Swiss National Science Foundation. Key takeaways from these workshops were shared in a blog report that outlined common principles for the use of narrative CVs in grant evaluation. DORA’s work around narrative-style CVs is ongoing and will focus on testing the results of implementation through experimentation.
Following the success of the funder discussion groups, in 2021 DORA convened the quarterly National and International Initiatives Discussion Group. This group is composed of organizations and initiatives, like DORA, that are advocating for research assessment reform and is helping to avoid duplication of work and support collective action.
Finally, researchers and research-enabling staff at academic institutions have also expressed a desire for a community space to discuss challenges, innovations, and opportunities for collective action. To meet this need, we plan to create a Community of Practice for members from academic institutions later this year.
Resource development and building awareness
Research assessment reform is a systems problem that can only be addressed by the collective action of all actors in the scholarly ecosystem. To support this, in 2020 DORA created a resource library with both community- and DORA-produced resources to support change across a wide range of actors, including publishers, funders, and research institutes. Since 2020, DORA has produced or co-produced a number of key resources for a broad range of needs and audiences. Balanced, broad, responsible: A practical guide for research evaluators is a video and one-pager co-created with the Luxembourg National Research Fund to provide tips for best practices for grant reviewers. In collaboration with Ruth Schmidt, Associate Professor at the Institute of Design at the Illinois Institute of Technology, DORA has published numerous resources for academic institutions, including: The SPACE rubric, Unintended Cognitive Systems Biases brief and Ideas for Action brief.
As the number of academic organizations seeking to implement responsible academic assessment practices has increased, so too has demand for concrete examples of reform in action. To meet this demand, DORA, the European University Association (EUA), and SPARC Europe launched the case study repository “Reimagining academic assessment: stories of innovation and change” in 2020. As of May 2023, the repository includes 13 case studies of practical change processes.
The case study repository is not the only resource that DORA has created to provide concrete examples of change processes at academic institutions. In 2021, DORA was awarded a 3-year, $1.2M grant from Arcadia – a charitable fund of Lisbet Rausing and Peter Baldwin. The generous funding supports Tools to Advance Research Assessment (TARA), a collaboration with Sarah de Rijcke, Professor in Science and Evaluation Studies and director of the Centre for Science and Technology Studies (CWTS) at Leiden University, and Ruth Schmidt. Members of the Project TARA team also include Stephen Curry (DORA), Alex Rushforth (CWTS), Marta Sienkiewicz (CWTS), and Haley Hazlett (DORA). Project TARA consists of three key outputs:
- A community tool that will track processes, policies, and practices related to responsible research assessment at academic institutions. This will be released in 2023.
- A survey of U.S. academic institutions to gain a better understanding of their familiarity with DORA and approaches to responsible research assessment. de Rijcke and Rushforth shared these results via preprint in May 2023: Practicing Responsible Research Assessment: Qualitative study of Faculty Hiring, Promotion, and Tenure Assessments in the United States.
- A toolkit of resources for academic institutions to support their efforts to implement responsible research assessment reform. The first two tools were released in 2022: Debiasing Committee Composition and Deliberative Processes and Building Blocks for Impact.
Direct support
In 2021, DORA took its first steps to provide direct, financial support to community members working to support responsible research assessment policies and practices, by launching the pilot Community Engagement Grants Program. In 2022, 10 teams from 9 countries were awarded funds for projects ranging from awareness and coalition building, surveying attitudes towards research assessment, creation of frameworks for assessment, and much more (read their reports here). The Community Engagement Grants Program empowered advocacy and change in a context-specific manner, demonstrating that DORA could play a more direct role catalyzing change.
Beyond discouraging the JIF: Responsible research assessment, equity and openness
Although the declaration itself is a product of its time and does not explicitly mention “equity” or “open scholarship”, responsible research assessment practices are intimately intertwined with both. In 2020, Hatch, Curry and Barbour took a critical look at The intersections between DORA, open scholarship, and equity: “Here we examine the growing interactions between DORA and the open scholarship movement. By clarifying the alignment of the values and principles that underpin both endeavors, we see that they raise vital questions about equity and inclusion in research that must be central to reform within research organizations and the wider scholarly community.”
DORA has also endeavored to live up to its own standards of inclusion, announcing in 2022 a restructure of our governance procedures to redistribute power outside of Europe and North America on the Steering Committee and reduce financial barriers for organizations in developing or transitional economies that wish to support DORA. The Steering Committee now includes members from over 15 countries across North and South America, Africa, Europe, and the Asia-Pacific. Read more about DORA’s governance restructuring in Curry’s 2023 piece Ten years of Dora.
The future of responsible research assessment
While DORA has experienced an intensive period of growth in terms of scope and reach these past 5 years, we are not slowing down. This year, we announced a strategic plan that defines four objectives to provide the framework and focus for our work over the next three years. In the coming years, we will work to:
- Increase awareness of the negative impacts of research assessment practices that are too dependent on inappropriate metrics, and of the positive impacts of alternative practices
- Accelerate the development of clear and concrete measures to reform research assessment
- Support advocates of research assessment reform worldwide
- Secure the funding needed to deliver DORA’s mission as efficiently and as rapidly as possible.
Change on a global scale and across so many scholarly actors would not be possible without many related initiatives and organizations such as the Leiden Manifesto, the Hong Kong Principles for assessing researchers, the Latin American Forum for Research Assessment (FOLEC), International Network of Research Management Societies (INORMS), and the new Coalition on Advancing Research Assessment (CoARA). As we celebrate a decade of DORA, we are cognizant that there is still much to be done. Changing research assessment requires system-wide approaches and DORA will continue to collaborate globally to support and drive change towards responsible research assessment.
“DORA owes a great deal to a great many people, from the original founders in that San Francisco conference room in 2012, to the allies from around the world who have rallied to the declaration’s call ‘to improve the ways in which the output of scientific research is evaluated’. In its first 10 years, DORA has moved research assessment reform to center-stage and collaborated widely to develop real solutions. The challenge for the years ahead is to ensure these solutions take root so that our assessment practices align fully with our highest aspirations for the best that research has to offer humanity.”
Stephen Curry, DORA Chair
Ginny Barbour is DORA’s Vice-Chair
Haley Hazlett is DORA’s Acting Program Director