DORA at 10: A look at our history and the bright future of responsible research assessment (Africa, Americas & Europe Plenary)

This is the second in a two-part series reviewing and recapping DORA’s two 10th Anniversary plenary sessions (watch the recording of both sessions here.)

With the evolution of a research landscape driven by technological progress and societal needs, there has been a growing demand to rethink and reform research assessment that could enhance the quality and culture of a scholarly ecosystem. The Declaration on Research Assessment (DORA) is a worldwide initiative to address these needs.

To celebrate its 10th Anniversary, DORA hosted a plenary session for the Africa, Americas, and Europe region on Tuesday 16 May 2023. The event started with opening remarks from Stephen Curry, DORA’s Chair, followed by a keynote address by Sarah de Rijcke, Leiden University, Netherlands. The second half of the session was composed of a panel discussion by Andiswa Mfengu of University of Cape Town, South Africa; Karen Stroobants, Vice-Chair CoARA, UK; Judith Sutz of Universidad de la República, Uruguay; and Stephanie Warner of University of Calgary, Canada. DORA’s Acting Program Director, Haley Hazlett, shared closing remarks at the end of the session. Chris Hartgerink, Liberate Science, and Queen Saikia, DORA Policy Associate, provided technical support during the session.

Opening remarks: DORA into the next decade

To recognize the depth and breadth of research and innovation talent, the scholarly community needs to advocate for best practices with an understanding that responsible research assessment is a systems challenge composed of interlocking and intersecting issues. Stephen Curry began the plenary session with a discussion of the history of DORA, which was drafted in 2012 at the annual meeting of the American Society of Cell Biology (ASCB) in San Francisco and published in 2013. Ten years later, DORA has been signed by over 23,000 signatories worldwide and its recommendations have been adopted by research funders, publishers, academic institutions and professional societies. Curry highlighted one key and recent initiative of DORA, the Tools to Advance Research Assessment (TARA)  Project, which aims to support community implementation of responsible academic assessment through the creation of tools and resources for hiring, promotion, and tenure decisions. The key objectives of Project TARA include an interactive web resource (Reformscape) which will help scholarly actors discover policies and processes for the implementation and uptake of responsible research assessment (RRA) practices worldwide. Project TARA also includes a survey of US institutions to understand their approaches to RRA, and a toolkit to address identified issues, which is currently under its development process. Details about the significant milestones and activities that have shaped DORA’s progress can be found in the recent retrospective blog.

Keynote address: The future of research evaluation

The future of research evaluation is likely to encounter unforeseeable changes driven by advancements in technology, evolving scholarly practices, and the need for more comprehensive and transparent assessment methods. In her keynote lecture, Sarah de Rijcke pointed to a few trends and possibilities that could shape the approaches to future assessment reform. For example, de Rijcke discussed the move to open science, technological developments such as Artificial Intelligence (AI), and the rise of social media. Despite the perks that technological advancements could bring, research communities, governments, civil societies and other actors governing the space should keep thinking about how these align with their values and norms around, for instance, integrity.

Signing DORA is a mechanism to demonstrate support for RRA and, for organizations, signals intent to implement reform. Over 2800 organizations from across the world have signed DORA, many of them in Europe. One striking trend that has been consistently observed among DORA signatories is in the relative lack of signatories from the US. For example, as of June 2023 there are 258 organizational signatories in the UK and 166 in the US. This suggests a need to understand institutional attitudes and approaches to qualitative assessments over quantitative indicators in the US. To meet this need, Alex Rushforth from Leiden University developed a survey and held interviews as part of Project TARA that spotlighted plurality in opinions of US researchers and evaluators. de Rijcke, also a member of the Project TARA team, outlined these results. Some respondents showed strong endorsement towards responsible metrics; some showed partial alignment but were unprepared to make it happen; and a fraction of them showed pragmatic rejection expressing no desire to move away from research evaluation “norms”. The survey results suggest that it is essential to raise awareness among US institutions to adopt evaluation principles aligning with the values of RRA, scientific excellence, innovation and the broader goals of the society. However, it is noteworthy that some other initiatives such as HELIOS, PTIE, and HuMetricsHSS have been successfully accepted in the US and are actively working to address these challenges. On being asked if we need a consensus before driving for a change in the context of US, and how DORA could help: de Rijcke said that consensus is need-driven and there was no overall consensus from the US study, however there’s a progressive change for the ones that are supportive of research culture. DORA can help by asking questions, putting forward concrete practices, and collaborating with US-based initiatives.

de Rijcke later brought the audiences’ attention to the discussion paper, The Future of  Research Evaluation that presents an outlook on research assessment reforms across the globe. Out of several case studies, de Rijcke highlighted a case study on China’s biggest funding organization, the Natural Science Foundation of China (NSFC),that reflects important shifts in the systemic reforms by moving away from bibliometrics to a system that encourages curiosity driven research, to fit the broader goals of the society.

Panel discussion: Research assessment practices across the globe

The second half of the plenary session was focused on understanding RRA and the impact of DORA in a range of different regional contexts. Andiswa Mfengu shared that changes are happening but currently in a budding stage around her region, and that her University of Cape Town is engaged in fostering conversations and setting up working groups with a view to creating awareness and developing an impact assessment framework for holistic evaluation of researchers. However, commitments and translational changes are still afar, and it is still a challenge to rule out systemic norms like ‘publish and perish’ that govern University rankings. According to Mfengu, initiatives like DORA can serve as an ‘idea of hope’ that can inspire and support researchers are working to champion RRA. Mfengu envisions research assessment to be laying fairer principles in future, by decolonizing knowledge systems in order to meet local needs of the community.

Research assessment reforms vary across different regions of the world due to diverse institutional attitudes, cultural norms and priorities. In Europe, as Karen Stroobants mentioned, organizations are starting to move away from heavy reliance of quantitative metrics like university rankings, the JIF, H-index etc., to more holistic evaluation practices such as use of Narrative CVs and evaluating societal impact.  Along with DORA, initiatives like CoARA are driving increased momentum for assessment reform in (and increasingly beyond) Europe. Some countries, including the Netherlands, the UK, Finland and Norway have already progressed considerably in the implementation of new approaches. When asked what she envisioned for the future of research assessment, Stroobants foresees 1) assessment practices that would balance between achieving organisational strategies and leaving space for individual creativity and 2) organizations and individuals not just to be assessed   based on their track record but also on their potential and competencies to create a future impact.

Judith Sutz discussed that RRA in Latin America has been and continues to develop heterogeneously. There is great appetite for reform supported both by researchers and by international organizations like CLACSO-FOLEC. Public universities are gaining autonomy from government restrictions and are driving research and excellence these days. Sutz mentioned that in Uruguay, DORA has helped set up alternative indicators of research evaluation such as qualitative narration of work progress as opposed to quantitative scales. On being asked her vision on research reforms in next years, Sutz laid out seven points:

1. “Narratives will become the standard way of documenting research performance
2. Researchers should be encouraged to build research problems based on the needs of common people with the help and guidance from non-academics, government experts or civil society. This is important because “unearthing” problems is a complicated task and requires a researcher’s involvement, otherwise suffering of common people will continue to be invisible to research
3. Ethical aspects of research will thoroughly be assessed and researchers would be asked to reflect upon those aspects
4. A sound heterodoxy will be rewarded, the power of mainstream scholars would be diminished, and experimentation (in the sense of following uncertain paths) will be encouraged through an evaluation system that assesses the processes not the outcomes
5. A vibrant international community of scholars through common procedures of RRA will be free to set research agendas that would address fundamental problems of time without any fear of punishment in choosing too difficult problems in small harvest of papers
6. Many more women will be enrolled and enjoy doing research because harmonization between work and life will be at last be possible
7. Everybody will find a place in the wonderful enterprise of research, every talent will be appreciated, and the differences notwithstanding will be appreciated in their due value”

Judith Sutz

The Canadian academic environment too is gaining momentum on holistic assessment. Several national funding organizations are now signatories of DORA and have incorporated DORA’s recommendations into their practices, along with academic institutions, including the University of Calgary. These organizations are recognizing different pathways to impact, as well as understanding what ‘impact’ or ‘high quality’ means in different disciplinary contexts. Stephanie Warner hopes to see a more equitable and transparent research landscape in a decade’s time, with open practices and policies for individuals to be chosen on the basis of their merit by committees well-versed in metrics literacy, and where there will be true respect of different world views, particularly Indigenous and community values.

Conclusion

Conversations surrounding RRA should be a cultural norm within the research community, especially among young researchers that are currently at the forefront of driving culture change tomorrow. There was strong agreement between all the panellists that conversation and awareness alone is not enough; training, capacity building, and understanding research literacy is equally important. At the same time, local advocates of systemic changes should be encouraged to ask questions, carry their values forward, and stand strong even in solidarity. DORA will be working in the coming years to address just such points.

Reading List

Non-DORA resources referenced or shared during the call:

1. Canadian funders and DORA: https://www.nserc-crsng.gc.ca/NSERC-CRSNG/policies-politiques/DORA-DORA_eng.asp
2. University of Calgary and DORA https://research.ucalgary.ca/research/our-impact/DORA
3. University of Cape Town’s statement for the principles of DORA https://uct.ac.za/research-support-hub/outputs/research-assessment
4. Coalition for Advancing Research Assessment https://coara.eu/
5. The Latin American Forum on Research Assessment (FOLEC-CLACSO) https://www.clacso.org/en/folec/

DORA resources:

1. The Declaration on Research Assessment: https://sfdora.org/read/
2. Fritch R, Hatch A, Hazlett H, and Vinkenburg C. (2021). Using Narrative CVs. https://zenodo.org/record/5799414#.YeM-41lOlPY
3. Ginny Barbour and Haley Hazlett (2023). From declaration to global initiative: a decade of DORA. https://sfdora.org/2023/05/18/from-declaration-to-global-initiative-a-decade-of-dora/
4. Anna Hatch, Ginny Barbour and Stephen Curry (2020). The intersections between DORA, open scholarship, and equity. https://sfdora.org/2020/08/18/the-intersections-between-dora-open-scholarship-and-equity/

Queen Saikia is DORA’s Policy Associate