The DORA team is excited to have celebrated its 12th anniversary with the launch of a Practical Guide, to help research performing organizations (RPOs) develop responsible research assessment (RRA). This Guide is designed to support RPOs aiming to build capacity and support the implementation of responsible research assessment (RRA) practices. The Guide reflects a collective effort and draws upon learnings from the DORA community. To celebrate this milestone and gather global perspectives, we recently hosted three virtual launch events across the Asia-Pacific, Africa & Europe, and Americas regions.
Global Launch Events: Regional Perspectives on Implementing RRA
To introduce the Guide and foster discussion on implementing RRA in diverse contexts, DORA hosted three virtual launch events. These sessions featured experts and advocates from various regions, who shared their insights, challenges, and hopes for RRA. Across the three launch events, several common themes emerged: the critical importance of community building and dialogue, the necessity of leadership support and buy-in, the value of case studies and practical examples, the need to address resistance and fear associated with change, and the understanding that implementing RRA is a cultural shift requiring sustained effort and collaboration amongst various existing initiatives and diverse interest holders. The conversations reinforced that while achieving system-wide change is a significant undertaking requiring sustained effort and collaboration, it is essential for fostering more diverse, inclusive, and impactful research systems. Read on or watch the event recordings to explore how RRA is being addressed globally and how the Guide can help your actions locally.
Asia-Pacific
The Asia-Pacific event on May 15, 2025, was the first of three global launches for DORA’s new practical guide. Hosted by Ginny Barbour (DORA Co-Chair, QUT), this session featured panelists Moumita Koley (Indian Institute of Science, DORA Steering Committee, India), Yu Sasaki (Kyoto University, Japan), Fang Xu (Chinese Academy of Sciences, DORA Steering Committee, China). Discussions highlighted the current challenges in the region and the potential utility of the Guide.
Moumita Koley highlighted the pitfalls of the current system in India, which is heavily reliant on evaluative bibliometrics and journal publication as the sole measure of performance. Koley pointed to the National Institutional Ranking Framework (NIRF) as an example, noting that many institutions (excluding some established “top ones”) intensely focus on increasing publication numbers and engage in practices like citation manipulation to improve their rankings. Peer review has become largely driven by these quantitative metrics and incentives for publishing in high-impact journals have compounded these issues. The continued popularity of indicators like the H-index and inclusion on “top researcher” lists was also raised as inhibiting RRA initiatives and practices..
Koley shared how the Guide’s recommendations and discussions highlight the necessity of a different evaluation system that values diverse contributions, helping to reorient the purpose of research away from simply increasing quantitative outputs and allowing for institutional leaders to benchmark their progress against other countries that are adopting RRA practices. While acknowledging the difficulty in finding alternatives to simple numbers and that solutions won’t be universally applicable, the Guide’s strength lies in its ability to bring the community and researchers together for discussions to find solutions suitable for their specific ecosystem. She expressed hope that it will help India move beyond the “publish or perish” culture, stressing that this requires cultural change, not just top-down directives.
“The Guide will be helpful in creating that space, creating that narrative so that we will be able to take the Guide and make it for our own ecosystem.” – Moumita Koley
Yu Sasaki recognized similar issues, including the lack of awareness about RRA across academia in Japan. While the broader academic community may not be fully engaged, she noted that research administrators like herself are keenly following the debate. She explained that this focus among administrators isn’t primarily due to a similar ranking system as seen in India or elsewhere, rather, it is driven by the significant emphasis currently placed on evidence-based decision making within higher education policy debates in Japan which is over-reliant on problematic metrics/indicators. She stressed the need for credible evidence and case studies to argue against quantitative metric-only evaluations and underlined the importance of creating and joining communities of like-minded people to share concerns and strengthen capacity.
For Sasaki, a key strategy for change is to create or join a community of like-minded people and openly share your concerns. She noted that speaking with others, even online, helps individuals realize that their local challenges are often not significantly different from cases described globally, such as those in the guide. She stressed that sharing an understanding of what a good research assessment model should look like versus the current reality, engaging in discussions, and potentially organizing talks or seminars can help strengthen both capacity and understanding of these issues.
“What we need is many case studies like you have in this Guide and also professionally analyzed and with credible evidence. Those professional kinds of compilation of case studies and analysis will help us university research administrators to equip ourselves to argue against such moves of wrongly looking only at the metric based approach.” – Yu Sasaki
Fang Xu shared that China began their latest reform a decade ago to move beyond quantitative indicators towards research quality and positive research culture. She highlighted three key takeaways from the guide relevant to China: the importance of gathering consensus from different interest holders, implementing change through pilot studies before wider rollout, and the need for continuous study on RRA theory and practice, for which the guide serves as a useful library of examples. She also provided examples of RRA in practice in China, including peer review-based hiring at a top mathematics institute and training peer reviewers at the national funding agency. In her first example, Xu explained how the Chinese Academy of Sciences’ Institute of Mathematics uses a peer review-based approach for hiring and promotion, rather than focusing on papers, journals, or citation metrics. They employ a two-step process: candidates’ materials are first sent to international peers for review, who compare the candidate to their own colleagues at similar career stages, followed by an internal expert review. The second example highlighted how the National Natural Science Foundation of China (NSFC) is actively training peer reviewers, including young researchers, on what constitutes a good peer review using examples of both good and bad practices.
Since China is currently midway through its reforms, Xu feels she can use the Guide from this perspective to monitor progress, review the process, summarize findings, and manage the ongoing changes. This information can then inform her advisory reports to the central government, detailing the current status, progress made, remaining challenges, and future plans. She concluded that the Guide is beneficial whether an institution is just starting their RRA journeyor is already in the middle of the process.
“I would say this Guide is quite useful and with all this rich information. So whether you are a research manager or a researcher in research evaluation or whether you are just generally interested, I think this is quite helpful for you too.” – Fang Xu
Along with the Guide, we celebrated the launch of the University of Tokyo case study during the panel. Yuko Shinzawa, from UTokyo, briefly joined the panel and shared some of her perspectives. For the University of Tokyo, deciding whether or not to sign DORA was a small first step, but a very concrete one, which allowed their discussion to progress. Having signed the declaration, they now feel they are at a starting point again on their RRA journey. She sees the guide as a kind of map or source of options from where to start next steps, drawing on the paths other institutions have taken. Shinzawa believes the specific examples in Chapter 3 (Key Moments in Research and Researcher Assessment) will be particularly useful for the University of Tokyo. As a concrete next step, they might use this chapter to check the current state of their process for selecting internal awards or research grants.
“I’m deeply impressed by the Guide because it is very comprehensive and very well structured. I particularly like the ‘Things to Think About’ section on the right side of each activity because it provides practical examples of what to consider when taking on specific responsibilities when thinking about signing DORA.” – Yuko Shinzawa
Africa & Europe
On May 16, the second of three global launch events for DORA’s new resource took place, focusing on the African and European regions. Hosted by Rebecca Lawrence (DORA Vice-Chair, F1000), the online meeting brought together distinguished speakers Emmanuel Boakye (African Reproducibility Network AREN), Henk Kummeling (Utrecht University, Netherlands, Chair of CoARA); and Andiswa Mfengu (University of Cape Town, South Africa). Panelists shared valuable insights from their regional perspectives.
Andiswa Mfengu highlighted challenges in the South African context, including a lack of buy-in and urgency within academia and leadership, particularly from those who have benefited from traditional quantitative metrics. Mfengu stressed that moving towards RRA in Africa is complicated by the systemic ties that institutions have to traditional evaluation methods, noting that while individual universities might wish to change, they are often caught within a system where funding and appointments are still heavily influenced by quantitative metrics. Thus creating a misalignment between theoretical support for RRA principles and institutional values, and the practical realities of assessment processes.
Mfengu also discussed strategies for advancing reform, emphasizing the critical role of champions within institutions, particularly those in senior positions who can influence resource allocation and set direction. She suggested that a combination of top-down support from leadership (like directors of research who allocate funds) and bottom-up advocacy from professors influencing their disciplines and students is most effective in driving change. She underlined the importance of communities of practice for advocates to connect, share experiences, and gain support, especially given that change can be uncomfortable. Mfengu concluded that Africa has a long journey ahead in learning from others and needs to advocate for governments to prioritize funding in ways that truly reflect the value of research contributions to society.
“As the continent, Africa, we havea long way to go in terms of assessment reform and we need communities of practice and support from government for this progress to be realised, as funding is key in research assessment. However, while funding is key, the issue is also about what we prioritise in the regions and we need to prioritise and make resources available for this reform to ensure our research positively contributes to society.” – Andiswa Mfengu
Henk Kummeling drew parallels between the challenges in South Africa and the Netherlands. He agreed that there is often a lack of urgency or recognition of the need for reform, particularly from those who have benefited from traditional quantitative metrics. Kummeling emphasized that moving towards RRA is fundamentally about reflecting on the core purpose of research institutions and what they aim to achieve for society. He noted that at Utrecht University and more broadly in the Netherlands, they concluded that the old metric-based system was simply not effective in helping them achieve these goals, necessitating a shift towards other ways of assessing quality and impact. He described this reform as a major cultural shift in academia and research that requires considerable time and effort. Crucially, he highlighted that gaining buy-in from leadership at institutions is essential for change. In the Netherlands, the collective agreement among university rectors and research leaders on the need to change the model allowed them to collaboratively advocate to political leaders regarding necessary changes in funding and budgeting systems, a process that took years but ultimately proved successful.
Commenting on challenges such as the use of rankings in response to a question from the audience, Kummeling agreed that they rely heavily on quantitative metrics and benefit institutions at the top, creating resistance to change. To overcome this, he suggested that institutions facing this issue should form a collective voice, partner with organizations like DORA and CoARA, and work together to expose the deficiencies of the current system. He reiterated that achieving this collective system change requires collaboration, support for those leading initiatives, and dedication of resources and time.
“I think [the Guide] is a great ‘starter kit’ for a lot of institutions. At CoARA we are now working with working groups and national chapters on parts of the implementation of what’s already there in the Practical Guide of DORA but it takes a lot of time and effort and I think working together and creating a better understanding of what are the needs and what are the constraints in several systems would be very helpful.” – Henk Kummeling
Emmanuel Boakye highlighted the significant challenges and resource needs for implementing RRA across the African region. He described the reform as a daunting task that requires buy-in from both the leadership and researchers within an organization. A major hurdle he identified is the mobilization of resources, including finding passionate people with dedicated time, securing remuneration for this work, and identifying funding sources. He also stressed the importance of gaining buy-in from major interest holders across the continent or in specific countries.
Regarding advocacy, Boakye emphasized that tackling RRA as a “single champion” within an institution can feel overwhelming. He suggested starting advocacy efforts by gaining buy-in from peers to form a group of people willing to work to advance RRA. Boakye noted that AREN’s training program helps build this supportive community. He also touched on the status of funders in Africa, noting that while they may support RRA in theory, the practice, such as funding applications and appointments, often still relies heavily on quantitative metrics, indicating a fundamental misalignment.
“Start from your peers and get them involved. That way, if you get institutional buy-in, you already have a group of people that are willing to work on this. ” – Emmanuel Boakye
DORA Steering Committee member Sean Sapcariu and Suz Garrard also contributed to the panel discussion, offering perspectives from their roles within funding agencies, the Luxembourg National Research Fund (FNR) and Research Ireland, respectively. They joined the conversation from the audience when invited to comment on the role of funders in advancing RRA following a lively debate in the chat of the session. They reinforced that both their institutions are highly committed to DORA and CoARA, and emphasised how implementing change requires bringing the research community along through conversation, collaboration, and extensive community engagement.
Americas
The Americas launch event marked the third and final event for our 12th anniversary and launch of DORA’s “A Practical Guide to Implementing Responsible Research Assessment at Research Performing Organizations”. Hosted by Kelly Cobey (DORA Co-Chair, University of Ottawa, Canada) the session featured a panel of experts from across the region: Christian Gonzalez-Billaut (Universidad de Chile, DORA Steering Committee, Chile), Judith Sutz (Universidad de la República, DORA Steering Committee, Uruguay), and Stephanie Warner (University of Calgary, Canada). As with previous events, the panelists discussed the challenges and opportunities for implementing RRA in their respective countries, often highlighting themes that resonate globally despite regional differences.
Judith Sutz spoke to the unique context of Latin America, where she feels public universities with co-governance, autonomy, and a mission including societal extension are different from European and North American universities. She highlighted that while there is recognition among researchers that the current evaluation system encourages poor academic practices and hinders engagement with interest holders beyond academia, and contributes significantly to workplace stress, there is significant resistance to change from the academic elite. This reluctance often stems from a fear that moving away from quantitative metrics will lessen research quality. Sutz emphasized that simply denouncing the system isn’t enough; the key is to transform it. Her personal advocacy for RRA stems from seeing how the current system negatively impacts researchers’ passion and works against doing good science. She stressed the urgent need to resolve the conflict where national assessment systems’ criteria often override institutional duties, creating tensions for researchers.
Sutz highlighted the Guide’s value in showing how premier institutions are adopting different approaches, helping to dismiss the myth that RRA reduces research quality by providing excellent arguments and examples to convince diverse actors that RRA is necessary, valuable, and feasible. Uruguay is currently working on inter-institutional guidelines for RRA, and Sutz noted the value of the Guide’s flexibility argument (“one size does not fit all”) for allowing institutional specificities while ensuring common practices.
“When I read the Guide I had the feeling that the Guide speaks to me, speaks to us Latin Americans as if those that made the Guide were aware of our most inner difficulties.” – Judith Sutz
Christian Gonzalez-Billaut discussed the complex challenge of RRA in Chile and globally, emphasizing the need for a collective effort involving universities, funding agencies, researchers, publishers, and the private sector. Gonzalez-Billaut stressed the crucial need to engage senior and established researchers, acknowledging their potential reluctance to change, and suggested that university leaders like vice provosts are a good starting point for cultural change. He underscored that funding agencies are absolutely essential to this reform and must redefine their assessment practices for allocating competitive resources, recommending an escalated mechanism for implementing changes, moving from small to large adjustments continuously. Critically, he stated that funding agencies need to develop the capacity to evaluate the outcome of each change and share the results with the community, specifying clear evaluation criteria and indicators for longitudinal follow-up.
Gonzalez-Billaut also emphasized the need to “lead by example” within institutions and on funding panels, urging academics to move away from bibliometric shortcuts towards assessing significance, originality, rigor, and societal impact, including through narrative CVs. He stressed the critical need to evaluate the effectiveness of any changes made and advocated for piloting initiatives, revising policies (like hiring and promotion), and sharing successful stories. Furthermore, Gonzalez-Billaut emphasized the need to educate on RRA by incorporating training into PhD programs, and to share successful stories to support change and highlight positive examples.
“We need to share successful stories. Everybody is waiting to highlight individuals or institutions or departments that have implemented, successfully, responsible research assessment. So we need to show how this effort gives some outputs and the Guide is very helpful for that.” – Christian Gonzalez-Billaut
Stephanie Warner discussed the Canadian context, highlighting a “groundswell” of support and the value of connecting with people and having open, collaborative discussions within emerging communities of practice. She suggested leveraging existing institutional efforts around equity, diversity, inclusion, and decolonization, as well as fostering dialogue across different sectors (academia, government, funders, etc.) to expose the limitations of traditional assessment practices. Warner also shared insights from the University of Calgary’s experience, noting the importance of mapping existing practices and recognizing that implementing RRA is a non-linear, consultative process.
Along with the Guide, we also celebrated the launch of University of Calgary’s case study, in which Stephanie helped detail their approach to implementing RRA, emphasizing it was a partnership between the Provost’s office and the Vice President Research portfolio. Key successes included securing institutional buy-in from leadership to sign the DORA declaration, which helped coalesce efforts. A significant practical step was integrating DORA language into the academic staff criteria and processes handbook, a key policy document governing hiring, tenure, and promotion, to emphasize valuing diverse outputs and assessing research on its own merits. She noted the value of linking RRA efforts to existing equitable and inclusive hiring training.
“I think one of the most practical and easiest ways to start with advocacy is to connect with people and to have conversations. There are organizations from coast to coast to coast that are having these discussions about what matters in scholarship, what matters in our society, and aligning what we’re doing in research assessment with what we say we value institutionally or nationally policy-wise.” – Stephanie Warner
Leslie Chan (University of Toronto, DORA Steering Committee) also contributed to the conversation, echoing the importance of training young scholars and sharing his positive personal experience with narrative CVs. He shared a concrete example from his experience at the University of Toronto, where he was involved in a program called the Public Impact Fellowship for PhD students. This program encourages students to reflect on different types of scholarship and the pathways they can take. Within this program, he introduced students to different ways of sharing research impact, including the narrative CV. Chan noted that many of these students were previously unaware of such possibilities, having been trained in traditional methods of defining success and excellence. He described exposing them to these alternative pathways as liberating, particularly for students engaged in work with communities internationally. He expressed hope that this program would continue to be funded and expand to other disciplines and faculties. Regarding narrative CVs specifically, Chan mentioned that the tri-agency in Canada is moving in this direction and shared a link to their template. He also shared his personal, recent experience transitioning from a traditional CV to a narrative one:
“I can tell you it makes me happy to say: looking back actually I have done things that are not just ‘how many pages are being published, what journals and what awards did I get or not get’ and that kind of thing so I echo that these kind of practices are actually liberating, creative and happy so this is all good particularly for young scholars.” – Leslie Chan
Resources shared during the events
On university rankings: https://www.universityworldnews.com/post.php?story=20240423081048420
On narrative CVs in Canada: https://www.sshrc-crsh.gc.ca/funding-financement/apply-demande/tri-agency_cv-cv_des_trois_organismes-eng.aspx
On training for PhDs: Connaught PhDs for Public Impact Fellowship Program: https://www.cgpd.utoronto.ca/public-scholarship/connaught/
More about the Guide
The Practical Guide has been created as part of Project TARA and which, alongside Reformscape, the Building Blocks for Impact, and the Debiasing Committee Composition, forms a suite of tools designed to help organizations who are seeking to refocus on research quality and impact in their assessment practices.
The touch points and influences on research assessment policies and practices across most research performing organizations are highly diverse. Our target audience is, therefore, intentionally broad and inclusive: those in research leadership roles, faculty, administrative managers and staff; those responsible for shaping organizational policies and recruitment practices; research communication; library and outreach staff; researchers at all career stages; as well as individuals and teams who may be shaping research culture-related initiatives.
We know that many organizations are keen to reform their research assessment practices but lack the time, space and/or expertise to reflect on how to do this and often don’t know where to start. The Guide presents practical advice and tips – such as engaging the organization’s leadership, creating a working group, and developing a communication and engagement plan – that we know from our work with the DORA community over the last decade can make a real difference in delivering change. To inspire action, the Guide is threaded with examples from organizations across the world who have developed and adopted RRA approaches – including three new case studies from universities in Canada, Denmark and Japan.
Of course, each organization is different, and one size does not fit all in the practice of research assessment. To this end the Guide is not intended to be prescriptive; we encourage organizations to use the Guide flexibly – use it in its entirety or draw upon pieces that are most relevant.
The Guide reflects the hard work of many contributors. We particularly want to thank the participants of a January 2025 working meeting in Washington, DC, USA who provided valuable reflections that have helped to shape this Guide. We would also like to thank the Arcadia Foundation who provided the funding for Project TARA. Arcadia helps people to record cultural heritage, to conserve and restore nature, and to promote open access to knowledge. Since 2002 Arcadia has awarded more than $1.2 billion to organizations around the world.
Please reach out to us (info@sfdora.org) if we can answer any questions or to share how you used this resource. DORA is eager to support the use of these resources and to hear how they are being used.