A DORAat10 Local Event Report
In May 2023, DORA celebrated it’s 10th Anniversary with two plenary sessions and a decentralized weeklong program of local events organized by community members from around the world. Event organizers were given the option to write brief reports on their events that summarize key takeaways and recommendations.
By Rebecca Hill, Simon Hettrick and David Moher
How do we assess research today and what has changed since the DORA declaration? And what can open research practices contribute to future research assessment systems? To explore these questions in more depth and to celebrate the 10th anniversaries of both DORA and the open research publisher F1000, F1000 hosted a webinar in May 2023, asking ‘What can open research values bring to research assessment reform?’
The panel discussion included David Moher (University of Ottawa/Hong Kong Principles); Simon Hettrick (Software Sustainability Institute/the Hidden Ref/University of Southampton); and Becky Hill (F1000).
What has changed since the DORA declaration?
David: For most researchers, career advancement is based on the currency of the day. Evidence indicates the global currency is usually the number of publications generated in a finite period. This reward scheme typically benefits certain disciplines and ‘public facing’ researchers. It excludes the many talented personnel working to bring the research to fruition. Along with the number of publications, assessors are often interested in the impact factor of the journal the research report is published in, as well as the total dollar figure associated with any awards. Clearly this tends to favor expensive research, often conducted in biomedicine. This reward ecosystem has not materially changed in 50 years.
Re-imagining incentives focused on more inclusive, transparent, and open ways of working are important. The 2020 Hong Kong principles are one such effort combining many aspects of community and research integrity. The principles intend to guide organisations’ research assessment, promotion and tenure practices to focus on incentivizing and rewarding researchers who incorporate open scholarship practices into their research. This aligns with the 2022 US Office of Science and Technology Policy recommendation on data sharing and public access to research. Implementing these frameworks as part of a researcher’s assessment make sense and is a movement towards using evidence as part of the assessment process.
Alongside what DORA is doing, it will be important to watch the development of the Coalition for Advancing Research Assessment (CoARA), an initiative committed to reimaging research evaluation and how we incentivize and reward researchers for the future.
Becky: The last five years have seen the introduction of major policy imperatives aimed at improving how we do and deliver research. These are often underpinned by a drive to more open and collaborative ways of working. The publication of the 2021 UNESCO recommendation on open science was a major landmark for attitudes to research practices and behaviors, endorsed by 193 nations. It highlights ‘Open Scientific Knowledge’ as integral to an effective research system –publishers can play a pivotal role in realising this.
What can publishers do to support responsible research assessment?
Becky: As David notes, the focus on publications as the currency for researchers to demonstrate their value in research assessment systems is widely acknowledged as problematic. DORA sets out clear guidance for publishers to help improve how research is evaluated, which include practicing more responsible use of research metrics, but also highlights the integral role that publishers can play in bringing greater transparency to the research dissemination process.
Publishers are enablers of open scientific knowledge by a combination of services they can provide, be that providing open access to research outputs, facilitating data sharing, creating routes for the publication of a diversity of research outputs, and ensuring research integrity. Publishers can also provide greater visibility of the myriad of specific and valuable contributions to research output through, for example, adoption of the Contributor Role Taxonomy (CRediT), or through providing more transparent and open peer review, bringing the often unseen role of reviewers to the fore.
What are great examples of innovation to help shape research assessment reform?
Simon: Research relies on the efforts of many non-traditional academic roles, such as technicians, data stewards, and research software engineers and the value of such roles is largely ‘hidden’ in traditional research assessment systems. The UK’s Research Excellence Framework (REF) is the national assessment framework for universities to present the impact of their research, the outcome of which determines the distribution of billions of pounds of funding. Despite a broad framework encouraging universities to present the impact of research across everything from musical compositions to software, universities place almost all their focus on published outputs. In the last REF, only 2.4% of outputs presented for assessment were not related to publications.
A focus on publications is problematic because research articles alone rarely describe all of the techniques, methods and technologies involved, nor are all the people who contributed to the research named in publications. The Hidden REF, a community-led initiative, was founded on the principle that if we do not recognise and provide incentives for everyone involved in the conduct of research, we will limit our ability to conduct research now and in the future. The Hidden REF started with a UK national competition in 2021, which recognised the vital work of a number of “Hidden Roles” without which, much research would be impossible. Partially due to lobbying by the Hidden REF, the next UK research assessment exercise will now recognise all roles – not just traditional academic ones.
What needs to happen now?
To drive reform across the research process, in both research culture and adoption of open practices we need to incentivise, recognise, and reward the behaviors we want to see – whether that’s sharing data openly or celebrating ‘hidden roles’ in research.
Each stakeholder has a role to play to drive change and adoption, but collaboration and partnerships across the research system are essential to deliver the promise of open research and reform in research evaluation. The question remains as to who needs to drive forward this change and how to ensure all parts of the system are in alignment – but DORA has certainly laid important foundations.
Want to learn more about what open research practices can contribute to research assessment reform? Watch the recording of the webinar.
