A DORAat10 Local Event Report
In May 2023, DORA celebrated it’s 10th Anniversary with two plenary sessions and a decentralized weeklong program of local events organized by community members from around the world. Event organizers were given the option to write brief reports on their events that summarize key takeaways and recommendations.
By Tilmann Kiessling
A hybrid panel discussion hosted by EMBO and EMBL on 12 May marked the anniversary of DORA, a worldwide initiative aiming to advance approaches to the assessment of scholarly research. The panel discussed issues with current methods of research assessment, as well as solutions and actions for improvement in which panel members have been involved.
Wolfgang Huber, co-Chair of the EMBL responsible research assessment working group, kicked off the discussion. “Research assessment is an integral core aspect of doing science as it helps decide on the recruitment of the next generation of scientists and the allocation of funding,” he said. “Publications are actually not the scholarship itself. They are more of an advertisement of scholarship as the actual scholarship consists of the complete set of reagents or data analysis code, for instance in my field, that generates a paper. We should think about research outputs in a broader way than just the papers,” Huber said. He cited what is known as Goodhardt’s law: Whenever a measure becomes a target, it ceases to be a good measure. The real value would lie in a culture change, Huber said.
Bernd Pulverer, Head of EMBO Scientific Publishing and DORA co-founder, talked about the development DORA underwent from its inception during the 2012 meeting of the American Society for Cell Biology. “At the meeting we quickly converged that the journal impact factor as a single metric is at the heart of some of these issues,” he said. Today, DORA has become a global initiative endorsed by more than 23,000 individuals, institutions, publishers, and funders: “DORA has turned into an advocacy group and is developing tools for more balanced research assessment.” Practices and policies at EMBO include: Applicants for fellowships and grants are not allowed to indicate journal impact factors or other metrics in their applications; instructed reviewers not to use journal impact factors in the evaluation; guidelines for reviewers and applicants are published on the EMBO website; clear-cut conflict of interest policies are applied throughout selection committees; no journal names in files of the candidates shortlisted for the EMBO Gold Medal; reviewed preprints made it realistically possible to assess research outputs at a much earlier stage than journal publications. “And we dropped impact factors for the promotion of the EMBO Press journals,” he said.
Guillermina Lopez-Bendito, from the Institute of Neuroscience in Alicante, Spain, and Chair of the EMBO Young Investigator Committee, emphasized the lack of standardized and comprehensive methods for evaluating quality and impact of scientific work as a major obstacle to advancing research assessment. “Contributions to science go beyond research papers alone. We need to consider mentoring, outreach activities, peer review, and evaluation. We should incorporate assessments of whether researchers have translated their results and discoveries in ways other than publishing,” Lopez-Bendito said. Candidates’ narratives in evaluations are important, as they provide an opportunity for candidates to explain the impact of their research. “DORA is refocusing the attention of reviewers and evaluators on what truly matters, which is the quality of the work.”
Karim Labib, from the University of Dundee and Chair of the EMBO Installation Grant Committee, addressed the assessment of research outside one’s own field. “A key challenge is how best to assess research in areas that one is not extremely familiar with, without relying solely on simple metrics,” he said. Labib supported the idea of interviewing all shortlisted candidates, as it provides equal opportunities for candidates. Labib also emphasized the role of the lead reviewer in interview panels. He advised that lead reviewers should wait until after the interview before sharing their views with the panel, so that the panel members can assess the candidate’s performance in a less biased manner. The panellists agreed with Labib’s view that another bottleneck in research assessment is time. “Scientists are generally interested in participating in research assessment, but lack of time is the primary constraint. “
Brenda Andrews, from the Donnelly Center at the University of Toronto, Canada, and Vice Chair of the EMBL Scientific Advisory Committee, is actively involved in research assessment. As the founding editor of the open-access journal G3 Genes Genomes Genetics, her goal is to publish valuable research findings without considering the impact factor or subjective opinions about the importance of the work. “Senior colleagues bear a significant responsibility in leading by example and changing how we think about research assessment,” Andrews stated. However, she acknowledged that impact factors are still discussed in evaluations. Review committees have become increasingly aware of this issue in recent years. Andrews explained that there is now a clear emphasis on the description of the work and the progress made in setting up labs and training people, rather than solely focusing on the publication venue.
Cecilia Perez, postdoctoral researcher at EMBL, became interested in topics related to social justice in research assessment during her PhD studies. She shared her experiences when applying for scholarships and the biases present in assessment processes. “I would like to highlight the arbitrary nature of assessment processes and emphasize the need for fairness to achieve greater diversity and equality,” Perez expressed. She reflected on the challenges of organizing authors on papers, particularly in collaborative projects that are becoming more common agreeing that we need to move away from simplistic assessments based on authorship order and focus more on author contributions.
The panel discussion on the occasion of the DORA anniversary was co-organized by EMBO (Sandra Bendiscioli, Senior Policy Officer) and EMBL (Katherine Silkaitis, Strategy Officer) and chaired by Sandra Bendiscioli.
This is a cross-post (original post here) that has been lightly edited for length. The quotes from the panel discussion were edited.