The European Molecular Biology Laboratory

The motivation for the DORA working group was a desire to further strengthen EMBL’s commitment to research assessment reform since signing DORA in 2018.¹ The new EMBL initiative aims to articulate and refine its existing research assessment practices for faculty recruitment and research unit review. An example of existing good practice in research assessment at EMBL is the use of external experts on its quadrennial faculty performance review panels, which follow defined criteria and take into account a range of research outputs.

As of 2021, the DORA working group is developing a set of shared values and best practices to help standardize, codify, and refine current research assessment practices for implementation across all EMBL sites. To accomplish this, the working group will:

1. Review EMBL’s assessment practices and propose improvements if and where needed.
2. Ensure that EMBL’s core values of research quality are communicated both internally and externally.

Given that EMBL is an international organization with multiple sites that specialize in a range of life science disciplines, a particular challenge for EMBL is the complexity of creating best practices that are flexible enough to be applicable in different national and disciplinary contexts.

EMBL is addressing these challenges by including a variety of perspectives in the DORA working group, including site heads, senior scientists from across EMBL’s sites, and staff that are central to recruitment and training of PhD students and postdoctoral fellows. The range of inter-site stakeholders involved in the process is important for the creation of a flexible and collectively applied set of assessment policies. Importantly, the DORA working group has identified EMBL's objectives for research assessment reform and is in the process of working to address them.

An additional goal of the DORA working group is to seek more bottom-up insight from its early-career researchers, for example, by working with EMBL postdoctoral representatives to review draft guidelines and share their views on how research should be assessed. 

Both internal and external drivers motivated the creation of a DORA working group. Internally, a major driver for change at EMBL has been the growing recognition among its leadership that to fulfil their mission to train researchers and future research leaders, EMBL needs to place research training more formally in the broader context of good scientific practice, of which DORA recommendations are a part. To fulfill its training mandate, EMBL ensures regular staff turnover by allowing staff to work at EMBL for a maximum of nine years. The regular turnover at EMBL makes it possible for highly trained scientific personnel to transfer the skills they obtain during their time at EMBL to national institutes. These researchers often go on to implement or advocate for similar research assessment reform at their subsequent academic and industrial research positions. 

External influences for change were driven by research assessment reform initiatives implemented by funding agencies, such as Wellcome, which focus on reducing journal-based metrics in assessment. 

In addition to the top-down change initiated by senior leadership and members of the DORA working group, there is robust bottom-up support from faculty members at EMBL sites. Alongside its research and training mission, EMBL provides scientific services and facilities. EMBL has a history of leading open data initiatives, supporting community standards, developing open source software, and data curation. The most recent example of this is the delivery of the COVID-19 Data Portal. These shared institutional experiences, which recognize the value of research outputs other than research articles, demonstrate that the implementation of research assessment reform aligns well with institutional priorities. Moving forward, EMBL hopes to further recruit bottom-up input from research faculty, postdocs, and PhD students by convening workshops to acquire feedback on policy development and implementation.

The DORA working group is aware that the implementation of better research assessment practices will require resources to support the communication of values and offer professional training, and to evaluate the impact of the approach. The working group recognizes the importance of communicating its work to all staff, as well as people considering a career at EMBL. An example of clearly communicating EMBL’s commitment to fair and responsible research assessment will be the inclusion of its DORA commitment on job advertisements. For internal communications EMBL plans to use its intranet, staff newsletter, and the regular email communications from the Director General to share news related to the research assessment reform process. The working group also hopes to develop a set of assets, such slides and information sheets, to support communication efforts. 

The EMBL DORA working group was formed in 2020. Implementation of the practices agreed by the working group is expected to start by the end of 2021. Evaluation of the new policies will be ongoing after implementation in 2021.