The European Molecular Biology Laboratory

Having signed DORA in 2018, EMBL convened a DORA working group in late 2020 that developed policies and guidelines for research assessment of graduate students, postdoctoral fellows, and research faculty. This includes EMBL’s practices for recruitment, performance review, and promotion. The working group also reviewed the roles of EMBL scientists on external grant and recruitment panels.

In 2021, the working group began developing a framework to a set of shared values and best practices to help standardize, codify, and refine research assessment across all EMBL sites. In 2022, they conducted a review of research assessment practice areas within EMBL. They developed four recommendations spanning four areas at EMBL that focussed on a) recruitment of scientists, b) internal research assessment, c) awards and d) culture.

The recommendations were:

1. Clearly state EMBL’s commitment to the principles of DORA and CoARA
2. Update the type of research outputs requested of candidates
3. Provide guidance to internal and external decision-making committees
4. Communicate EMBL’s Research Assessment practices

The new EMBL initiatives aim to articulate and refine its existing research assessment practices for faculty recruitment and research unit review. An example of existing good practice in research assessment at EMBL is the use of external experts on its quadrennial faculty performance review panels, which follow defined criteria and take into account a range of research outputs.

In 2021, the DORA working group developed a set of shared values and best practices to help standardize, codify, and refine their existing research assessment practices for implementation across all EMBL sites. In 2022, EMBL signed the Coalition on Advancing Research Assessment (CoARA).

Given that EMBL is an international organization with multiple sites that specialize in a range of life science disciplines, a particular challenge for EMBL is the complexity of creating best practices that are flexible enough to be applicable in different national and disciplinary contexts.

EMBL has been addressing these challenges by including a variety of perspectives in the working group, including site heads, senior scientists from across EMBL’s sites, and staff that are central to recruitment and training of PhD students and postdoctoral fellows. The range of inter-site stakeholders involved in the process is important for the creation of a flexible and collectively applied set of assessment policies. Importantly, the working group has identified EMBL's objectives for research assessment reform and is in the process of working to address them.

An additional goal of the working group is to seek more bottom-up insight from its early-career researchers, for example, by working with EMBL postdoctoral representatives to review draft guidelines and share their views on how research should be assessed.

Both internal and external drivers motivated the creation of a Responsible Research Assessment working group. Internally, a major driver for change at EMBL has been the growing recognition among its leadership that to fulfil their mission to train researchers and future research leaders, EMBL needs to place research training more formally in the broader context of good scientific practice, of which DORA recommendations are a part. To fulfill its training mandate, EMBL ensures regular staff turnover by allowing staff to work at EMBL for a maximum of nine years. The regular turnover at EMBL makes it possible for highly trained scientific personnel to transfer the skills they obtain during their time at EMBL to national institutes. These researchers often go on to implement or advocate for similar research assessment reform at their subsequent academic and industrial research positions.

External influences for change were driven by research assessment reform initiatives implemented by funding agencies, such as Wellcome Trust, which focus on reducing journal-based metrics in assessment.

In addition to the top-down change initiated by senior leadership and members of the Responsible Research Assessment working group, there is robust bottom-up support from faculty members at EMBL sites. Alongside its research and training mission, EMBL provides scientific services and facilities. EMBL has a history of leading open data initiatives, supporting community standards, developing open source software, and data curation. The most recent example of this is the delivery of the COVID-19 Data Portal. These shared institutional experiences, which recognize the value of research outputs other than research articles, demonstrate that the implementation of research assessment reform aligns well with institutional priorities. Moving forward, EMBL hopes to further recruit bottom-up input from research faculty, postdocs, and PhD students by convening workshops to acquire feedback on policy development and implementation.

The working group is aware that the implementation of better research assessment practices will require resources to support the communication of values and offer professional training, and to evaluate the impact of the approach. The working group recognizes the importance of communicating its work to all staff, as well as people considering a career at EMBL. An example of clearly communicating EMBL’s commitment to fair and responsible research assessment will be the inclusion of its DORA and COARA commitments on job advertisements. For internal communications, EMBL is using its intranet, staff newsletter, and the regular email communications from the Director General to share news related to the research assessment reform process. The working group has also developed a set of assets, such slides and information sheets, to support communication efforts.

The working group created an implementation plan with specific actions to enact each recommendation. Working group members were allocated responsibility for completing actions based on their job function, and progress was discussed at each working group meeting.

To implement Recommendation 1: Clearly state EMBL’s commitment to the principles of DORA and CoARA, a commitment statement has been included in the following areas for job ads and on relevant HR webpages; applications for doctoral and post-doctoral programmes; in award nominations (where appropriate) and on the EMBL Research Assessment internal and external websites.

The Commitment Statement states: “EMBL supports fair and responsible research assessment, which includes its recruitment and performance assessment processes. We recognise a range of research outputs, discourage inappropriate use of proxies such as journal impact factors, and value research outputs based on their intrinsic merit. EMBL is a signatory of the San Francisco Declaration on Research Assessment (DORA) and the Coalition on Advancing Research Assessment (CoARA)”

To implement Recommendation 2: Update the type of research outputs requested of candidates, research output instructions were updated for any situation requiring a CV, particularly in staff recruitment and faculty search reports, and promotion documents; in applications for doctoral and post-doctoral programmes; in award nominations and in Unit Review documents.

The CV guidelines ask for a short narrative summarizing the impact of qualitative output such as contributions to new ideas, key skills, influence on policy and practice, instead of highlighting quantitative measures such as impact factors, h-indices, and the like.

To implement Recommendation 3: Provide guidance to internal and external decision-making committees, guidance to internal and external review committees that made decisions about funding, hiring, tenure, or promotion were updated, as well as associated templates and reports. This specifically included:

• Updates to the EMBL Recruitment and Selection Guidelines for use by hiring managers and/or chairs of evaluation panels for staff recruitment and faculty report template
• Updates to the instructions on the e-recruitment platform for evaluators and issued guides for doctoral programme and postdoctoral programme reviewers and associated report templates
• Updates to the guidance and award selection committee reports
• Updated guidance for committees evaluating promotions
• Updated guidance in the Unit Review Procedures, and panel report template for Unit Review

To implement Recommendation 4: Communication of EMBL’s research assessment practices, an internal communications plan was developed. Key steps included developing:

• Communications to all staff and discussions at the 2022 Faculty Retreat
• Slide decks to be used by senior staff in internal presentations
• Communication guidance for senior staff to be used when presenting EMBL research to external audiences
• Articles published on EMBL’s website and internal newsletters
• Research Assessment Reform champions program
• Training materials for inclusion in courses for pre-docs, staff, and new group and team leaders

The EMBL working group was formed in 2020. Implementation of the practices agreed by the working group is expected to start by the end of 2021. Evaluation of the new policies have been reviewed and implemented since 2021.

EMBL regularly seeks input and guidance from its community on its actions and endeavors to incorporate feedback and constantly enhance its research assessment culture. It does this through focus groups and solicited feedback, and it is currently building a Responsible Research Assessment champions program to formalize a continuous feedback framework.

Monitoring efficacy of these actions is an important component of implementation and work is ongoing to devise appropriate measures and KPIs.