National Research Institute for Agriculture, Food and the Environment (INRAE)

The first major reorientation of INRAE’s faculty assessment criteria occurred when France decided to create a national division for research organization assessments in 2006, then called AERES and now known as HCERES. The second major reorientation began in 2020 (even if it was initiated before) when INRAE shifted away from reliance on quantitative assessment practices and adopted a number of holistic approaches to promote, recognize, and reward Open Science practices among its researchers. To this end, the Institute has developed two main principles for assessment: consideration of multiple criteria and qualitative assessment.

The “multicriteria” principle covers a wide spectrum of factors beyond traditional outcomes like publications. For example, researchers are assessed on production of knowledge, expertise, outreach, training, and management. Qualitative assessment, the second principle, complements the multicriteria approach by asking researchers at INRAE to provide important context beyond what is generally included in traditional academic assessments. Researchers are evaluated by a panel of their peers based on self-assessment reports written in a narrative story-telling mode. These assessments are viewed as an opportunity to support the scientific process and participate in shared learning. For example, researchers at INRAE are encouraged to include a more realistic picture of their research process that includes positive and negative data and how challenging experimental problems were resolved. Researchers are encouraged to mention three realizations or lessons learned in the entire process.

Additionally, qualitative assessments take into account Open Science practices: Researchers are asked to highlight the open accessibility of their publications, data, code, and software. To help researchers make this transition, INRAE published a guide for “Best practices for achieving 100% open-access publication by 2030” in 2023.

The process of research evaluation at INRAE is directed by French law. Researchers in permanent positions are mandated to undergo an assessment every two years. This process allows their peers to evaluate their work, with the dual aim of benefiting both the scientist being assessed and the research organization.

The shift to qualitative assessment of scientists at INRAE was inspired by the work of psychologist Christophe Dejours, which emphasizes the importance of evaluating the work process rather than the outcomes. A narrative or story-telling mode can help peer assessors better understand the work process and support a more holistic assessment of their fellow scientists. At INRAE, both qualitative and quantitative approaches are considered important and complementary. While quantitative metrics help in assessing outcomes, qualitative approaches contribute to the evaluation of the work processes.

Under the influence of organizations like DORA and CoARA, INRAE has eliminated the excessive use of quantitative metrics  (such as the impact factor or the H-index) for research assessment. Researchers are asked to provide a comprehensive list of outputs beyond publication, such as projects that they contributed to, collaborations, and teaching. This is meant to help assessors more holistically evaluate the scholarly activities of researchers.  Importantly, the uniqueness of every researcher’s career journey is valued and the ‘outputs’ lists are not used to compare one researcher with another. Finally, committing to the Paris Call on Research Assessment has been a major driver to recognizing Open Science practices undertaken by researchers at INRAE.

The drive for change at INRAE combines both top-down and bottom-up approaches. Because INRAE is a nationally funded research institute, change is ultimately decided by governmental policy, followed by the Director of INRAE. To help shape the form and function of policy, the Director works with intermediate leadership, which consists of a team of INRAE fellows involved in the research assessment process. Based on their experience with practical assessment implementation at INRAE, the intermediate leadership team is situated to propose realistic changes to policy and support implementation.

Concretely, INRAE started to integrate qualitative assessment and Open Science practices for research evaluation in 2020. This effort was first initiated via internal discussions with heads of the fourteen INRAE scientific divisions and the presidents of the twelve INRAE Specialized Scientific Commissions of peers (i.e., INRAE evaluation structures). French experts in the field of evaluation and Open Science were also invited to provide their insights.

Consequently, new frameworks were designed for the reports used for research evaluation, and guidelines were updated for qualitative assessment involving peers. Webinars were organized to share the developments with INRAE researchers and allow for an exchange of their thoughts and knowledge on the new practices. In parallel, these developments were discussed by the national and European working groups on Open Science and research evaluation.

INRAE became a DORA signatory in 2018. In 2020, the Institute chose to continue and develop tailored criteria for labs based on finalized objectives, leading to the introduction of a multicriteria approach for assessing scientists. One of the long-term missions of the assessment team at INRAE is to follow the trends the way science evolves.

In 2022 INRAE signed the Paris Call on Research Assessment and reaffirmed its commitment to support Open Science practices among its researchers in its 2022 action plan. At present, INRAE is committed to adapt assessment procedures to better recognize impact and scientific integrity. It is also working to fulfill its commitment after joining the European commission “Agreement on performing research assessment” (2022).