Each quarter, DORA holds a Community of Practice (CoP) meeting for National and International Initiatives working to address responsible research assessment reform. This CoP is a space for initiatives to learn from each other, make connections with like-minded organizations, and collaborate on projects or topics of common interest. Meeting agendas are shaped by participants. If you lead an initiative, coalition, or organization working to improve research assessment and are interested in joining the group, please find more information here.
The scope for change in evaluation systems in Brazil
In order to make science work for national betterment, a country’s research institutions need to align themselves with societal impact, environmental responsibility, open science, and responsible research practices. Often, these are the values that academic institutions espouse. However, a narrow overreliance on inappropriate metrics as proxy measures of quality can fail to recognize and reward open practices, rigor, reproducibility, locally relevant research, and more. Towards the goal of creating and supporting more responsible evaluation systems in Brazil, the Sao Paulo State Research Foundation (FAPESP) founded and funded Projeto Métricas, a multi-institutional project involving the six major public universities in Sao Paulo. The aim of Projeto Métricas is to boost awareness about the appropriate use of bibliometrics and improve university governance and science communication. The project offers a number of awareness programs, workshops, and courses for professionals and researchers all across Brazil.
On November 8, 2022, Justin Axel-Berg and Jacques Marcovitch presented the work of Projeto Métricas at DORA’s National and International Initiatives for Research Assessment Community of Practice meeting. On the call, Justin Axel-Berg highlighted that although a few departments and institutions are improving their assessment practices responsibly, a majority of evaluation methods in Brazil are still highly orthodox. Broadly, they are quantitative and restrictive, both at the level of the Council (i.e., the national governing body of research and research funding agencies) for institutional-level evaluation and/or by the institutions in their hiring and promotion policies. In light of this, Projeto Métricas is currently working on monitoring and identifying the gaps in research assessment practices at public universities, and also mapping the impacts of ongoing reform efforts. As part of their project for the DORA Community Engagement Grant program, Projeto Métricas held a workshop and conducted a survey to help the development of a roadmap for supporting the implementation of DORA’s responsible research assessment principles at Brazilian institutions. Axel-Berg pointed out during the discussion that, although the number of individual and organizational DORA signatories in Brazil are comparable to those in Europe and the UK, only two research-focused Brazilian organizations have signed so far: The University of São Paulo (USP) and The State University of Campinas (UNICAMP). In an attempt to gain a broader understanding of the obstacles in committing to responsible research evaluation practices and to identify areas for intervention, Projeto Métricas took a 3-step approach. The details of their work can be found online.
Course of action
In their first DORA Community Engagement Workshop, they held an exploratory quantitative and qualitative survey based on the five pillars of the SCOPE rubric. Respondents included individual DORA signers from USP and UNICAMP and were from a diverse range of academic career stages. Because of this, Projeto Métricas was able to gain a holistic understanding of the reasons behind the challenges to implementing responsible evaluation practices. Survey results suggested that the major barriers were: i) institutional resistance to change, ii) lack of training of evaluators, iii) dearth of society engagement and consultations for evaluations, and iv) senior colleagues being unaware of the advancements for responsible research. The workshop was then followed by a public event that was open to the wider Brazilian research community. During this event, the key points from the preliminary survey were discussed to identify the main action areas and the possible ways to resolve the issues. Lastly, a consolidated report, with the key findings from the workshop, the survey, and the public event, was evaluated by field experts within Brazil who helped to narrow down the results to three concrete target areas of action:
- Awareness of responsible evaluation practices: The community needs to be made more aware of responsible evaluation practices and initiatives (e.g., DORA, Leiden, etc.) that are working towards reform. Informing and empowering the community through discussions about impact-driven and responsible assessment would be the first step to understanding, interpreting, and reflecting on what performance indicators are useful and what indicators are being misused in Brazil. Reforming the traditional, fixed, and limited evaluation models that are currently in use will require a collective broadening of mindsets. Therefore, awareness programs should actively engage community members at various career stages. It is also important for universities and research institutes to incorporate community ideas in developing frameworks that are suitable for a wide variety of evaluations, including career advancement, recruitment, grants, etc. while aligning with personal, institutional, local, and national values.
- Training and capacity building of evaluators and participants in processes: Workshops, training programs, debates and discussions to be conducted regarding meaningful and conscious evaluation tools for evaluators to those being evaluated.
- Planning and execution of new evaluation practices: The evaluation methods must also be flexible and accommodating to dynamic evolutions in the academic system. Developing a well-structured plan of action, for short-, medium- and long-term goals, with consistent monitoring, is essential to the successful implementation of newer practices. This would enable universities and institutions agility into the future.
Underlying challenges and possibilities in the Brazilian context
Axel-Berg elaborated on some of the underlying features that are unique to Brazil and hindering its progress toward responsible research assessment reform. The Brazilian higher education system is highly heterogeneous in terms of the nature of the institutions, the fields of study, etc., which complicates the process of developing uniform evaluation policies. Compounding this, cultural factors contribute to resistance in changing mindsets and a lack of participation from government bodies and academic faculty and staff. Because the universities and academic institutions are bound by strict federal laws, there is not much room for change other than on an individual basis currently. Another unique challenge that Axel-Berg highlighted was the language and digitalization gaps in disseminating scholarly knowledge from Brazilian researchers. Strategies for overcoming some of these issues were discussed during the meeting and a number of suggestions were made, including: involving learned societies in the reform process to facilitate more wide-ranging communications in a context- and discipline-specific manner; enhancing trainings on structuring; substantiating and popularizing narrative CVs; retaining staff to support a strong institutional memory; and promoting institutional autonomy for making changes more easily.
Projeto Métricas is taking on these challenges one step at a time, and its immediate goal is to provide guidance and assistance at the individual level. Axel-Berg concluded his presentation by reiterating their long-term goals to bring about a more holistic change by incorporating not only Open Science and research integrity but also society-centric metrics and environmental responsibility as parameters into evaluations.
Sudeepa Nandi is DORA’s Policy Associate