Each quarter, DORA holds a Discussion Group meeting for National and International Initiatives working to address responsible research assessment reform. This community of practice is a space for initiatives to learn from each other, make connections with like-minded organizations, and collaborate on projects or topics of common interest. Meeting agendas are shaped by participants. If you lead an initiative, coalition, or organization working to improve research assessment and are interested in joining the group, please find more information here.
During the DORA National and International Initiatives Discussion Group meeting on August 13, 2024, members of the group discussed changes in research policy happening at each of their respective organizations. The group also heard a presentation from Fernanda Beigel, Chair of the UNESCO advisory committee for Open Science (2020-2021), Principle Researcher at the National Scientific and Technical Research Council (CONICET), Argentina, and Chair of Argentina’s National Committee for Open Science. Beigel, who was joined by Celia Quijano Herrera, Maria Soledad Gutierrez Parodi, and Erika Teliz Gonzalez of the Uruguay National Council for Innovation, Science and Technology (CONICYT), presented on her 2024 report Un estudio de la evaluación académica en Uruguay en perspectiva reflexiva or A study of academic evaluation in Uruguay from a reflective perspective. The report and supporting materials are available in Spanish and the executive summary is available in English here.
A study of academic evaluation in Uruguay from a reflective perspective was commissioned based on a public call for proposals by CONICYT out of an interest in evaluating the evaluation of researchers in Uruguay and supporting new institutional assessment practices. This work provides: 1) an important foundational understanding of academic incentives (or disincentives) in Uruguay, and; 2) a set of evidence-based recommendations for how to implement more responsible research assessment practices in Uruguay.
Beigel began her presentation by providing context on Uruguay: Uruguay is a relatively small country of over 3 million citizens, with 1.84 researchers per 1000 inhabitants. One public university (the University of the Republic Uruguay) accounts for 75% of the national research output. In Uruguay, there is extensive overlap between national-level research assessment and institution-level research assessment. Beigel’s work provides a unique perspective and insight into the landscape of assessment and reform in the context of a small country with extensive national oversight over research assessment. This is important because the level of interconnectivity between institutional and national assessment policy determine the key challenges, logistic considerations, and approach that must be taken into consideration when advocating for or reforming assessment practices.
Beigel outlined the different major systems of research assessment in Uruguay and their relationship to one another: the University of the Republic Full Dedication Regime (a university-level assessment system known as RDT) and the National System of Researchers (a national-level assessment system known as SNI), the latter of which is integrated into the Uruguayan Curriculum Vitae system (CVUy). Within these assessment systems, reviewers retain a high degree of autonomy to make decisions regarding faculty promotion. The report includes analyses between the evaluation systems and professional trajectory, methods of knowledge circulation (e.g., publications, books, etc.), combining those data with interviews and focus groups with “members of evaluation committees, officials, academic-scientific referents, and researchers.”
The report includes twenty recommendations for each of these systems across a range of topics. The recommendations for the production indicators and circulation of knowledge include:
- 9. Broaden the notion of scientific production to include diverse profiles and value both traditional publications and technological production, technical contributions, artistic productions, social reports with public policy recommendations.
- 10. Promote publication in quality scientific journals, in open access diamond, edited in the country and in Latin America, stimulating quality communication circuits and expansion of audiences.
- 11. Value the tasks of academic editing (journal management, participation in editorial teams) in the permanence and promotion of the country’s academic evaluation systems.
Beigel, Quijano Herrera, Gutierrez Parodi, and Teliz Gonzalez also discussed the challenges inherent to implementing reform when assessment standards are set at a national level. While a long-term goal is changing national policy to incorporate the report recommendations, they highlighted that an immediate and critical starting point would be the autonomous reviewer panels that make assessment decisions. Sharing resources on the misuse of quantitative indicators and best practices in research assessment with these reviewer panels will be an important step towards building buy-in and support for reform.
We also heard updates from other discussion group member organizations, which are briefly summarized below:
- Researchers’ views on diversity of career assessment criteria in Finland: a survey report. https://doi.org/10.5281/zenodo.11612535
- Global Research Council Dimensions of Responsible Research Assessment. https://figshare.com/articles/online_resource/Dimensions_of_Responsible_Research_Assessment_full_report_and_summary_/26064223
- The SCOPE framework – implementing ideals of responsible research assessment https://f1000research.com/articles/12-1241
- The CoARA Reforming Academic Career Assessment Working Group released the initial findings on its survey of institutional assessment practices (scroll down to Activities). https://coara.eu/coalition/working-groups/wg-reforming-academic-career-assessment/
Haley Hazlett was DORA’s Program Manager.