The Latin American Forum for Research Assessment

A key hurdle to collective research assessment reform across the 52 countries within the CLACSO network is the breadth of practices among nations and institutions, which revealed the need for specific infrastructure to facilitate communication on policy reform. Therefore, CLACSO created FOLEC to coordinate collective research assessment reform in Latin America and the Caribbean. FOLEC is working to effect the following elements of change:

• Facilitate transparency among CLACSO research member institutions on current research assessment practices.
• Facilitate spaces for discussion among countries and institutions within the region on how to approach research assessment reform by hosting forums and workshops with international CLACSO research members, national research funding and policy agencies, and specialists from the region and other regions. This allows for disparate institutions, regions, and policymakers in Latin America to identify key areas for reform and discuss how to address these areas through policy development and cooperation across the region.
• Create collective documentation for CLACSO research member institutions using iterative feedback gained during FOLEC-facilitated discussion on current research assessment practices in the region and how to approach reform.

The first Latin American Forum for Research Assessment held by CLACSO and CONACYT in November 2019 in Mexico City catalyzed change on the continent. A range of stakeholders attended the forum, including regional research assessment experts (e.g., academic researchers and government specialists), representatives of National Science and Technology Organizations, and CLACSO research member centers. Stakeholders were brought together to discuss international trends and good practices in research assessment. Additionally, FOLEC facilitated stakeholder “discussion about the processes of evaluation of scientific work” with the purpose of originating reform proposals from Latin America and the Caribbean." ¹

According to the meeting report, “The cooperation between CLACSO and CONACYT … is a milestone in the construction of new proposals with a real capacity for implementation and impact.” ¹

This meeting resulted in several key agreements among stakeholders:

• The importance of transforming evaluation practices of scientific work.
• The need to liaise with government representatives from multiple governments across the region to impact policy development.
• Acknowledgment of tension between the privately owned international publications circuit and the more autonomous Latin American and Caribbean regional publications circuit.
• The need to address elements of knowledge production that are qualitative and thus more difficult to measure.
• A dialogue between regional publication repositories, such as Redalyc, SciELO, LATINDEX, and others.

Multiple commissions were formed to discuss and generate proposals that address:

• The meaning of research and evaluation policies
• The evaluation and accreditation of undergraduate and graduate degrees
• The impact and social utility of knowledge generation

So far, FOLEC has published three documents in Spanish and English as part of a series, entitled “For a transformation of the evaluation of science in Latin America and the Caribbean,” which is meant to serve as guidance for assessment considerations and practices in the region:

• Evaluating the evaluation of scientific production" ²
  This document is devoted to the analysis of global and Latin American standards related to bibliometric academic assessment.
• Diagnosis and proposals towards a regional initiative" ³
  The aim of this document is to serve as a foundation for the creation of academic assessment recommendations agreed upon by all CLACSO research member centers.
• Proposal for a declaration of principles" ⁴
  This document is a proposal for a declaration of FOLEC principles to underpin FOLEC’s regional initiatives and its interaction with the global research community.

In 2020, FOLEC conducted a survey of directors of member centers of CLACSO´s network to assess their perceptions regarding the evaluation of science during the Covid 19 pandemic. FOLEC also hosted multiple virtual meetings, both national and international, to bring together stakeholders for substantive discourse on research assessment reform proposals and practices.

As of April 2021, FOLEC has convened a working group to address academic publishing and scientific evaluation in collaboration with the Spanish Higher Council for Scientific Research (CSIC) and the OLIVA Project of the National University of Cuyo, Argentina.

Moving through 2021 and into 2022, FOLEC is collaborating with global and regional organizations like DORA, the Arab Council of Social Sciences (ACSS), International Science Council (ISC), and the European Network for Research Evaluation in Social Sciences and Humanities (ENRESSH) to inform its initiative to improve research assessment. Additionally, FOLEC plans to produce a proposal on research assessment reform to be presented at the 9th CLACSO Latin American and Caribbean Social Sciences Conference on “Patterns of inequalities in Latin America and the Caribbean. Knowledge, struggles and transformations” in 2022. This conference will include a special session for regional and international debate, allowing for iterative feedback from a variety of stakeholders.

Ultimately, FOLEC aims to publish a roadmap to research assessment reform in November 2022 to guide the implementation of new policies and practices at CLACSO research member institutions.

CLACSO’s decision to establish FOLEC to improve research assessment in the region in 2019 was largely driven by internal motivations. There was a general discomfort with the outsized focus on journal-based metrics in research assessment among researchers in the social sciences and humanities spaces, as well as among CLACSO administrators.

External factors, such as the increasing influence of publishing models outside of Latin America, contributed to the growing sense of unease. The publishing system in Latin America is different from North America and Europe in terms of who manages publications. In North America and Europe, journals are published primarily through non-profit and for-profit organizations that require payment to access their publications, although open access modes of publication are clearly on the increase. In Latin America, over 90% of journals are run by scholar-led non-profit initiatives, mainly universities, and use the diamond open access model. Journals from Latin America are indexed by regional services: Latindex-Catálogo, Redalyc-AmeliCA, and SciELO. However, the indicators provided by these Latin American indexing services are not considered by government research evaluation systems within the region, which rely on the Web-of-Science (WoS) and Scopus indicators, produced by Clarivate Analytics and Elsevier, respectively. This is problematic because WoS and Scopus journal collections poorly reflect research outputs published within the region; e.g., in 2016-2017 WoS and Scopus included less than 1% of regional journals in their collections. ¹

The use of institution-run journals avoids certain pitfalls associated with assessing research quality based on proxy indicators like Journal Impact Factor (JIF) or H-index. However, there is still a strong focus on ‘prestigious’ journals because they are traditional indicators of value in North America and Europe, which then puts pressure on Latin American researchers to publish in these venues. The biggest pressure comes from funding agencies and national science councils whose research assessment practices are often guided by international mainstream metrics, such as JIF.

The challenge is that Latin American academic institutions generally do not purchase licenses required to access high-JIF journals, resulting in limited access both for the public and for researchers who cannot view their own work and the work of their peers due to unaffordability.

Working within the international organizational structure of CLACSO, FOLEC has over 30 organizations and academic institutions under its purview. By working within CLACSO and with non-CLACSO stakeholders, FOLEC aims to facilitate further dialogue among organizations and institutions to develop responsible research assessment policies and practices in Latin America and the Caribbean.

Although FOLEC works internationally to facilitate knowledge exchange on research assessment, change is facilitated in a context-specific manner from the bottom-up by working groups across partner institutions and organizations. For example, working groups participate in institutional training initiatives, drive internal change within the institutions of CLACSO, and generate policy briefs for regional research assessment-related issues. Stakeholders involved in these processes include academic leadership, researchers, institution library staff, research management staff, and governmental policy staff. Given the broad scope of countries in Latin America and the Caribbean that FOLEC encompasses, flexibility in a local context is essential to facilitate relevant and effective reform. Additionally, FOLEC uses a participatory design process to collect feedback from stakeholders during national and international meetings to refine and inform research assessment reform proposals moving forward.

In November 2019, FOLEC held its first forum with CLACSO and CONACYT. The event brought together a range of stakeholders and facilitated discussions and proposals for research assessment reform in the Latin American and Caribbean regions.

In 2020, FOLEC conducted a range of activities to facilitate knowledge transfer and discussion among organizations and institutions. They seek to create a common and regional proposal on scientific research assessment by November 2021.