A DORA Community Engagement Grants Report
In November 2021, DORA announced that we were piloting a new Community Engagement Grants: Supporting Academic Assessment Reform program with the goal to build on the momentum of the declaration and provide resources to advance fair and responsible academic assessment. In 2022, the DORA Community Engagement Grants supported 10 project proposals. The results of the Co-creating a Responsible Use of Metrics for Research Assessment in Colombian Science, Technology and Innovation System project are outlined below.
By Salim Chalela Naffah (Universidad del Rosario), Maria Alejandra Tejada, Diana Lucío-Arias, César Pallares Delgado — (Colombia)
The time has come
Following Derek de Solla Price’s contributions (1), in Colombia scientific capabilities —evidenced in the human, technological and social efforts mobilized around the creation, circulation and appropriation of scientific knowledge— have grown exponentially in the last 20 years. This evidence can be found in the simplest of indicators, such as the growth of publications with at least one author affiliated to a Colombian institution in scientific journals but also in more complex ones, such as the growth of authors publishing their first article, PhD Programs, PhD graduates, involvement in international collaboration networks, research groups, or researchers in the country, just to name a few. The growth in scientific capabilities has motivated a national reflection on the most suitable infrastructure, mechanisms and tools to promote consolidation and decentralization, as until now capabilities concentrate around the biggest cities in the country.
Colciencias —Colombia´s former public organization responsible for promoting science and technology— was formalized in 1968 as a public fund attached to the Ministry of Education (2) and has adapted in order to attend the requirements of a more diverse and demanding scientific community, in one hand, and, in the other, to contribute in the transition to a knowledge base economy and society in Colombia. In 2020 Colciencias became the Ministry for Science, Technology and Innovation. Regardless of this transition, and the growth in the human, technological, institutional and social capabilities for generating, circulating and promoting the appropriation of scientific knowledge, in the past 20 years, less than 0,3% of Colombia´s PIB was spent on R&D (3) while 64% of research groups and 79% of researchers in the country registered their address in one of the 5 major cities of the country (4).
Until the mid-90, public policy efforts to promote R&D in Colombia followed a model of research based on small elites producing and validating the produced knowledge through peer review processes. This encouraged the consolidation of few strong, internationally visible and very specialized disciplinary closed clusters attracting most of the national funds. Additionally, a slow transition in the manufacturing and service sector of Colombia to more digital, technology and knowledge based environments, has impacted the low levels of incorporation of PhDs to sectors different than the academic one. In a more macro level, this led to the omission of the productive sector as valuable spaces for validating the application of scientific knowledge generated in universities, research centers, groups or labs of the country.
Two important elements contributed in the first decades of this century to the discussion for a shift in the ways money was allocated to research and thus in the ways that research products and results were valued: the diversification of scientific skills and the expansion of national PhD programs —some interdisciplinary by nature— in the social and human sciences, in engineering and arts. This discussion led to the implementation of systems for the recollection of geographically dispersed information that could inform science policy; but because these information systems have been modeled, in thus used, following the natural and exact sciences, they can constrain and demotivate the plurality and diversity in the forms of generation, communication, circulation and appropriation of scientific knowledge and, in occasions, at the expense of more direct impacts in the local contexts.
At this moment, the growing number of PhD graduates from a variety of disciplines demands for a comprehensive and participative scientific policy that privileges a more diverse and informed-based allocation of resources, and integrates an unbiased system of recognitions in the ways knowledge is produced, transferred and used. In this historical and contextual background, the project Responsible Metrics was launched in 2021 and was subsequently selected as one of the initiatives to receive funding from the community engagement grants program in DORA. The project capitalized from the experiences in research, monitoring and assessment exercises from diverse sectors, institutional backgrounds, and geographical location, with the purpose to diagnose the main challenges and difficulties of the current research assessment efforts. The collective nature of the project meant that the financial resources obtained were to be used in broadening the scope of the project as to involve more actors in the reflection around responsible metrics for assessment and evaluation of research. The discussion was organized in eight meetings of the scientific commission, but also five “chairs”, which were designed as formative spaces. The funding obtained allowed as well the implementation of specific spaces for action and proposition with Vice-chancellors that concluded in doathon around specific actions that should be attended in the short term to improve the evaluation of research results. The state of the art in evaluation instruments and mechanisms was balanced against the urgent challenges in the different dimensions involved in the assessment of the research process; its results and their impacts, acknowledging their disciplinary plurality and incommensurability.
The information generated in the different spaces was then systematized using 7 categories aligned to the recommendations gathered in the different meetings and which, in broad terms, were related to: (1) creation of spaces with the participation of multiple actors to accord evaluation principles, instruments and contexts (2) study the viability of including, among the relevant criteria in the evaluation, the local needs and demands for new scientific knowledge from the regions and diverse territories of the country (3) transform the systems of incentives and monetary recognitions for production so that it considers a broader diversity of results and not only articles in quartile-positioned journals (4) promote the qualitative perspective in evaluation to complement traditional quantitative indicators (5) articulate assessment of higher education institutions to the efforts that have been sustained in the Ministry of Science, Technology and Innovation (6) evaluation must be robust, transparent, participative and fair to diversity and (7) implement a mechanism that allows for actor´s characterization in order to protect research diversity. The systematization of information following the recommendations proposed allowed a schematic visualization of the interrelations among recommendations, missing elements and what other topics will be important to add to the final version of the recommendations.
Among the recurrent topics proposed by the participants in the different spaces was the need to recognize the richness in the information that the country has collected but that has lacked a deep and systematic analysis. The model to “recognize and classify” (5) researchers and research groups has allowed to capture information of more than 5,700 research groups and more than 16,500 researchers. This information should be analyzed applying analytical techniques such as text mining and triangulated with more qualitative data to comprehend the different needs to consolidate a scientific workforce and infrastructure at service of the different regions in the country. An invitation to profit from the richness of the information collected through the different information systems, together with the quest for a critical reflection on the negative externalities that might arise from the differential valuation of research products by the characteristics of the circulation means rather than by the products characteristics alone.
The need to break from traditional indicators built from information in traditional indexing and abstracting systems (Scopus and Web of Science primarily) and recognize other means of circulation of scientific knowledge resonated as well in different spaces. Ahead of its time, Latin- America bet for open access to scientific knowledge as a way to promote circulation and visibility through free access to scientific content. A different rationale perhaps than today´s movements that through costly processing charges reinforce the asymmetries in the knowledge circles. This led to a consensus of the participants in the need to raise awareness on the inconvenience, and unfairness, of valuing more a scientific contribution for the quartile it classifies than for its impact, albeit only scientific and measured in citations.
Perhaps the most valuable contribution of this resources for the project on Responsible Metrics is the consolidation of a national network, promoted in the name of universities and higher education institutes by ASCUN and in collaboration with public entities, that will continue the constructive dialogue around a Policy Brief with a series of recommendations to orient responsible assessment of research and at the same time advocate for informed based policies instead of policies aimed at metrics. Recognition of the importance of diversity and participation, in the purposes and levels in the assessment, in the research forms, its products, their circulation, appropriation and impact, and therefore in the assessment conditions, ponderations and evaluations. The presence in this network of actors, at the national and institutional level should nurture a healthy ecosystem for the generation, circulation and appropriation of disciplinary diverse scientific knowledge.
Resources
- Price, D. J. de Solla
- Price, D. J. de Solla (1963). Little Science, Big Science. New York: Columbia, University Press.
- Price, D.J. de Solla (1965). Networks of scientific papers. Science, 149 (3683), 510-51
- Price, D.J. de Solla (1978). Toward a Model for Science Indicators. In Y. Elkana, J.Lederberg, R.K. Merton, A. Thackray, & H. Zuckerman, H. Toward a M
- Its full name was Colombian Fund for Scientific Research and Special Projects “Francisco José de Caldas” (Colciencias) ( https://ocyt.org.co/wp-content/uploads/2021/06/colciencias40.pdf
- https://ocyt.org.co/indicadoresctei2020.ocyt.org.co/Informe%20Indicadores%20CTeI%202020%20v1.pdf
- https://minciencias.gov.co/la-ciencia-en-cifras/grupos
- scienti.minciencias.gov.co