A DORA Community Engagement Grants Report
In November 2021, DORA announced that we were piloting a new Community Engagement Grants: Supporting Academic Assessment Reform program with the goal to build on the momentum of the declaration and provide resources to advance fair and responsible academic assessment. In 2022, the DORA Community Engagement Grants supported 10 project proposals. The results of the The Colombian responsible metrics Project: towards a Colombian institutional, methodological instrument for research assessment are outlined below.
By César Pallares, Salim Chalela, María Alejandra Tejada, Elizabeth Bernal, César Rendón, Alida Acosta, Lorena Ruíz, Hernán Muñoz — Asociación Colombiana de Universidades, Asociación Colombiana de Editoriales Universitarias, Consorcio Colombia, CoLaV, Observatorio Colombiano de Ciencia y Tecnología, Red GCTI, COREMA (Colombia)
In March 2021, Colombia launched a new edition of its biannual assessment of research groups. As usual, this event generated reactions from the academic community, and a debate emerged around the suitability and use of this assessment exercise. However, that reaction differed from previous years since the community was more organized than before, and the critiques and recommendations were more structural and intended to contribute to the country. Different academic networks summed up efforts to raise the level of the debate and to include representation from those who have studied the evaluation framework and those responsible for helping the researchers who are evaluated.
One of those efforts was the responsible metrics initiative. Leading Colombian institutions, such as the Colombian Association of Universities -Ascun-, the Colombian Association of University Publishers -Aseuc-, the Colombian Observatory of Science and Technology -OCyT- , the Colombian Association of Research Managers -COREMA, the Colombian Network of Management and Governance of Science and Technology -RedGCTI-, the Collaboratory for Computational Social Sciences -CoLaV UdeA-, and the Consorcio Colombia, worked together to facilitate and foster scenarios to discuss the potentialities and limitations of responsible metrics for research evaluation in Colombia.
The coordination team, consisting of representatives of the institutions mentioned above, was responsible for gathering the insights and reaching the two purposes of our initiative: to propose a policy brief aimed at changing the research assessment at the national level, and to develop a Colombian rubric that helps institutions to design their self-assessment framework. To reach this goal, we defined two methodological steps. First, we organized seven international seminars in which experts shared their perspectives and experiences around research assessment. We had the participation of Inorms, Research on Research Institute, Ingenio Institute, the CWTS of Leiden, DORA, Folec, and others. Second, with the contribution of universities, we organized two commissions, one to propose the policy brief and the other to develop the Colombian rubric.
Our first output was a concerted definition of 13 problems associated with the Colombian research evaluation system. Those are:
- Evaluation disconnected from the country’s reality
- Lack of knowledge of alternative ways of doing research evaluation
- Standardization of the measurement method
- Incentive schemes that generate inappropriate behavior
- Lack of articulation among the actors of the science, technology, and innovation system on research evaluation criteria
- Lack of funding for STI
- Economic interests that skew the evaluation and focus on individuals
- Delegitimization of evaluation as a valid exercise to promote research
- Economic interests of external stakeholders as responsible for the definition of evaluation models
- Definition of quantitative metrics focused on journal indexing systems that lose sight of research quality.
- Lack of open spaces to discuss evaluation models based on consensus building.
- Resistance to change in certain system actors reduces the possibility of exploring other possibilities.
- The lack of interoperability between existing information systems in the country makes it challenging to generate alternative metrics and indicators.
From these problems, this project found that the institutions of higher education (IHE) are mainly affected by three of them: lack of knowledge of alternatives to assess research(2), the national assessment ecosystem(10), and resistance to change(12). Focusing on them, we developed a strategy to design our tool to promote assessment change in Colombian institutions.
The next step was to interact with international standards and rubrics. We selected three: 1) Scope from INORMS, 2) SPACE from DORA, and 3) FOLEC. We studied their steps, the recommendations they supplied, the cases that implemented those rubrics, and which lessons they learned. This information allowed us to see the common points we can use in our exercise: the need to configure an assessment committee to prevent the possible indirect impact the assessment might cause and evaluate the assessment strategies.
Using, as a starting point, the insights from these already developed frameworks, we settled ourselves to develop a rubric tailored to the Colombian system and research management practices in the country.
We develop a Colombian rubric from those inputs to help design the assessment exercise. Our rubric has five stages. In the Ideation stage, the University creates the steering committee of the assessment, with principles of diversity (gender, discipline, age, ethnicity, among others), whose role is to define why the evaluation is necessary for the University. In the Design phase, the University establishes different options using design thinking tools to solve the institutional challenge that requires an assessment. The selected way is tested in
the pilot stage to find unintended outcomes, identify the public that might be discriminated against by the review, and receive feedback on the process. In the implementation stage, the evaluation procedure is carried out by the University, but it should include the possibility of being changed if the institution finds any significant problem; in this sense, the process to change the evaluation should be clear from the beginning for all stakeholders. Finally, the evaluation stage tries to understand what worked and did not in the assessment, so the institution learns for future assessment exercises.
Once the rubric was completed, we organized five focus groups with stakeholders and experts on research metrics. They gave us feedback to improve the tool and alerted us about shortcomings its implementation might carry. Lastly, six months later (July 2022), we ran a workshop where research vice presidents gathered to discuss responsible metrics. To analyze our proposal, we organized the workshop around two moments:
First, presentation of responsible metrics and rubrics: the first activity we conducted was to present the responsible metrics framework and the Colombian initiative. We focused more on describing SCOPE, SPACE, FOLEC, and our proposal.
Second, working groups around selected topics: we organized eight working groups, each responsible for analyzing a specific hypothetical scenario. Each group had to solve three challenges: 1) Select the principles that should orient research evaluation, 2) Define the conception of quality and which should be the desired characteristics for the University in that scenario, 3) Construct the profiles of the members of the steering committee for the assessment scenario.
The topics around which the workings groups were working are:
- Research Awards (César Pallares)
- Select which research projects should get a (Salím Chalela)
- Hire new professors at the University (Hernán Muñoz)
- Promotions in the research career (Elizabeth Bernal)
- To give incentives (financial or not) to increase research performance (María Alejandra Tejada)
- To select postdocts or Ph.D. holders to work in the institution (Alida Acosta)
- Define the criteria to select the research papers that could get their APC funded (César Rendón)
- To select research books to be published (Lorena Ruíz)
This workshop was a terrific opportunity to show the research directives that new frameworks to assess research is possible. The next step was to build a website for the initiative, making it easier for the scientific community to access information on responsible metrics in Spanish and to see the alternatives they can use to assess their research performance (https://ww.metricasresponsables.co). We produced this website thanks to the support of DORA’s community grant.
In addition, we developed resources that can help researchers to understand responsible metrics and institutions to apply new frameworks to access research. To do that, we have developed infographic materials to disseminate the logic of responsible metrics and the results of our initiative. We are pleased with the results of our work. Responsible metrics are now a known concept in the Colombian system of R&D, and increased agents are exploring it to change their evaluation practices. We understand that the work does not stop with what we have done, but our goals have changed as we finish this project. First, we will update our resources and our website with new developments that might be built in our country (for example, some institutions members of this initiative are working on technical guidelines of research metrics) or at the international level (as the new toolkits that DORA has been working on). We look to supply a website that researchers can see as an updated source of information. Second, as organizations, we hope to keep our efforts to promote and disseminate the use of responsible metrics in institutions so the rubrics keep momentum. Finally, we will contribute to the academic community by analyzing responsible metrics and the experiences we gained with this initiative. Therefore we can contribute to expanding the available knowledge about new ways to assess and measure research.