Open University of Catalonia

After the publication of the DORA Action Plan, the university is working toward changing their assessment criteria for career progression and recruitment practices.² Faculty recruitment and career progression align with the three central university missions: teaching, research, and the so-called “third mission,” which is related to societal interactions and impacts.

The change from journal-based metrics to a narrative discussion of “achievements” is currently in effect for postdoctoral recruitment. Examples of achievements include a paper, group of papers, research project, a patent, and more. Under the new criteria, “increasing importance is given to research content and its social impact, rather than just its appearance in various journals and metrics.” With the changes proposed within the Action Plan, UOC aims to transform “evaluation methods using more qualitative, transparent, fair, inclusive and socially relevant formulas that take into consideration not only research quality, but also the societal impact of our research.”³

For the recruitment of research group leaders (full-time researchers at UOC who are not professors), a specific international Scientific Advisory Board analyzes the applicant’s CV and research plan by means of a written proposal and a public presentation. Assessment for the career progression of research group leaders takes place every four years with the advisory board producing a qualitative evaluation report. 

Professors are assessed by the Dean, Vice Rector for Teaching, Vice Rector for Research—as well as the Rector in case of promotion to full professor—together with two or three professors. Assessment is, therefore, peer-reviewed, but journal-based metrics still play a role because UOC is highly dependent on the Catalan University Quality Assurance Agency professor accreditation system, which is mainly based on publications and journal impact factor. However, the agency has recently signed DORA and future changes are expected.

The history and academic culture at UOC is rooted in Open Science. As such, there were several internal drivers to push career progression and recruitment evaluations to be more open. Identifying ways to recognize and reward the social impact of research was also a major motivation for change. “Much of the impact of research cannot be explained or assessed with quantitative indicators, instead it requires a report.”

While UOC has autonomy in decisions regarding postdoctoral recruitment and promotion, the major barrier for the implementation of new academic assessment practices at the faculty level is the centralized external accreditation system for faculty career progression.

Because journal-based metrics are still commonly used to measure academic success at other institutions, there were also some concerns about the new approach to assessment. For example, would it disadvantage UOC doctoral and postdoctoral populations for future academic opportunities?

Instead of creating meaningful culture change, another concern is the risk of creating a “check the box” exercise; early career researchers may see the new assessment criteria as something they need to comply with before moving on to careers based on quantitative publication metrics (e.g., professorial staff).

After initial advocacy by researchers and staff at UOC, the Research and Innovation Committee, chaired by the Vice President for Research, created a DORA task force. This combined bottom-up/top-down approach facilitated the necessary dialogue and advocacy in linking open science with research assessment reform. The DORA Action Plan created by the task force is part of the wider Open Knowledge Access Plan at UOC.²

The task force was pivotal for capacity building, in terms of both defining achievements and for designing rubrics to guide evaluators. It was also critical for advocacy and community engagement to help explain the new process and ensure that evaluation would be consistent in the absence of journal-based metrics. Importantly, the task force stopped short of giving achievement examples. In the action plan, the task force states:

It is vital to provide spaces for discussion to reflect on assessment practices and practices that help to create knowledge and awareness between UOC researchers about new research assessment models and the DORA Declaration. In addition, training sessions should be offered on how to demonstrate the impacts of research beyond metrics.²

Specific obstacles faced were: absence of incentivizing policies or guidelines from external actors (e.g., national/regional governments, research funding organizations); alignment of institutional assessment procedures with nationally and internationally dominant procedures; and lack of institutional autonomy due to national/regional rules and regulations.

The DORA task force was created by the Research and Innovation Committee in early 2018. The task force presented the “UOC signs the San Francisco Declaration (DORA) Action Plan” in December 2018. The progress on action items for open science and assessment is currently ongoing.