Case Study

Open University of Catalonia

Interview conducted 7 October, 2020  Compare case studies

The Open University of Catalonia (UOC) shifted the focus of their assessment criteria and practices for recruitment and career progression away from journal-based outputs to a much broader discussion of achievements. Due to the centralization of career progression for faculty nationally, the new assessment criteria apply to postdoctoral fellows and UOC research staff, not professorial staff. Because the history and academic culture at UOC are rooted in Open Knowledge,1 there were several internal drivers to push career progression and recruitment evaluations to promote open scholarship. After initial advocacy by researchers and staff at UOC, the Research and Innovation Committee, chaired by the Vice President for Research, created a DORA task force, which published the university’s DORA action plan online in 2018.2

Who: Organization profile

Country Catalonia (Spain)
Profile of institution distance learning university
Number of FTE researchers 500-1,000
Further information UOC signs the San Francisco Declaration (DORA) Action Plan
Organization of research evaluation Institutional/university levelresearch unit levels
Who is involved? academic leadershipacademic researchers

What: What changed and the key elements of change

After the publication of the DORA Action Plan, the university is working toward changing their assessment criteria for career progression and recruitment practices.2 Faculty recruitment and career progression align with the three central university missions: teaching, research, and the so-called “third mission, which is related to societal interactions and impacts.

The change from journal-based metrics to a narrative discussion of “achievements” is currently in effect for postdoctoral recruitment. Examples of achievements include a paper, group of papers, research project, a patent, and more. Under the new criteria, “increasing importance is given to research content and its social impact, rather than just its appearance in various journals and metrics.” With the changes proposed within the Action Plan, UOC aims to transform “evaluation methods using more qualitative, transparent, fair, inclusive and socially relevant formulas that take into consideration not only research quality, but also the societal impact of our research.”3

For the recruitment of research group leaders (full-time researchers at UOC who are not professors), a specific international Scientific Advisory Board analyzes the applicant’s CV and research plan by means of a written proposal and a public presentation. Assessment for the career progression of research group leaders takes place every four years with the advisory board producing a qualitative evaluation report. 

Professors are assessed by the Dean, Vice Rector for Teaching, Vice Rector for Research—as well as the Rector in case of promotion to full professor—together with two or three professors. Assessment is, therefore, peer-reviewed, but journal-based metrics still play a role because UOC is highly dependent on the Catalan University Quality Assurance Agency professor accreditation system, which is mainly based on publications and journal impact factor. However, the agency has recently signed DORA and future changes are expected.

Why: Motivation for change

The history and academic culture at UOC is rooted in Open Science. As such, there were several internal drivers to push career progression and recruitment evaluations to be more open. Identifying ways to recognize and reward the social impact of research was also a major motivation for change. “Much of the impact of research cannot be explained or assessed with quantitative indicators, instead it requires a report.”

While UOC has autonomy in decisions regarding postdoctoral recruitment and promotion, the major barrier for the implementation of new academic assessment practices at the faculty level is the centralized external accreditation system for faculty career progression.

Because journal-based metrics are still commonly used to measure academic success at other institutions, there were also some concerns about the new approach to assessment. For example, would it disadvantage UOC doctoral and postdoctoral populations for future academic opportunities?

Instead of creating meaningful culture change, another concern is the risk of creating a “check the box” exercise; early career researchers may see the new assessment criteria as something they need to comply with before moving on to careers based on quantitative publication metrics (e.g., professorial staff).

How: Processes and dynamics for developing, implementing and managing change

After initial advocacy by researchers and staff at UOC, the Research and Innovation Committee, chaired by the Vice President for Research, created a DORA task force. This combined bottom-up/top-down approach facilitated the necessary dialogue and advocacy in linking open science with research assessment reform. The DORA Action Plan created by the task force is part of the wider Open Knowledge Access Plan at UOC.2

The task force was pivotal for capacity building, in terms of both defining achievements and for designing rubrics to guide evaluators. It was also critical for advocacy and community engagement to help explain the new process and ensure that evaluation would be consistent in the absence of journal-based metrics. Importantly, the task force stopped short of giving achievement examples. In the action plan, the task force states:

It is vital to provide spaces for discussion to reflect on assessment practices and practices that help to create knowledge and awareness between UOC researchers about new research assessment models and the DORA Declaration. In addition, training sessions should be offered on how to demonstrate the impacts of research beyond metrics.2

Specific obstacles faced were: absence of incentivizing policies or guidelines from external actors (e.g., national/regional governments, research funding organizations); alignment of institutional assessment procedures with nationally and internationally dominant procedures; and lack of institutional autonomy due to national/regional rules and regulations.

When: Timeline for development and implementation

The DORA task force was created by the Research and Innovation Committee in early 2018. The task force presented the “UOC signs the San Francisco Declaration (DORA) Action Plan” in December 2018. The progress on action items for open science and assessment is currently ongoing.

References

  1. Open knowledge action plan: Frame of action (2019). Retrieved 18 December 2020 from: https://www.uoc.edu/portal/en/coneixement-obert/index.html
  2. UOC signs the San Francisco Declaration (DORA) Action Plan (2018). Retrieved 22 November 2020 from: https://www.uoc.edu/portal/_resources/CA/documents/coneixement-obert/dossier-dora_en.pdf
  3. DORA Working Group, UOC. Overcoming the journal impact factor and transforming research assessment, a perspective from the Open University of Catalonia. EUA Expert Voices (2019). Retrieved 22 November 2020 from: https://www.eua.eu/resources/expert-voices/133:overcoming-the-journal-impact-factor-and-transforming-research-assessment,-a-perspective-from-the-open-university-of-catalonia.html