Participant Commentary – Driving Institutional Change for Research Assessment Reform

Scientific societies have a vested interest in research assessment as standard bearers for their profession. We represent members who are at all career stages and in many different career paths. Societies have multiple roles in the assessment infrastructure. They publish scientific journals; they host large and small meetings; they provide professional development training; they give recognition through awards and fellowships; and they set standards for the profession. Collectively, this gives societies a variety of leverage points to affect change.

The conundrum is easy to understand:  Conventional teaching assessments rely heavily on student feedback, which, whether through metrics or narrative comments, is often fraught with bias.  It is even more difficult to assess teaching when done in “engaged” settings, not in the classroom (e.g., for medical schools, in association with patient care).

Research assessment can be extremely useful as a tool to evaluate the quality and the impact of the activities needed to achieve the Sustainable Development Goals (SDGs). The SDGs, adopted in 2015 by the General Assembly of the United Nations, envision that by the year 2030 the world could be transformed.

Current discourse on research assessment places high emphasis on “impact.” However, there are many different concepts of impact, and many different concepts of how research achieves impact. The resulting ambiguity and confusion confound efforts to improve research assessment. To reliably assess research, we need clarity about what it is we want to assess.

Human-centered design is well-positioned to supplement the ongoing activity of sharing best practices and specific, successful examples of new research assessment strategies, contributing a deep understanding what matters to individuals and entities, and a perspective on realigning incentives, social norms, and points of leverage where we might redefine and reward what’s valued in the future.

Much of the emphasis of DORA’s initiatives has revolved around appropriate metrics and assessments.  Equally important is designing mechanisms that employ those assessments at decision-making steps.

On most campuses, the coalition (faculty, research officers, administrators, librarians, and department chairs) needed to move the needle on research assessment reform has yet to come together. Librarians have every reason to take an active role in help making this happen.

In order to fully realise the benefits of open research, we must fundamentally change the way research is assessed.  Wellcome is seeking to implement the DORA principles in our own funding processes and support research institutions in changing their own assessment practices.

The retention, promotion, and tenure (RPT) process is a critical part of a faculty member’s career, during which they are ostensibly evaluated on scholarship, teaching, and service. However, a faculty member’s funding and publication track records are typically weighted more heavily as indicators of productivity. As a result, flawed metrics of teaching and service persist.