By Brian Belcher (Royal Roads University)
Current discourse on research assessment places high emphasis on “impact.” However, there are many different concepts of impact, and many different concepts of how research achieves impact. The resulting ambiguity and confusion confound efforts to improve research assessment. To reliably assess research, we need clarity about what it is we want to assess.
Typical measures of “research impact” have focused on indicators of influence on other research (e.g., publications in “high impact” journals, citations, impact factors). DORA has very helpfully highlighted flaws in this approach.
At the same time, there has been a push from many quarters for research to generate more impact on society, and for more evidence of that impact. Within academia, this approach is most advanced in the UK’s “Research Excellence Framework”, which defines “impact” as an “effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia.” But many other academic research funding and evaluation approaches have likewise increased emphasis on influences beyond the academic. This concept of impact is wide-ranging; the effect, change, or benefit could be small or large, proximate or distal, ephemeral or long-lasting.
And there is also a large and growing body of “research for development,” “use-oriented research,” “transdisciplinary research,” etc. that explicitly aims to contribute to tangible changes in social, economic, and/or environmental conditions. Importantly from the perspective of evaluation, the research being assessed could be sufficient to cause the change or, much more likely, it could serve as one of many contributing factors within a complex system.
I have been involved in a process of developing and implementing a new research assessment approach in a research-for-development context. Because much of this research is funded from international development assistance budgets, there has always been pressure from funders to demonstrate impact at mission level (i.e., poverty alleviation, increased food and nutrition security, and/or improved environmental condition). An organizational change process that promoted results-based management further increased this pressure and stimulated a series of responses including:
- increased emphasis on modelling research within a theory of change;
- concerted effort to conceptualize change as a process to which research contributes;
- re-defining research quality beyond narrow disciplinary bounds;
- defining research effects and related vocabulary by the nature of change;
- developing theory-based evaluation approaches appropriate for interventions in complex systems;
- training and engagement of staff to build capacity and help overcome natural concerns about the broader evaluation focus.
Management support and explicit policy measures, such as requiring new projects to develop explicit theories of change, have helped facilitate the transition.
While there are elements of this process that are unique to the context, many of the principles and many of the lessons are applicable to research assessment more generally.