Each quarter, DORA holds a Community of Practice (CoP) meeting for National and International Initiatives working to address responsible research assessment reform. This CoP is a space for initiatives to learn from each other, make connections with like-minded organizations, and collaborate on projects or topics of common interest. Meeting agendas are shaped by participants. If you lead an initiative, coalition, or organization working to improve research assessment and are interested in joining the group, please find more information here.
Socially impactful research is multi-faceted, requiring transdisciplinary approaches that involve a wide range of actors, approaches, and pathways to influence change. However, addressing the epistemological and methodological variability of transdisciplinary research (TDR) has been challenging. There is a need for standards and methods to define and assess quality in TDR. To address this, Brian Belcher and his colleagues from the Sustainability Research Effectiveness Program at Royal Roads University, Canada conducted a systematic literature review that provides several definitions and ways of assessing research quality in a transdisciplinary context. During our May CoP meeting, Brian Belcher, Rachel Claus, and Rachel Davel presented some excerpts of their study to highlight how their Transdisciplinary Research Quality Assessment Framework (QAF) has advanced and been refined through testing.
The Linear Model of Research Impact assumes that the production and dissemination of new knowledge will eventually reach those who will find it useful, leading to uptake, scaling, and inevitable impact for society at large, where the world becomes a better place. In reality, there are multiple pathways to research influence. Researchers aiming for social impact take on a more active role in relationship-building through partnerships and networking, knowledge co-generation, capacity-building (particularly to do and use research), public discourse and policy processes, lobbying and negotiation, and more. Transdisciplinary approaches facilitate and generate value through these processes to support greater research impact. Yet, Belcher noted that there are limits to a project’s influence as it progresses through the research process. When passing the boundary from a project’s sphere of control (i.e., activities and outputs) into its sphere of influence (i.e., actors whom the project works with and through to influence changes in behaviours), there are more external processes and influences at play that can affect the realization of a project’s intended outcomes and ultimately social, economic, and environmental impacts.
As background to the team’s latest revisions of the QAF, the QAF originates from a systematic review of literature to determine the most appropriate principles and criteria to define and assess TDR quality . The literature identified dominant themes that should be considered in TDR evaluation: engagement with problem context, collaboration and inclusion of stakeholders, heightened need for explicit communication and reflection, integration of epistemologies and methods, recognition of diverse outputs, focus on outcomes and impact, and reflexivity and adaptation throughout the process. The objective of the systematic literature review was to pinpoint suitable principles criteria to delineate and gauge research excellence, and subsequently structure these into a comprehensive assessment framework. The QAF was published in 2016, organized by the four principles of : Relevance, Credibility, Legitimacy, and Effectiveness.
Over the past several years, the team has conducted a series of case study evaluations of completed research projects where they applied the QAF to learn lessons about how TDR qualities can support the realization of outcomes. Through this testing, the team also identified several aspects of the QAF that could be improved. Some of the main changes include revisions to the principle and criteria names and definitions to support clarity, the addition of missing criteria, the removal of overlap in criteria definitions that led to double-counting, and replacement of the former rubric with practical guidance. The revised principles are defined as:
- Relevance: the appropriateness of the problem framing, research objectives, and approach for intended users and audiences;
- Credibility: the rigor of the research design and process to produce dependable and defensible conclusions;
- Legitimacy: the perceived fairness and representativeness of the research process; and
- Positioning for use: the degree to which research is likely to be taken up and used to contribute to outcomes and impacts.
A key emphasis of the latest version of the QAF is to assess and score each criterion against the project’s purpose. QAF scores can be visualized in spidergrams to support further analyses, such as identification of TDR qualities that are present or absent in projects (Image 1) as well as comparisons between projects (Image 2).
For example: Project A has a strong socially relevant problem but scores lower on the criteria pertaining to research objectives, communication, and explicit theory of change (Image 1). Project A satisfied most of the positioning for use criteria and relatively fewer of the credibility and legitimacy criteria.
Image 1: Spidergrams used to visualize how a project scores against the QAF’s principles of relevance, credibility, legitimacy, and positioning for use
This visualization can also be used to compare projects. For instance, Project A might be strong in addressing a socially relevant research problem, but Project B is much stronger in terms of proposing 1) relevant research objectives, 2) clearly defined problem context and 3) engagement with the problem context (Image 2).
When asked how this framework could be more impactful, Belcher said that involving stakeholders in the research process would help them understand how the research contributes to change as well as their role within the change process.
In addition to ex-post evaluation, the QAF enables a structured assessment of project proposals ex-ante to identify and allocate funding to projects that are relevant, credible, legitimate, and well positioned for use. It can also be used to appraise the weaknesses in a proposal from a transdisciplinary perspective. Belcher also mentioned that the QAF has been applied to some projects within a single traditional discipline that had several strong elements and contributed to social change processes. Additionally, some Canadian and international organizations have incorporated these principles and criteria into their research proposal guidelines. These include (1) the Pacific Institute for Climate Solutions (a collaboration of four universities in British Columbia), (2) the Natural Sciences and Engineering Research Council of Canada (NSERC; a Canadian federal funding agency), and (3) CGIAR (an international agricultural research group aiming to deliver science for food security).
Image 2: Comparison of two projects’ QAF scores for the Relevance principle
Belcher highlighted that the application of the QAF to research evaluation will add more transdisciplinary qualities that will achieve more impactful outcomes. However, more widespread application and testing to build the evidence base is needed, and this is possible when the framework is shared among other users. Expanded applications would enable users to identify challenges and opportunities to inform revisions.
A copy of the presentation can be accessed on the Sustainability Research Effectiveness team’s website.
Queen Saikia is DORA’s Policy Associate
References
- Belcher, B. M., Rasmussen, K. E., Kemshaw, M. R., & Zornes, D. A. (2016). Defining and assessing research quality in a transdisciplinary context. Research Evaluation, rvv025. http://doi.org/10.1093/reseval/rvv025
- Cash, D., Clark, W. Alcock, F., Dickson, N., Eckley, N., and Jager, J. (2002). Salience, Credibility, Legitimacy and Boundaries: Linking Research, Assessment and Decision Making. KSG Working Paper Series RWP02-046
- Guthrie, S., Wamae, W., Diepeveen, S., Wooding, S., and Grant, J. (2013). Measuring Research: A Guide to Research Evaluation Frameworks and Tools. Rand Europe.
- Ofir, Z., Schwandt, T., Duggan, C., and McLean, R. (2016). Research Quality Plus: A Holistic Approach to Evaluating Research. IDRC, Canada.
- Stokes, D. (1997). Pasteur’s Quadrant: Basic Science and technological innovation. Brookings Institution Press
Relevant resources shared on the call
- Incentivizing Open in Reappointment, Promotion, Tenure, and Hiring: https://www.heliosopen.org/news/spotlight-series-recap-incentivizing-open-in-reappointment-promotion-tenure-and-hiring
- Dutch knowledge institutions and research funders publish road map of concrete plans for Recognition & Rewards: https://recognitionrewards.nl/2023/04/04/all-knowledge-institutions-create-new-career-paths-this-year/
- ARIS Rubric for Assessment of “Broader Impacts” https://aris.marine.rutgers.edu/
- Resource Quality Plus (RQ+) : https://www.idrc.ca/en/rqplus
- HELIOS: https://www.heliosopen.org/news/spotlight-series-recap-incentivizing-open-in-reappointment-promotion-tenure-and-hiring
- CoARA: https://coara.eu/news/coara-holds-first-brokerage-event-as-part-of-the-working-groups-proposal-process/
.