On Wednesday April 3, 2019, we hosted a #sfDORA community interview with Janet Halliwell to learn more about the 2017 report from the Federation of the Humanities and Social Sciences in Canada, Approaches to Assessing Impacts in the Humanities and Social Sciences. We also wanted to hear about a new project she is working on to improve assessment in the Humanities and Social Sciences (HSS) by creating a standardized vocabulary of terms related to research assessment.
Engaging colleagues in conversations about impact and assessment can be challenging. According to Halliwell, many in the HSS community feel that focusing too heavily on impact undermines the value of curiosity-driven research. And that is not the only problem. Impact can take on a number of different meanings, which makes measuring it difficult. But by providing a framework for assessment, the report encourages bottom-up discussions in the community.
To make these discussions manageable, the report bins impacts into five categories:
- Scholarship
- Capacity
- Economy
- Society and Culture
- Practice and Policy
While scholarship and capacity are internal to academe, the other categories are focused on societal impacts. Taken together, the broad categories form an overall impact landscape. However, projects will not always have an impact in every location. And this is a key point, because it highlights the need for a certain degree of flexibility in how we approach assessment. Impacts vary; there is not a one-size-fits-all approach. But by separating them into categories, evaluators can select what impacts are most appropriate for the project and choose the best way to assess them.
Timing isn’t everything
The report uses case studies to demonstrate how some of its principles can be applied and used for the purposes of research assessment. One case study examines how long it takes to recognize HSS impacts, where the research produced by a professor of folklore studies is used as evidence in a court case many years after he made the discovery. The professor never learns what role his work played in the case. Impact takes time. Even a 5-10 year period may be too short to capture the return-on-investment of a research project.
Halliwell believes that while timing and research recognition can’t be reconciled, they can be managed. “The most you can ask for curiosity-driven research is to think about and address possible uses and impacts at the start.” Managing the expectations of funders and institutions is vital. Researchers should not be penalized if they do not achieve something they never set out to do.
Creating Clarity
Now Halliwell is involved with a project that takes research assessment reform one step further by developing a standardized vocabulary for evaluation. The project is a joint initiative of the Federation for the Humanities and Social Sciences in Canada and CASRAI, which is the group that devised the Contributor Roles Taxonomy (CRediT). Halliwell quoted the abstract of a recent paper from Brian Belcher, Royal Roads University, and Markus Palenberg, Institute for Development Strategy, to emphasize why clarity is needed.
“The terms “outcome” and “impact” are ubiquitous in evaluation discourse. However, there are many competing definitions that lack clarity and consistency and sometimes represent fundamentally different meanings. This leads to profound confusion, undermines efforts to improve learning and accountability, and represents a challenge for the evaluation profession.”
So far the group has three draft definitions to share:
- Output—new knowledge, technical or institutional advances, or other direct products or services produced by research;
- Outcome—a change in knowledge, attitude, skills, and/or relationships that manifests a change in behavior that would result in whole or in part from research;
- Impact—a change in flow or a change in state resulting in whole or in part from a chain of events that may have many other actors and stakeholders involved.
When we apply these definitions, outputs, outcomes, and impacts can be arranged as a sequence of events. Outputs are the first step and within a researcher’s sphere of control. Outputs then influence outcomes, which can lead to impacts. Halliwell sees an attenuation of influence, control, and attribution moving through the chain. One approach they are taking now is to see if they can measure and optimize impact along on a given path.