Narrative CV formats are becoming a popular alternative to the traditional CV for the purposes of research evaluation, because they provide room to recognize a broad array of outputs and outcomes from research. But that is not their only advantage. Because narrative formats instill standards and structure into evaluation processes, qualitative information can be assessed more evenly across a pool of applicants.
The Dutch Research Council, Science Foundation Ireland, Swiss National Science Foundation, and UKRI are all experimenting with narrative CV formats as part of their evaluation of grant proposals. At DORA’s virtual funder discussion on September 23, 2020, Rochelle Fritch and Laura Mackey presented Science Foundation Ireland’s (SFI) ongoing work to develop a narrative CV format for its funding mechanisms.
SFI is the largest research funder in Ireland. According to Fritch, it funds individual, collaborative, and center-scale awards for STEM research. Most SFI programs have three areas of assessment that are usually weighted to form the final score: quality of the applicant(s), quality of the research proposal, and impact of the research proposed. SFI is changing the way it defines excellence by moving towards a more holistic view of researchers’ achievements. Fritch emphasized that SFI aims to fund researchers who not only conduct excellent research but also positively contribute to developing the next generation of research talent, society and the economy, the research community, and a positive research culture.
The narrative CV format was first introduced to all funding calls at SFI in 2019, said Fritch. In this version of the CV, applicants were asked to describe five key research achievements and their three most impactful publications. Reviewers were briefed on DORA principles and instructed to focus on the breadth of researcher achievements. Panel chairs were also individually informed of DORA principles. But even with these modifications, applicants and reviewers still relied on proxy measures of success, such as journal-based metrics, publication venue, and H-index. Gauging researcher value by association in this way leads to availability bias. By overweighting information that is easier to recall, reviewers may miss other important pieces of evidence and fail to see the big picture of a researcher’s accomplishments.
So Fritch and Mackey revised their narrative CV format after reviewing best practices and consulting with other international funders to reduce the influence of proxy measures in evaluation. As a result, the four modules from the Royal Society’s Résumé for Researchers were integrated into their CV format: generation of knowledge, development of individuals, contributions to the wider research community, and contributions to broader society. In addition to de-emphasizing proxy measures, the modules support the more holistic vision of researcher excellence that SFI is moving towards. A second section in the CV allowed researchers to describe and contextualize relevant publications.
According to Fritch, SFI took a harder stance against metrics in the revised format, saying applications could be deemed ineligible for review if metrics are included (except for citations).
The CV format was not the only part of research assessment SFI modified, Fritch said. The review process was updated too. Now reviewers are asked to score the four modules independently on a scale of one to five. According to Fritch, this is meant to allow reviewers time for more deliberate thinking, so that they will actively consider each category. Then, the reviewer, using their own judgement, provides a composite score based on their initial assessment of the four modules. Notably, the composite score is not directly dependent on the individual module scores and is the final score used in the review. By using this two-step process, data can be gathered to understand which modules reviewers have put the most weight on in their final score and if reviewers are effectively recognizing a broader range of researcher accomplishments.
It is no surprise that Fritch and Mackey see the implementation of DORA, and responsible research assessment in general, as an iterative process. Based on data gathered from the revised narrative CV format and review process, they expect to make further modifications and improvements to the process.
DORA’s funder discussion group is a community of practice that meets virtually every quarter to discuss policies and topics related to fair and responsible research assessment. If you are a public or private funder of research interested in joining the group, please reach out to DORA’s Program Director, Anna Hatch (info@dora.org). Organizations do not have to be a signatory of DORA to participate.