The Dutch Research Council (NWO) is piloting a narrative CV format in the Veni scheme, its major funding instrument for early career researchers. The format advances showcasing diverse types of talent and encourages assessment of quality rather than quantity.
By Kasper Gossink-Melenhorst – Dutch Research Council (NWO)
NWO Veni
The Veni scheme is aimed at early career researchers who can show they are among the very best in their peer group. But how can we determine who is part of that target group? What shows that a researcher belongs to this group? Well, different things do, for different people.
In our view there is no ideal type of academic, no particular set of qualities that is inevitably most important. No universal definition of “quality.” Different research projects require varying types of talent. Academics may excel through specialisation or versatility. They may be team-scientists, or best able to create breakthroughs individually. They might be bridge-builders, communicators, mono-, multi-, or interdisciplinary. We want to stimulate applicants to show their particular qualities, and relate those to the proposed project.
The revised format and DORA
Like DORA, we recognize the need to improve the ways in which the outputs of scholarly research are evaluated. With this in mind we started the design of a new CV format in 2017. We reviewed internal data, conducted research on assessing academic qualities, looked at examples, and consulted applicants, committee members, and university policy officers. We tested a first design in 2018 and then refined it. This process resulted in a relatively simple narrative CV – that is being tested in the current Veni round – with only two categories: Academic profile and Key output.
In the Academic profile the applicants highlight their research focus and vision. They describe their position in their (inter)national academic field, their motivation for conducting research and, for the project in particular, the academic and societal potential of their work.
The Key output section allows a maximum of ten output items. The applicant explains the importance of each output, how it is related to the project, and/or how it shows the applicant’s abilities. Diverse types of output are allowed, ranging from journal articles and books, data and software, to exhibitions and performances. Preprints are also allowed. Special attention is paid to contributions to open science; candidates are required to indicate which outputs are openly available.
Responsible use of metrics
The use of metrics is restricted, to avoid narrow assessments that merely look at proxies of academic impact, but disregard quality, originality, validity, sentiment, societal impact, etc. Only indicators that refer to one of the mentioned Key outputs are allowed. The applicants are asked to provide context: Why is the chosen indicator a good quality measure? What does it imply?
We have banned what we consider unreliable metrics. We exclude the use of journal impact factors and any kind of metric that refers to journal, publisher, or publication platform, and also ban (H-)indexes, citation averages, totals, and sums, as these measures are sensitive to bias. We know, for instance, that publication lists are often longer and total citation numbers are higher for male than female scholars. H-indexes can be gender biased and inflated by self-citation.
In both sections the focus is on quality over quantity. Limiting the number of outputs gives committee members the opportunity to look at the output itself instead of simply counting numbers. The motivation provided by the applicants creates possibilities to explain the importance of the work, also if their career path or output is atypical.
Both sections are assessed with the formal time applicants have had in mind for research. We ask applicants to calculate the formal time spent on research since obtaining their PhD. If an applicant has a full-time research appointment, more indications of (research) quality are expected than from an applicant with a similar PhD date, but an 80 percent teaching position.
Making a difficult assessment
The new setup does come with its dilemmas. The adjustments require a major cultural change, as we aim to get rid of long-standing common practices in academic evaluation. Such big steps take some getting used to. They necessitate a coherent explanation to all people involved of the reasoning behind the changes made.
Also, we recognize that the committee members have a more complicated job than they had before. Comparison of narratives is inherently more difficult than determining that twelve articles is more than eleven. We try to instruct the committee the best we can, but the success of this approach is mainly due to the members’ knowledge as well as their openness towards diverse approaches, profiles, and career paths. Thus, we owe a great debt of gratitude to those committee members who recognise the urgency to change the recognition and rewards system, make an effort to really focus on quality, and provide us with valuable feedback.
Creating the right incentives
The assessment of the CVs relies on argumentation of why certain aspects convincingly show particular qualities in certain circumstances, and why others do not. By conveying these considerations to the applicants they can actually see that various types of (explained) qualities are recognised and rewarded. This way we hope to do away with the paradigm of publish or perish. We aim to foster inclusive and substantiated selection, to stimulate aiming for high quality and merit over large quantity, and content over reputation.
For more information, see the CV-format in the Veni application form.
Guest blog posts reflect the opinions of the authors and not necessarily those of DORA.