As Alison Mudditt described in her Scholarly Kitchen post last month, the path to reforming research assessment has been met with significant challenges. We agree with her that culture change is often a slow process. However, as DORA demonstrates, it is possible to identify tangible progress on the path to large-scale research assessment reform.
Working with research institutions to implement DORA
The Wellcome Trust has published draft guidance for the research organisations we fund on implementation of the core principles of DORA. It aims to help organisations develop and adopt meaningful changes to research assessment practices that instil the buy-in and trust of their staff, and to encourage them to be proactive in reporting their progress and sharing their learning.
2019 in review: DORA’s list of the top 10 advances in research assessment
As 2019 winds down, the DORA steering committee and advisory board wanted to highlight the ways research assessment reform has advanced in the last year. From new data on assessment policies to the development of new tools, the scholarly community is taking action to improve research assessment in concrete ways.
Quality over quantity: How the Dutch Research Council is giving researchers the opportunity to showcase diverse types of talent
The Dutch Research Council (NWO) is piloting a narrative CV format in the Veni scheme, its major funding instrument for early career researchers. The format advances showcasing diverse types of talent and encourages assessment of quality rather than quantity.
Rewarding robust and reproducible research: Experiences from the pilot phase of implementing research assessment reforms for hiring professors at BIH/Charité (MERIT-PROF)
In the fall of 2017, the Charité Universitätsmedizin Berlin introduced five additional items to the online application form for professorships. These new items are a) A narrative on the candidate’s overall scientific contributions, b) Statements on the impact of the candidate’s self-selected top 5 publications , c) A record of the candidate’s open science and reproducible research activities, d) Information on the candidate’s contribution to team science, e) Academic age.
The Myth of the Three-Legged Stool
New faculty members at most (if not all!) research universities are given the same spiel: “your success here, and eventual promotion and tenure, is built on a three-legged stool, with one leg being research, one being teaching, and one being service.” And shortly thereafter, well-meaning mentors and department heads will tell the newbie, sotto voce, “do as little service as you can get away with, preferably none, and the bar you have to meet for teaching is just, don’t suck!”
How a working group began the process of DORA implementation at Imperial College London
The precise path taken to implementing DORA will depend on the history and organisational idiosyncrasies of each institution. Nevertheless, it is likely that the establishment of an internal working group or committee to consider how best to infuse the spirit of the declaration within the institution will be a sensible move in most cases. This is the approach we took at my university, Imperial College London, after we signed DORA in January 2017.
What do preprints need to be more useful in evaluation?
When scientists publish a journal article, they are doing more than just disseminating their work: they’re attaching it to a journal title that will, rightly or wrongly, telegraph signals about its quality. Preprints help to unbundle communication from the other functions of journal publishing, and they allow evaluators—funders, hiring committees, and potential mentors—to read a candidate’s most recent work.
The impact of research assessment on diversity
Unfortunately, science is rife with examples where research assessment diminishes diversity. Hiring, promotion, and grant decisions are made with incomplete information that is also poorly predictive of success—the perfect conditions for bias to emerge.
Role of societies in helping improve research assessment
Scientific societies have a key role to play in changing and improving assessment of researchers. Many are key publishers of quality content and many of their journals are recognized as such without the burden of journal impact factors. They also play key roles in shaping the scientific culture of disciplines, including around ethics, authorship, and outreach, including in discussions at meetings and in career workshops.