As a graduate student, I signed DORA to speak out against the misuse of the impact factor. Even with my career before me, I knew that something about the way research was being evaluated in hiring, promotion, and funding decisions needed to change. It did not make any sense that the value of a researcher is largely based on the journal where their work is published, and yet this was the system that I, and my fellow PhD cohort, felt we were being forced into. I had wanted to be evaluated based on the quality of my research, how it was re-used by my colleagues around the world, and how others shared and discussed it online.
The local community at Imperial College, where I completed my PhD work, contained a vibrant group of forward-thinking individuals, who were constantly engaging with other researchers and the wider community around London about topics to do with Open Science and assessment. Through this group, I realized that I was not alone in my concerns, and that others were working to create a better system around research evaluation. Thanks to the financial support being offered towards Open Access at Imperial, I was in a fairly unique position that allowed me to both fight the present system while also being granted the liberty to play the ‘publish or perish’ game. I was in a privileged position where I could publish much of my research in high-impact-factor Open Access journals and pay the high fees for Open Access that my colleagues at other institutions were often unable to afford. This model, however, is not easily translated to other institutions, or countries, and still relies on journal-based metrics for controlling where research is published and how it is assessed.
Speaking out is necessary to create a more rigorous, reliable, and equitable system of research evaluation. A few years on, the message seems to be percolating through many levels of the academic ecosystem. As of September 2018, DORA has been signed by more than 12,000 individuals and 500 organisations from around the world, demonstrating a strong collective commitment to a fairer research evaluation process.
DORA is undoubtedly making progress, and is now manifesting into real change in the research ecosystem. Progress may seem slow, but in a system defined by inertia, and accompanied by a complex web of ‘stakeholder’ interactions and power dynamics, any amount of change is positive. Preliminary results from research conducted by the Scholarly Communications Lab at Simon Fraser University in Vancouver, Canada, shows that the impact factor is only mentioned explicitly in about 20% of review, promotion, and tenure documents in North American research institutes, much less than we might perhaps expect. This is a positive sign that steps towards a more fair, sustainable, and equitable system are being made.
The declaration is a statement of intent, and signatories are not required to be fully compliant before they add their names. Signing is easy, but creating real lasting change in the community is significantly more challenging. However, the recent release of ‘Plan S’ in Europe could catalyse widespread change.
Plan S has one primary principle: “After 1 January 2020 scientific publications on the results from research funded by public grants provided by national and European research councils and funding bodies, must be published in compliant Open Access Journals or on compliant Open Access Platforms,” as well as 10 key sub-principles. This is the sort of disruptive change needed to bring European Open Access publishing in line with other regions like Latin America.
One key aspect here is what Plan S says about incentives and metrics in research evaluation. Importantly, it recommends the support of the 18 recommendations in DORA, which is a strong move from such a powerful collective of funding bodies. In the context of publication choice, Plan S makes the following statement: “We also understand that researchers may be driven to do so by a misdirected reward system which puts emphasis on the wrong indicators (e.g. journal impact factor). We therefore commit to fundamentally revise the incentive and reward system of science, using the San Francisco Declaration on Research Assessment (DORA) as a starting point.”
This is a significant announcement, reflecting years of anguish aimed at the impact factor. Now, those behind Plan S have the opportunity to empower the individuals in charge of research assessment and hiring, tenure, and promotion decisions to actively implement DORA principles, and cultivate a real culture of fairer assessment.
Plan S and DORA together provide fertile ground for dispelling the myth that impact factors or journal brands have anything to do with research quality, and the potential ramifications for this go beyond just Europe. A combined approach of bottom-up, grassroots campaigns supported by organisations like DORA, coupled with high-level policy action at the international level is essential. Only then can we break out of the current academic reward system and accommodate a better system of research evaluation.
Dr. Jon Tennant is a vagrant palaeontologist and the founder of paleorXiv and Open Science MOOC.
Guest blog posts reflect the opinions of the authors and not necessarily those of DORA.