In many ways, 2018 was a groundbreaking year for the San Francisco Declaration on Research Assessment (DORA). It marked the fifth anniversary of the declaration’s release, and we were re-invigorated with a newly formed steering committee, chaired by Prof. Stephen Curry, and a community manager, Dr. Anna Hatch, all determined to effect real change in the scholarly community. Nine organizations pledged financial and in-kind support for DORA, which enabled us to begin the challenging work of transitioning the declaration from a statement of intent into a tool for meaningful policy change in hiring, promotion, and funding decisions.
In June, we announced a two-year Roadmap outlining three strategic goals to guide our vision of advancing practical and robust approaches to research assessment:
- Build awareness of the issues,
- Catalyze reform and improved behavior, and
- Extend the disciplinary and geographic reach of DORA.
This recap summarizes the various strands of our activity in the last year and the progress we have made.
Elevating our web presence
One of the first things we did was release a new website (sfdora.org) for DORA in February. While this certainly was not the most important thing that we did all year, it provides a central space for many of our activities and resources. In addition to the webpages where users can read the declaration and sign it, the website also contains materials, including examples of good practices and other resources, to help individuals and organizations create new policy for hiring, promotion, tenure, and funding decisions. Another important feature on the website is the blog, where we publish posts related to advances in research assessment. We capitalize on social media, in particular Twitter, to engage with members of the scholarly community about research assessment reform.
Building a community of practice
Research assessment reform requires cultural and behavioral change from both individuals and institutions. To do this, we are building a strong community of supporters, both to initiate local change and share ideas. We have also updated the the confirmation email that new signers receive to provide a message of welcome along with ideas of how to pioneer change at their organization. The email includes links to the Roadmap as well as badges and slides that we specifically created to help signers voice support for research assessment reform. A number of individuals and organizations have added badges to their websites or shared them on social media. The slide decks come in three different forms—a one-slide summary as well as short and long versions explaining what DORA is about. They were designed to seed conversations about research assessment depending on how much time people have available. We also established regular email contact with our supporters and news subscribers sharing updates and announcements about DORA activities.
Catalyzing reform
To celebrate the fifth anniversary of DORA’s release, we launched a community interview series to explore how supporters are introducing new policies and procedures related to research evaluation within their organizations. It provides an opportunity to discuss innovation in research assessment and a space for individuals and groups to connect. The interviews take place online and are free to attend. Summaries are published on the blog with an audio recording of the discussion. Our first interview was with Sandra Schmid (UT Southwestern) in May. We have also interviewed Shahid Jameel (Wellcome Trust/DBT India Alliance), Philip Campbell (Springer-Nature), and Christopher Jackson (Imperial College). The interview series will continue in 2019.
At the ASCBEMBO Meeting in December, we hosted a workshop on research assessment as part of the career enhancement programming. Participants worked in small groups to provide feedback on job and grant application materials. Several administrators from universities and funders attended. After the review period, there was a large group discussion of ways to improve applications. The feedback collected during the session is being captured in a summary that will be shared with the community as well as the organizations that allowed us to critique their application materials. We hope that providing organizations with this type of concrete peer feedback will translate into action.
We are continuing to collect examples of good practices and other resources to promote research assessment reform on the website. Individuals looking to creating change at their university can draw on the experiences of others as they develop new policies and approaches to assessment. Good practices and resources are regularly shared on Twitter to reach a broader audience.
The Royal Society is developing a Resume4Researchers that will function as tool to encourage recognition of a broad range of scholarly activities when hiring, promotion, or funding decisions are made. Providing effective alternative tools is key to moving away from journal-based performance indicators. DORA is working with the Royal Society to gather feedback on the Resume4Researchers and will help promote its release in 2019.
Extending the disciplinary and geographic reach of DORA
There are many shared challenges in research assessment around the world and these have to be tackled internationally. At the same time we appreciate that some obstacles are unique to certain areas and that solutions are not always one-size-fits-all. To help address both of these aspects of research assessment reform, in October 2018 we announced the formation of a new international advisory board, comprising members from six continents and 15 countries. The board provides strategic guidance to DORA and its members also act as local liaisons. Members represent different stakeholders in the assessment landscape, including researchers (at different levels of seniority), university administrators, publishers, librarians, and nonprofit groups.
As the Declaration was originally published in English, we launched an effort to translate it into other languages to increase our geographic reach. With the help of more than 25 volunteers, DORA has now been translated into 11 languages, including Arabic, Chinese, French, Hindi, and Spanish. We continue to recruit new translations and receive offers from volunteers.
Policy changes
The introduction of new policies related to open access during the year had a significant impact on raising awareness about DORA and the need for research assessment reform. Plan S, a new drive to open access pubication of publicly funded research, was announced on September 4, with support from national research funding organizations across Europe and both the European Commission and the European Research Council (ERC). In recognition of the key role that research assessment has to play in opening up scholarly communication, DORA is explicitly mentioned in the preamble to Plan S written by Marc Schlitz, President of Science Europe:
“We therefore commit to fundamentally revise the incentive and reward system of science, using the San Francisco Declaration on Reseach Assessment (DORA) as a starting point.”
At the end of September, Redalyc, an organization that indexes more than half a million research papers published by journals from 22 countries in Latin America as well as Spain and Portugal, mandated the 1,360 journals they index sign DORA and adhere to its principles. Arianna Becerril-García, executive director of Redalyc, described the motivation for their decision in a blog post published on the DORA website.
The Open Research Funders Group (ORFG) released guidance for other funders called, Incentivizing the sharing of research outputs through research assessment: a funder implementation blueprint. Signing DORA and committing to its principles is part of the blueprint’s first step. Included in Wellcome’s updated open access policy is a requirement that the organizations where funded researchers work commit publicly to the principle that it is the instrinsic merit of the work, not journal title or Impact Factor, that matters in decisions on hiring, promotion, and tenure.
All of these new initiatives recognize the key role that DORA has been playing in campaigning for the reform of research assessment. We aim to continue and intensify our work in the coming years.
Future plans
The Roadmap will continue to guide our strategy as we work to increase awareness, promote tools and processes, and extend the disciplinary and geographic reach of DORA in 2019. While we plan to continue our core activities to advance change—which include the community interview series, translation project, blogs on topics related to assessment, and examples of good practice—we will also be participating in and hosting events related to research assessment. Our steering committee chair, Prof. Stephen Curry, and community manager, Dr. Anna Hatch, organized a session for the AAAS 2019 Annual Meeting in February to examine bias in hiring, promotion, and funding decisions. In the coming months Prof. Curry will be speaking about DORA’s work in Italy, Norway, Austria, Switzerland, the Netherlands, and the UK. One of our advisory board members, Dr. Christian Gonzalez-Billaut, is organizing a workshop on research assessment in Chile this summer. In October, we are hosting a meeting to drive institutional change for research assessment reform in the United States with HHMI.
A major project for 2019 is the expansion of the good practices section on the website. Building on the momentum that has accrued through recent policy announcements (see above), we will document how organizations are going about implementing reform. The scholarly community will be able to use our collection of good practices to drive change in their institution or organization.