By Stephen Curry (Imperial College London and DORA)
As declarations go, the San Francisco Declaration on Research Assessment (DORA) is relatively modest in its aspirations. For universities and research institutes, DORA’s main ask is to enhance research evaluation by focusing on the merits of individual papers and other valuable outputs (such as data, software, and training), thereby avoiding undue reliance on journal impact factors (JIFs).
Even so, it is much easier to sign DORA than to deliver on the commitment that signing entails. And while I would always recommend that universities sign as soon as they are ready to commit, because doing so sends such a positive message to their researchers, they should not put pen to paper without a clear idea of how signing will impact their approach to research assessment, or how they are going to develop any changes with their staff.
The precise path taken to implementing DORA will depend on the history and organisational idiosyncrasies of each institution. Nevertheless, it is likely that the establishment of an internal working group or committee to consider how best to infuse the spirit of the declaration within the institution will be a sensible move in most cases. This is the approach we took at my university, Imperial College London, after we signed DORA in January 2017.
Some of the groundwork for signing had been laid a couple of years earlier during an internal review of Imperial’s use of performance metrics. This review consulted widely – around 23% or academic staff responded with comments to a survey asking for their views on performance evaluation. Its recommendations, which took account of these views and drew on the principles of responsible metrics espoused by the Metric Tide report, established the important concept of ‘performance profiles.’ These profiles were explicitly constructed to recognise and reward academic contributions across a broad range of activities, including research, teaching and mentorship, departmental citizenship, and creativity.
While the development of performance profiles was very much aligned with the spirit of DORA, when Imperial signed the declaration a year or so later there was still work to do to ensure that our research evaluation procedures were compliant. Immediately after signing we therefore set up a working group composed of experienced academics from each of our four faculties (Natural Sciences, Engineer, Medicine, and Business) and senior staff from Human Resources – nine people in all (see report for details). The group was kept fairly tight to help focus discussions, though drew on the wide range of views collected as part of the earlier internal review. Its terms of reference were stated fairly simply:
- To examine the implications of DORA for the College’s recruitment and promotion policies and procedures, and for its submission to the next REF (Research Excellence Framework).
- To make recommendations on how the principles expressed in DORA can be embedded in the College’s culture and working practices.
In effect the group was charged with thinking through the details of what signing DORA would mean for all the key points when research or researchers are evaluated at the university. These include recruitment and promotion, annual appraisals, internal funding awards, and selection of outputs to be submitted to the UK’s national research assessment exercise, the REF. It met half-a-dozen times over a period of about nine months to complete its work and come up with recommendations, which were submitted to the Vice Provost’s Advisory Board for sign-off at the College level.
The main tasks involved were relatively technical and mostly involved reviewing processes and the phrasing used in internal documentation. Out went phrases such as “contributions to research papers that appear in high-impact journals” to be replaced by “contributions to high quality and impactful research.” The change is subtle but significant – the revised guidance makes it plain that ‘impactful research’ in this context is not a cypher for the JIF; rather it is work “that makes a significant contribution to the field and/or has impact beyond the immediate field of research.”
Other technical changes brought in a clear statement that is to be included in all advertisements for research or academic positions:
“The College is a proud signatory to the San Francisco Declaration on Research Assessment (DORA), which means that in hiring and promotion decisions we will evaluate applicants on the quality of their work, not the impact factor of the journal where it is published. More information is available at https://www.imperial.ac.uk/research-and-innovation/about-imperial-research/research-evaluation/.”
The working group also strongly recommended the adoption of a more narrative-based approach for promotion applications, which asks candidates to identify what they consider to be their four most important research papers and explain why they are so significant.
The report and recommendations of the DORA working group at Imperial, which were adopted in full in late 2017, were transmitted through staff briefings and published on the College’s dedicated Research Evaluation web-page.
Of course, it is one thing to come up with new policies and procedures, quite another to ensure that they take root within the university. The working group also had to address the challenge of shifting people away from embedded habits, a less technical but more difficult task. In part, that was done by placing renewed emphasis on more narrative than quantitative approaches that focus the attention of evaluators on people’s work, not their venue of publication. But there was also a concerted effort to engage the community in dialogue about DORA. To that end, a key recommendation was to hold a workshop to which the entire research community was invited.
This workshop, titled “Mapping the future of research assessment at Imperial,” was held in September 2018 and brought together researchers at all levels from across the university. To maximise its reach and to send a clear signal that Imperial is serious about reforming its research evaluation practices, the event was advertised internally and externally. The proceedings were also live-streamed and recorded – you can see the video on Imperial’s YouTube channel with presentations by Nick Jennings, Vice-Provost (Research and Enterprise), Stephen Curry, Chair of the DORA Steering Committee, Lizzie Gadd, Research Policy Manager at Loughborough University, and the Panel discussion chaired by Chris Jackson, Professor of Basin Analysis.
Even then, that is not the end of the matter. Like all universities, Imperial is a large and complex organisation, and real change will take time and regular repetition of the message that the institution is committed to DORA. While the workshop engendered good engagement from attendees with the thorny issues of research evaluation, which go well beyond Imperial, it obviously did not capture the attention of everyone who works here.
The work of the DORA working group at Imperial has concluded, but the effort needed to fully embed our commitment to the declaration has to go on. The effort is ably assisted by the strong advocacy of the Vice Provost for Research and the Director of Library Services, among others, along with vigilance, perhaps surprisingly, from the university’s Bibliometrics and Indicators team.
I doubt that everyone at Imperial yet knows that we have signed DORA, or that in every committee room across campus, our practices are entirely compliant with the declaration. But we have made a good start. This is a journey that necessarily involves many stakeholders: academics, universities, funders, publishers, learned societies – and DORA. I look forward to learning more about how institutions can make practical progress on reforming research evaluation at the DORA/HHMI meeting in October.