Each year, DORA reflects on progress toward responsible research assessment. How is research valued in different communities and how might that have changed in 2023? What tools are the community creating to support policy development? What types of research assessment policies are being developed to reduce the influence of journal-based metrics and recognize a broad range of contributions? How are communities coming together to improve practice and support culture change?
The following list of new developments were created by DORA staff. This list is not exhaustive and we might have missed something in the process of our search for new developments. Please let us know what other advances we should consider adding to the “From elsewhere” section or to the DORA resource library (email info@sfdora.org).
In 2023, there were several new developments in DORA’s leadership team. In July, DORA welcomed new Program Director Zen Faulkes and Acting Director Haley Hazlett returned to her role as Program Manager. Zen shared more about his background and passion for responsible research assessment in Words from DORA’s New Program Director. In November, DORA bid a fond farewell to Stephen Curry, whose six-year term as Chair came to an end. DORA also welcomed a new leadership team in November: Ginny Barbour of Queensland University of Technology and Kelly Cobey of the University of Ottawa Heart Institute became joint Chairs of DORA, and Rebecca Lawrence of F1000 became the new Vice-Chair of DORA.
New Developments
From DORA
- In February, DORA’s Community Engagement Grant Awardees shared the results of their work in a series of blog posts. DORA piloted the program in 2022 to directly support a broad range of ideas to promote assessment reform at academic institutions. Teams from around the world were awarded funds to support projects to promote assessment reform at academic institutions. Their work ranged from conducting research on assessment practices, developing guidance and training, holding workshops and events, and raising awareness.
- In March, DORA announced a new 3-year strategic plan. The plan was finalized after extensive consultation with the broader scholarly community and the DORA Steering Committee. The plan includes four new strategic objectives:
- Increase awareness of the negative impacts of research assessment practices that are too dependent on inappropriate metrics, and of the positive impacts of alternative practices
- Accelerate the development of clear and concrete measures to reform research assessment
- Support advocates of research assessment reform worldwide
- Secure the funding needed to deliver DORA’s mission as efficiently and as rapidly as possible
- In May, DORA celebrated its 10th Anniversary by featuring a decentralized program of 21 local events organized by advocates of reform from 15 countries. Community members the world over organized their own events to examine key topics in research assessment most meaningful to them, from locally relevant issues to publishing models to the future of research assessment and more. In addition to the locally organized program of events, DORA hosted two plenary sessions on May 15 to support global participation in the celebration. The plenaries saw over 300 attendees from over 50 countries and focused both on the history of DORA and, critically, the future of research assessment. In addition to opening remarks from DORA Chair Stephen Curry and DORA Vice-Chair Ginny Barbour, the plenaries featured keynote addresses from Sarah de Rijcke (CWTS Leiden) and Mai Har Sham (The Chinese University of Hong Kong) and panel sessions from experts around the globe. Additionally, the work of DORA’s Community Engagement Grant Awardees was shared during the plenary. Check out the blogs for the Asia-Pacific plenary and the Africa, Americas, Europe plenary for reading lists inspired by the presentations, quotes, and key takeaways from the meetings.
- Tools to Advance Research Assessment (TARA) is a project to facilitate the development of new policies and practices for academic career assessment supported by Arcadia, a charitable foundation that works to protect nature, preserve cultural heritage and promote open access to knowledge.. In May 2023, TARA team members Alex Rushforth (CWTS Leiden) and Sarah de Rijcke shared the results of their qualitative study of faculty hiring, promotion, and tenure practices in the United States. This work was also a feature of Sarah de Rijcke’s keynote address for the DORA 10th Anniversary Africa, Americas, Europe plenary session. Project TARA also aims to create a series of tools to support community advocacy for responsible research assessment policies and practices. In March, DORA released a summary of a public event introducing the first two tools, Debiasing Committee Composition and Deliberative Processes and Building Blocks for Impact. These tools were created by TARA team member Ruth Schmidt (Institute of Design at the Illinois Institute of Technology).
- “Reimagining Academic Career Assessment: Stories of innovation and change” is a collection of case studies from universities and national consortia examining their work to develop new policies and practices. The case study repository was published at the end of 2020 by DORA, the European University Association, and SPARC Europe. DORA continues to grow the repository and in August the University of Zurich was included. The case study for University of Zurich focuses on its HI-FRAME project, which worked to align the hiring practices of the University with open science practices.
- In September and October, DORA organized a series of symposia and workshops for its Research Funder Discussion Groups in partnership with the MoreBrains Cooperative and the Elizabeth Blackwell Institute of Bristol University that focused on identifying concrete steps to reduce the barriers for funding applications and increase transparency in funding calls.
- The University of Tokyo and the Queensland University of Technology (QUT) became DORA signatories, a further step in the Asia-Pacific region towards the implementation of responsible assessment reform.
From elsewhere
- Many of the local event organizers who participated in DORA’s 10th Anniversary Celebration drafted summaries of the key takeaways from their events. These summary blogs, some of which are available in both English and Spanish, cover topics that were important and relevant to the organizers, including university rankings, the intersection of open research and research assessment and more.
- The Agreement on Research Assessment reform and the resulting Coalition for Advancing Research Assessment (CoARA) exceeded 500 member organizations, assembled its members for a General Assembly, and released its first calls for Working Groups and National Chapters to develop outputs that will support CoARA members as they work to implement the commitments agreed upon when joining the Coalition.
- The Global Young Academy (GYA), the InterAcademy Partnership (IAP) and the International Science Council (ISC) organized The Future of Research Evaluation: A Synthesis of Current Debates and Developments. This report is a global perspective that captures a “broad cross-section of the research ecosystem whose diverse mandates can facilitate genuine systemic change”. The report was authored by experts from CWTS Leiden, FOLEC, NSFC, University of Malaya, Higher Education Commission (Mauritius), University of Pretoria, FAO OED, University of Rome, and the University of Melbourne.
- As funding organizations continue to adopt the use of more narrative-style CVs, there is a burgeoning need for research into how new CV formats influence review practices and funding outcomes. The Research on Research Institute launched its Narratives Project to “provide research funders and policymakers with evidence and analytical insights to facilitate the design and use of narrative CV formats”. UK Research and Innovation worked with funders and research institutions to develop guidance and infrastructure to support the use of narrative CVs.
- UNESCO released a report Open science outlook 1: status and trends around the world. The report identifies a primary hurdle to the progress of open science is the alignment of assessment practices used to evaluate scholars and institutions for funding and career decisions. To support the alignment of values in research assessment, UNESCO used case studies of initiatives, like DORA, FOLEC, CoARA and the GYA, that are working internationally to establish more responsible research assessment practices.
- University rankings were a major point of focus for many in the responsible research assessment space, catalyzed by the creation of the INORMS More Than Our Rank (MTOR) initiative. Aggregate and overly simplified measures of quality (which often are calculated opaquely and using other proxy measures like h-index) can reduce the broad range of an institution’s research activities down to a single number. MTOR encourages academic institutions to recognize the limitations of university rankings and to post statements of their “activities, achievements or ambitions that are not adequately captured by national or international university rankings.”
- The Latin American Forum for Research Assessment (FOLEC-CLACSO) released a new report with Fundación Carolina analyzing different recommendations and regulations on Open Science in Ibero-America. The study analyzes how to strengthen open infrastructures in interaction with their communities and makes recommendations to improve scientific information in open repositories. Additionally, FOLEC organized the international conference The evaluation of research in Social Sciences and Humanities under debate: dimensions, methodologies, practices and challenges. In addition to its role co-organizing the Global Summit on Diamond Open Access with Redalyc, UAEMéx, AmeliCA, UNESCO, UÓR, ANR, cOAlition S, OPERAS and Science Europe, FOLEC collaborated on Manifesto on Science as Global Public Good: Noncommercial Open Access. FOLEC collaborated with the National Scientific and Technical Research Council (CONICET) of Argentina and the National Agency of Research and Development (ANID) of Chile to organize Transformations in academic communication and the impact on evaluation systems at the 2023 Americas Regional Meeting of the Global Research Council (GRC).
- The Dutch Recognition and Rewards Programme held its third Recognition and Rewards Festival and released its first e-magazine, which features a range of concrete examples of responsible assessment practices. The Programme released a summary of a discussion on university rankings and how they fit with the mission of recognition and rewards. The Programme additionally released a roadmap of concrete steps that it will take to catalyze responsible assessment practices in the Netherlands. Additionally, several universities in the Netherlands implemented new career profiles to better recognize and reward the range of faculty expertise and career paths, including: Maastricht University’s Academic Profile for Assistant, Associate and Full Professors; the University Medical Center Utrecht’s career profiles for researchers and career framework; and Vrije Universiteit Amsterdam’s career paths (currently unavailable).
- The United States White House released a Report to Congress on Financing Mechanisms For Open Access Publishing of Federally Funded Research. Section 6.3 of the report includes findings from a series of listening sessions with over 1,000 early career researchers about open access publishing. Participants felt that the pervasive culture of “publish or perish” in high-impact journals was a barrier to adopting open access practices, and that moving away from the focus on journal impact factor could help with acceptance. DORA was cited in this report as an initiative working to move the academic community away from an outsized focus on inappropriate quality metrics.
- SPACE is a tool developed to facilitate academic assessment reform at universities by examining what infrastructure is needed to support the development of new policies and practices. In 2022, DORA worked with Ruth Schmidt collaborated with CINDA to organize a workshop on the practical use of SPACE. Building in part off of this workshop, CINDA began a project among its member universities to pilot the implementation of SPACE for responsible research assessment reform, sharing a work in progress report in December 2023.
What is in store for 2024?
DORA has quite a bit planned for 2024 and the below list is not exhaustive, but there are several exciting projects that will be released in 2024 that we are particularly keen to share:
- Project TARA
- Reformscape will be released in January 2024. Reformscape is an interactive database that contains examples of responsible research assessment practices and processes from academic institutions around the world. This will be a community fed database for researchers, academic policymakers and administrators who are seeking examples and resources to support their efforts to reform assessment practices at their institutions.
- Made available via preprint in 2023, Practicing Responsible Research Assessment: Qualitative study of Faculty Hiring, Promotion, and Tenure Assessments in the United States will be published in 2024.
- After releasing Debiasing Committee Composition and Deliberative Processes and Building Blocks for Impact in late 2022, a new set of tools is currently under development and slated for release in 2024.
- DORA is developing a new resource that will clarify the organization’s position on a range of metrics and success indicators, and how signatories can make informed decisions on whether and how to evaluate different metrics.
Haley Hazlett is DORA’s Program Manager