Reformscape methodology

Reformscape is an online tool to track responsible research assessment for faculty hiring, promotion, and tenure at research performing organizations around the world.

To be included in Reformscape, source materials must outline or report activities relevant to responsible research assessment in the context of individual forms of academic evaluation, namely faculty hiring, promotion, tenure and/or retention. Responsible research assessment is an emerging umbrella term, defined by a 2020 Global Research Council Research on Research Institute (GRC RoRi) report as: “approaches to assessment which incentivize, reflect, and reward the plural characteristics of high-quality research, in support of diverse and inclusive research cultures.”

Based on input and feedback from two DORA community events in 2021 and on the iterative sorting of source materials, the following overarching categories have been identified as constituent parts of responsible research assessment reform initiatives: open research, research integrity, academic contributions, responsible metrics, equity and inclusion, and portfolio assessment. Definitions for these and all other tags used to classify source materials can be found in the Reformscape glossary.

Potential source materials for Reformscape were identified through a combination of web-searches; consulting existing web resources which have compiled lists of research performing organizations engaged in activities related to responsible research assessment (e.g. Bibliomagician blog; Open Science Foundation; Hong Kong Principles); and the generous crowd-sourcing from participants at two DORA community events in 2021. Potential source materials were read through and reviewed to determine eligibility by the TARA team, and selected entries have been categorized to monitor trends and track progress for responsible research assessment at research performing organizations. In determining eligibility for inclusion, a research performing organization has to have generated online source materials which complies with all of the following criteria:

1. 1. Source materials describing research assessment activities have to be publicly available on the internet.
   2. Source materials have to give our team a clear and intelligible enough account of what a research performing organization has done to advance responsible research assessment.^[1]
   3. Source materials have to describe research assessment reform activities limited to individual academic faculty assessment procedures (hiring, promotion, tenure, retention for full-, associate-, assistant-, and/or adjunct-professors, or equivalent e.g. reader, lecturer).
   4. Source materials have to display research assessment reform activities aiming to incorporate one or more values the TARA team have associated with the term “responsible research assessment” (see above). The TARA team recognizes that responsible research assessment is a dynamic and changing term, and that the current list of values may still evolve with practices. As such, we will regularly revisit these values to ensure they are up-to-date with the DORA community’s understanding of responsible research assessment, and we expect them to evolve further as new source materials are uncovered and added.

Common grounds for excluding potential source materials gathered by the team in online searches, or suggested to the team via DORA community events or submitted via Reformscape, include:

1. 1. Potential source materials are not publicly available on the internet (e.g. behind an intranet).
   2. Potential source materials are not intelligible enough for the project team to determine with any confidence the activities a research performing organization had undertaken.
   3. Potential source materials describe reforms to processes in the context of funding proposal or institution-level assessments. While highly relevant to the responsible research assessment movement, reforms to these activities are outside of the initial scope of this tool.
   4. Potential source materials describe changes to individual assessment procedures that do not affect faculty populations – instead affecting populations like post-doctoral researchers and graduate students. Assessment reforms aimed towards non-faculty staff are expected to be incorporated in future iterations of the dashboard.

Although we follow a systematic process for classifying source materials, it requires interpretation and judgment on the part of the project team. Inevitably it will reflect the extent of the TARA team’s knowledge. We are constantly seeking to improve the quality of the dashboard, and if you have any comments or spot any errors in the tagging of a piece of source material, please contact reformscape@sfdora.org.

We are continuously searching for new source materials relating to innovative hiring, promotion, retention, and tenure assessment processes to add to Reformscape. If you have suggestions of source materials which you think would be suitable for inclusion, please submit them via this “Suggest a resource” form or by emailing reformscape@sfdora.org.

^[1] Due to resource constraints, this version of Reformscape collected source materials written in English or translated to English.