TARA Methodology

< Back to Dashboard

The Responsible Research Assessment Tracker (RRA-Tracker) is an online dashboard intended to track responsible research assessment for faculty hiring, promotion, and tenure at research performing organizations around the world. To be included in the dashboard, source materials must outline or report activities relevant to Responsible Research Assessment in the context of individual forms of academic evaluation, namely faculty hiring, promotion, tenure and/or retention. Responsible Research Assessment (RRA) is an emerging umbrella term, defined by the GRC RoRi (2020) report as: “approaches to assessment which incentivize, reflect, and reward the plural characteristics of high-quality research, in support of diverse and inclusive research cultures.” 

Based on input and feedback from two DORA community events in 2021 and on the iterative sorting of source materials, the following overarching categories have been identified as constituent parts of RRA reform initiatives: open research, research integrity, academic contributions, responsible metrics, equity and inclusion, and portfolio assessment. Definitions for these and all other tags used to classify source materials can be found here.

Source materials indexed within RRA-Tracker describe one or more of the following type(s) of intervention a research performing organization has pursued in relation to the implementation of RRA: DORA signatures, announcements from organizations subscribing to RRA including those that formally sign DORA, action plans, responsible research assessment principles, practices, policies, and outcomes. 

Potential source materials for the RRA-Tracker dashboard were identified through a combination of web-searches; consulting existing web resources which have compiled lists of research performing organizations engaged in RRA-related activities (e.g. Bibliomagician blogOpen Science FoundationHong Kong Principles); and the generous crowd-sourcing from participants at two DORA community events in 2021. Potential source materials were read through and reviewed to determine eligibility by the TARA team, and selected entries have been categorized to monitor trends and track progress for RRA at research performing organizations. In determining eligibility for inclusion, a research performing organization has to have generated online source materials which complies with ALL of the following criteria:

  1. Source materials describing research assessment activities have to be publicly available on the Internet.
  2. Source materials have to give our team a clear and intelligible enough account of what a research performing organization has done to advance RRA[1].
  3. Source materials have to describe research assessment reform activities limited to individual academic faculty assessment procedures (hiring, promotion, tenure, retention for full-, associate-, assistant-, and/or adjunct-professors, or equivalent e.g. reader, lecturer).
  4. Source materials have to display research assessment reform activities aiming to incorporate one or more values the Project TARA team have associated with the term Responsible Research Assessment (see above). The TARA team recognizes that Responsible Research Assessment is a dynamic and changing term, and that the current list of values may still evolve with practices. As such, we will regularly revisit these values to ensure they are up-to-date with the DORA community’s understanding of Responsible Research Assessment, and we expect them to evolve further as new source materials are uncovered and added. 

Common grounds for excluding potential source materials gathered by the team in online searches, or suggested to the team via DORA community events or submitted via the RRA-Tracker website, include:

  1. Potential source materials are not publicly available on the Internet (e.g. behind an intranet).
  2. Potential source materials are not intelligible enough for the project team to determine with any confidence the activities a research performing organization had undertaken.
  3. Potential source materials describe reforms to processes in the context of funding proposal or institution-level assessments. While highly relevant to the RRA movement, reforms to these activities are outside of the initial scope of this dashboard.
  4. Potential source materials describe changes to individual assessment procedures that do not affect faculty populations – instead affecting populations like post-doctoral researchers and graduate students. Assessment reforms aimed towards non-faculty staff are expected to be incorporated in future iterations of the dashboard.

Although we follow a systematic process for classifying source materials, it requires interpretation and judgment on the part of the project team. Inevitably it will reflect the extent of the TARA team’s knowledge. We are constantly seeking to improve the quality of the dashboard, and if you have any comments or spot any errors in the tagging of a piece of source material, please contact info@sfdora.org

We are continuously searching for new source materials to populate RRA-Tracker relating to innovative hiring, promotion, retention, and tenure assessment processes. If you have suggestions of source materials which you think would be suitable for inclusion, please get in touch: info@sfdora.org.

[1] Due to resource constraints, this version of the Dashboard collected source materials written or translated into the English language.