This is part of DORA’s toolkit of resources to support academic institutions that are improving their policies and practices. Find the other resources in the toolkit here. With thanks to volunteers in the DORA community, SPACE is also available in Catalan and Spanish. See below.
Improving research and scholarship assessment practices requires the ability to analyze the outcomes of efforts and interventions. However, when conducted only at the unit level of individual interventions, these evaluations and reflections miss opportunities to understand how institutional conditions themselves set the table for the success of new efforts, or how developing institutional capabilities might improve the effectiveness and impact of these new practices at greater scale. The SPACE rubric was developed to help institutions at any stage of academic assessment reform gauge their institutional ability to support interventions and set them up for success.
Organizations can use the SPACE rubric to support the implementation of fair and responsible academic career assessment practices in two ways: First, it can help establish a baseline for the current state of infrastructural conditions, to gauge an institution’s ability to support the development and implementation of new academic assessment practices and activities. Second, the rubric can be used to retroactively analyze how strengths or gaps in these institutional conditions may have impacted the outcomes of concrete interventions targeted to specific types of academic assessment activities—such as hiring, promotion, tenure, or even graduate student evaluation—either helping or hindering progress toward those goals.
The SPACE rubric is a result of DORA’s partnership with Ruth Schmidt, Associate Professor at the Institute of Design of the Illinois Institute of Technology, who led the iterative participatory design process. The creation of the rubric was informed by nearly 75 individuals in 26 countries and 6 continents, and benefited from multiple rounds of feedback.
Recommendations for use:
- Identify who to involve. The SPACE rubric can be used by an individual or as an exercise for a group of people, such as a research assessment task force or a departmental hiring committee, etc.
- Decide on the “level” of analysis. The rubric can be applied to different organizational levels, such as a department, school (i.e. Medical School or Law School), or an entire academic institution (university). Choose the largest unit level that you feel has direct and substantial influence on the kinds of assessment practices you are interested in learning about.
- Decide how you would like to use the rubric: a) to establish a baseline for institutional “maturity” or b) to retroactively analyze how institutional conditions may have impacted the outcomes of a specific intervention, such as hiring.
- To establish a baseline for institutional “maturity”:
Work through each of the five SPACE categories (Standards for scholarship; Process mechanics and policies; Accountability; Culture within institutions; Evaluation and iterative feedback) to capture the current state of organizational capability for the unit level you just identified. To help users collect and organize information, we provide a blank rubric that can be used as a worksheet (Download worksheet Word document). Consider things like:- Types of activity: What types of institutional activities to support the development of new research assessment practices are occurring, from building foundational capabilities, to expanding on them, to scaling them?
- Coverage: Where is there a lot of activity, and where are there organizational “blind spots”?
- Availability and use of resources: What resources (people, money, time) are used to support these institutional capabilities, and where are those resources being underutilized or underinvested?
- Tensions: Where do you see conflicts between stated organizational goals and what capabilities are actually in place to support those goals?
- To retroactively analyze how institutional conditions may have impacted the outcomes of a specific assessment intervention:
First, identify the specific academic assessment intervention or activity you are interested in exploring further. Then, work through each of the five SPACE categories (Standards for scholarship; Process mechanics and policies; Accountability; Culture within institutions; Evaluation and iterative feedback) to capture how institutional conditions supported or hindered the development, implementation, and success of the specific intervention you are exploring. To help users collect and organize information, we provide a blank rubric that can be used as a work sheet (Download worksheet Word document). Consider things like:- Strengths: What kind of capabilities helped this intervention succeed, or provided the right conditions for it to flourish?
- Gaps: Where did a lack of organizational capability contribute to challenges in getting buy-in, support, or execution?
- Investment: Where do you think additional investment or attention (of people, resources, time, or money) may have made a significant difference? This could either be developing new capacities where they are currently absent or bolstering ones that already exist by making them more robust or better integrated.
- Identify next steps to improve academic career assessment based on the information captured in steps 4 and/or 5.
Important considerations
- Using the rubric is not an exercise in completeness, and you should not feel obligated to feel out every square! Blank spaces may indicate where future work is needed.
- The “This might look like” content within the rubric merely as representative examples of what one might see, not as definitive criteria for how to define success. On the second page of the rubric, for example, we indicate some illustrative starting points and potential areas of focus for institutions to consider, depending on their current level of research assessment reform. Additionally, we have published an accompanying article that outlines examples of use based on a small pilot of the rubric (here).
Please reach out to us (info@sfdora.org) if we can answer any questions about using the rubric. DORA is eager to support organizations interested in using the rubric to advance fair and responsible academic assessment.
Catalan and Spanish translations
Catalan translations of the rubric, the blank rubric worksheet, and the DORA resource library rubric overview were contributed by Paula Adam, Esteve Arboix, Marta Aymerich, Ignasi Labastida, Ciro Llueca, Pastora Martínez-Samper, and Julie Wilson. Spanish translations of the rubric, the blank rubric worksheet, and the DORA resource library rubric overview were contributed by Soledad Aravena, Marta Aymerich Martínez, Ciro Llueca, Pastora Martínez-Samper, Josmel Pacheco-Mendoza, Alejandra Tejada-Gómez, and Julie Wilson. The translations are made available under the terms of the Creative Commons Attribution International License. We are very grateful to the volunteers who have produced and checked the translations. The Spanish and Catalan translations can be found here.