Background
Research funding organizations are well positioned to be drivers of change towards more responsible research assessment practices. They are in a unique position to shape research culture, determining the allocation of resources, shaping research priorities, and influencing who receives funding and for what projects. Funders around the world are experimenting with formats and processes, like the narrative CV, that move beyond the use of journal impact factor or journal prestige to assess research quality. These new formats aim to better assess research on its own merits, address unconscious biases, and value a wider range of research outputs. The design and supporting processes that go into creating funding calls or programmes influence factors like: who has access to funding, applicant population, the formatting of proposal materials, the types of work outputs that applicants highlight, and how proposals are assessed by reviewers.
DORA’s Asia-Pacific (A-P) and Africa, Americas, Europe (AAE) funder discussion groups were created in 2020 to support communication between funding organizations about research assessment reform, and to accelerate the development of new policies and practices that lead to positive changes in research culture. During their September 2022 meetings, these groups participated in a mapping exercise to identify existing research assessment interventions and areas to align on for future work. One of the key themes that emerged from this exercise was the importance of considering all steps in the funding process as part of responsible research assessment. This included the processes that take place before research is submitted for funding (pre-award processes), such as the timing of proposal calls and deadlines, transparency and guidance, phrasing and language use, selection and training of reviewers, and planning for “evaluating the evaluators” or “evaluating the evaluation process”.
In January 2023, the Elizabeth Blackwell Institute for Health Research (EBI) at the University of Bristol, in collaboration with the MoreBrains Cooperative, organized a symposium of researchers to analyze how pre-award processes can present obstacles to researchers and unintentionally reinforce biases like the Matthew effect, in which resources flow to those who have them. The findings from this symposium, and from subsequent discussions with research funders, resulted in a report with general recommendations on how the scholarly community can support transparency and equality*, diversity, and inclusion in pre-award processes.
A natural next step for the EBI report was to identify which recommendations were most practically applicable. Given the interest of the DORA research funders discussion groups in this topic, we partnered with EBI and MoreBrains to identify which of the report recommendations were most actionable for research funders to undertake. In September and October 2023, DORA, EBI, and MoreBrains organized a set of parallel symposia and workshops for each funder discussion group. The groups used the EBI report recommendations as a tool to develop and prioritize areas for intervention.
Key takeaways
Feasibility and impact
During the virtual symposia held in September 2023, Ian Penton-Voak (EBI at University of Bristol), Josh Brown and Alice Meadows (MoreBrains) presented the report rationale and findings to the groups. This was followed by a presentation from Gearoid Maguire (Wellcome) on the Equitable Funding Practice Library to prime the subsequent breakout discussions. Members of the research funder groups then conducted a deep dive into select recommended actions from the EBI report and identified the 1) feasibility and 2) anticipated impact(s) of each recommendation. Here, the term “impact” was used in the context of the anticipated benefit that researchers might have from a particular recommendation, in the form of reduced barriers to funding applications and/or greater transparency. Funders in both groups discussed several mechanisms by which this recommendation could be implemented. The actions and potential ways to address them practically included:
- Avoid training becoming yet another burden.
- Diversify the idea of research careers, and break down cultural silos.
- Explore and experiment fairly and transparently.
- Consider the impact of how funding opportunities are designed, structured, and shared.
- Leverage providing applicants with reviewer feedback to support reapplication and reviewer integrity.
During the large group discussion, several important points of consideration were highlighted by the group:
- Communication is a key facet of pre-award processes that can either encourage or discourage applicants.
- Funders should work to focus on inclusivity and reach (e.g., gender neutral language, opportunities to apply in other languages, broaden communications channels used to share funding calls, target smaller organizations and early career researchers). Of note, the value of protected or targeted opportunities in various forms was mentioned in the majority of Asia-Pacific breakout sessions.
- Unconscious biases around what constitutes research excellence can present a barrier to more inclusive call design. For example, if a call is co-created with researchers, the biases and priorities of those academics influence call design and limit flexibility on the part of the funder.
Reform in action
The work from the September symposia informed the design of the October workshops, which featured recorded lightning talks from members of each discussion group. These lightning talks highlighted practical examples of how funders have worked to reduce barriers for applicants and support transparency and/or EDI in their funding calls. We heard from:
Shea Robin-Underwood of the Ministry of Business, Innovation & Employment (MBIE), Aotearoa New Zealand. Robin-Underwood’s talk Supporting Indigenous Research focused on MBIE’s He aka ka toro investment fund for projects that advance iwi, hapū, hapori, and Māori research, science, and innovation (RSI). Robin-Underwood highlighted the importance of involving Indigenous peoples in every aspect of the fund design process, which was done for the RSI investment fund, and shared that there is a strong appetite and interest in the Indigenous researcher community for this type of protected funding.
Shomari Lewis-Wilson of Wellcome, United Kingdom. Lewis-Wilson’s talk Research culture and Communities focused on Wellcome’s values-based approach to supporting research culture in all of its programming: supporting ethical, open, and engaged research; enabling equitable, diverse, and supportive research cultures; and being an inclusive partner. There are several ways that Wellcome puts this approach into practice: embedding “Research Environment” in applications, recently launching the Institutional Fund for Research Culture, implementing a narrative CV, and supporting Black-led grassroots organizations.
Julie Glover of the Australian National Health and Medical Research Council (NHMRC). Glover’s talk Examples of Equity, Diversity and Inclusion Initiatives focused on initiatives to support NHMRC’s goals for equitable funding for female and non-binary chief investigators and for Aboriginal and Torres Strait Islander researchers. Glover highlighted that NHMRC uses structural priority funding as a mechanism for awarding funds to meet these goals, and this has resulted in more women and Indigenous researchers receiving funding. To address the attrition of more senior female applicants, the NHMRC has also recently prioritized funding equal numbers of grants to men and women.
Sarah Smith of the Natural Sciences and Engineering Research Council (NSERC), Canada. Smith’s talk NSERC’s Approach to Equity, Diversity and Inclusion in the Pre-Award Process highlighted that transparency and EDI are codified as organizational priorities in NSERC’s strategic plans, which also supports NSERC’s goals to support Indigenous researchers. Smith outlined several examples of how NSERC is working to implement these goals, including piloting a narrative CV, regular community engagement, removing procedural barriers within its programs, proportional representation of awardees, and target funding for Black and Indigenous trainees.
Click to watch the lightning talk recordings
Identifying realistic and transformative actions
Following the lightning talks, the attendees reviewed the list of actions that were discussed during the symposia and prioritized which actions they would like to focus on. During the subsequent breakout sessions, participants drafted brief action plans on how best to put each proposed action into practice. Examples include:
“Our group focused on making call structure simpler, clearer, and more inclusive. The activities needed to implement this idea are working with the communications department (including editing and creating plain language), and community consultation on what is needed. The resources needed to implement this idea are time and staff, for inter- and intra-organizational communications, access to experts, and community members. The people or groups who would need to be involved in this are community members, communications department, policy staff, EDI advisors, IT department. We think we should all take this idea forward because it’ll increase applications.”
Funders then voted on the top actions that would be 1) most realistic for funders to implement and 2) most transformative to the pre-award ecosystem. The areas of activity that were most impactful or transformative were:
- Making assessment more holistic
- Making call structures simpler, clearer, and more inclusive
- Improving training for reviewers and assessors
The areas of activity that were most feasible and realistically achieved were:
- Improving training for reviewers and assessors
- Making call structures simpler, clearer, and more inclusive
Across the two workshops, which represent different geographic regions, the participants felt that making call structures simpler, clearer, and more inclusive was one of the most realistically achievable for funders and potentially most transformative for applicants and the pre-award landscape.
“It has been so gratifying to see the work we did with EBI to improve equity, diversity, and inclusion in pre-award processes being taken up by the DORA community. It’s inspiring to hear about their existing efforts in this area, and we look forward to continuing to work with them to develop concrete guidance on opportunities for improvement across the wider research funding community.” – Alice Meadows, MoreBrains co-founder
Next steps
Building on the recommendations identified by the DORA funder discussion group in 2023 symposia and workshops, we are pleased to announce that EBI has secured additional funding from the University of Bristol for a new project to look at how three of the recommendations could be implemented. The three areas of focus are: 1) simplification of funding call structures; 2) changes to application processes to reduce likelihood of bias in outcomes (e.g. recognizing a broader range of research outputs, narrative CV formats); and 3) improvement in training for reviewers and evaluators. This project will include two parallel workshops to bring together members of the DORA funder discussion groups and research administrators. Participants will discuss how interventions relating to the three key areas could be implemented, practical barriers could be reduced, and progress could be benchmarked. The outcomes of this project will also include a report and other materials to support research funders in their implementation of evidence-based best practices.
Haley Hazlett is DORA’s Program Manager
* In the report, the EBI and MoreBrains team used the term ‘equality’ as in the acronym EDI, for equality, diversity, and inclusion to refer to approaches and values that may lead to equitable outcomes, noting that “the terms ‘equality’ and ‘equity’ are different and [that they] felt… the focus on equality of rights and opportunities was highly pertinent in the context of the pre-award process.”
