Each quarter, DORA holds two Community of Practice (CoP) meetings for research funding organizations. One meeting takes place for organizations in the Asia-Pacific time zone and the other meeting is targeted to organizations in Africa, the Americas, and Europe. If you are employed by a public or private research funder and interested in joining the Funder CoP, please find more information here.
Research funding organizations are often thought of as leaders in the movement toward more responsible research evaluation practices. Often, the perception of “excellence” in research culture is filtered through the lens of who and what type of work receives funding. However, when a narrow set of indicators is used to determine who receives funding, the result can be a subsequent narrowing of academia’s perceptions of research excellence (e.g., journal impact factor (JIF), h-index). This places funders in the unique position of being able to help “set the tone” for research culture through their own efforts to reduce reliance on flawed proxy measures of quality and implement a more holistic approach to the evaluation of researchers for funding opportunities. Whether funders are seeking inspiration from their peers or insight on iterative policy development, the ability to come together to discuss reform activity is critical for achieving widespread culture change. At DORA’s June Funder Community of Practice (CoP) meetings, we heard how DORA is being implemented by the Natural Sciences and Engineering Research Council of Canada (NSERC) and the New Zealand Ministry of Business, Innovation and Employment (MBIE).
During the first meeting, Alison Janidlo, Nathalí Rosado Ferrari and Brenda MacMurray presented on the work that NSERC has done to implement DORA. NSERC is a national funding agency that supports researchers and students through grants and scholarships. Applications are evaluated through peer review. In alignment with its existing good practices for research assessment (e.g., encouraging applicants to include outputs outside of journal articles and asking peer reviewers to focus on the quality and impact of outputs), NSERC signed DORA in 2019. Building on the previous version of its guidelines, NSERC developed a new assessment guidelines document that is more explicitly aligned with DORA principles. During the call, Janidlo, Rosado Ferrari and MacMurray described how the development of these new guidelines was informed by literature review, stakeholder engagement through committees, focus groups, and conference presentations. Key features of the guidelines include the promotion of NSERC’s support for research excellence, the explicit incorporation of DORA principles, and the emphasis that proxy measures of quality (e.g., JIF) must be avoided when evaluating research proposals.
Janidlo, Rosado Ferrari and MacMurray also described what is arguably one of the most important facets of implementing policy change: a reflection on what worked and lessons learned over the 2-year development process of their new guidelines. According to them, a key lesson was that it would have been useful for their community to have more clarity about the next steps for implementation at the time of the guidelines launch. The agency will also need to address the tricky question of how to track the impact of the new guidelines on funding decisions. In terms of strategies that were successful, the speakers highlighted that these new guidelines were developed iteratively and in consultation with a broad range of stakeholders within the NSERC community, allowing for greater buy-in with the new document. Additional successful strategies included listing forms of contributions and indicators of quality and impact in alphabetical order (to de-emphasize proxy measures of quality and impact), the language inclusivity of the new document (reflecting Canada’s official English and French bilingualism, consideration of the diverse postsecondary community and technical jargon used in the natural sciences and engineering domain), the use of the existing NSERC DORA Implementation Working Group to successfully liaise with stakeholders, and NSERC’s efforts to build awareness about DORA in both formal and informal meetings with stakeholders. To the final point, the NSERC team created a DORA email address for community members to send DORA-related questions and inquiries. The new guidelines were released May 6, 2022.
During the second meeting, Joanne Looyen and Farzana Masouleh presented on the work that the New Zealand MBIE has done to implement DORA. MBIE is a funding organization that offers contestable funds for a range of research. Looyen and Masouleh discussed the stewardship role that research funders hold and their responsibilities to support greater diversity in the research system. To fulfill this role, they emphasized that funders may need to rethink their existing guidelines to assess research “excellence”. Here, Looyen and Masouleh pointed out that DORA’s principles can be applied broadly to support more fair and consistent evaluation of funding proposals. To this point, MBIE is currently working with the Health Research Council and the Royal Society of New Zealand to develop a narrative-style CV for New Zealand researchers. The expectation is that researchers will be able to use this CV to apply for funds from any of the three agencies. This is important because interoperability of CV formats between organizations is an important concern often voiced by researchers. The speakers highlighted the specific benefits of using a more narrative-style CV for public funding applications, such as broadening the definition of what “excellence” is for Māori and Pacifica researchers, and better capturing the depth of their work. MBIE currently has two ongoing funding calls where it is introducing the narrative CVs: the Health Research Council Trial for the Ngā Kanohi Kitea (NKK) Community Fund, and the Annual Contestable Endeavour Fund. For example, NKK Community Fund applicants must submit a narrative-style CV that includes relevant experience to the proposed activity, contributions of knowledge generation, and contributions to iwi, hapū, or community.
Key considerations that were discussed included the importance of designing a narrative CV template that works with and for the New Zealand context. This means testing iterations of the template with research staff from New Zealand universities and research institutions, incorporating stakeholder feedback, and conducting a survey of applicants and assessors to gather data about their user experiences with the new template. Looyen and Masouleh concluded their presentation by returning the focus back to one of the key drivers of the MBIE’s reform efforts: increasing understanding of how research can contribute to the aspirations of Māori organizations and deliver benefits for New Zealand.
During both meetings, Funder CoP members discussed the actions that their peers have and are in the process of taking to implement a more holistic approach to the evaluation of researcher contributions. One of the key takeaways exemplified by both NSERC and MBIE is that there is no “one-size-fits-all” approach. Although the availability of examples of change are critical for drawing inspiration and demonstrating feasibility, ultimately the best policies and practices are those that fulfill the context-specific needs and goals of each funding organization’s community.
Haley Hazlett is DORA’s Program Manager.