As part of DORA’s effort to increase communication about innovative policies and practices for assessing research, we expanded the funder discussion series by hosting our first virtual meeting for the Asia-Pacific region on Tuesday, August 18, 2020. Boyana Konforti, Director of Scientific Strategy and Development at Howard Hughes Medical Institute (HHMI) in the United States, and Julie Glover, Executive Director of Research Foundations at the National Health and Medical Research Council (NHMRC) of Australia, provided updates on research assessment policies at their organizations.
HHMI works to promote discovery science and emphasizes public engagement, a healthy academic ecosystem, and diversity, equity, and inclusion in the scientific workforce. How researchers are assessed across the scientific community is vital to achieving these goals. Konforti highlighted challenges in current assessment practices within the scientific community generally, including a lack of transparency and accountability when peer review evaluations are not made available for published research articles. Sharing peer reviews of research articles, commonly referred to as open peer review, provides information about an article’s merits that goes beyond the prestige of the journal where the work was published.
To enable scientists to follow their curiosity and to promote discovery science for long-term impacts on scientific progress, Konforti said that HHMI supports “people, not projects.” HHMI recently increased the appointment term for their Investigator Program from five to seven years to provide the researchers they fund with even longer-term support and to enable them “to commit to the kind of rigorous, open-ended research that HHMI expects.” To increase transparency, information about the evaluation process for HHMI Investigators is published on their website.
In their approach to assessment, Konforti noted a focus is on assessing research impact and quality of contributions during the appointment period relative to others in the field. For example, as part of their written submission, individual applicants are asked to provide short impact statements for five of their most significant research papers. Consistent with HHMI’s dedication to open science, the papers under evaluation do not need to be published in a journal, but are required to be shared publicly (e.g. as a preprint). The peer reviewers are provided with a set of six evaluation criteria, one of which is activity in service and training in the Investigator’s host institution, and in the greater scientific community.
Konforti also discussed the in-person prong of HHMI’s review process, an aspect of which is that individual researchers give a short presentation. Importantly, HHMI’s expert advisors are not tasked with making final appointment renewal decisions; their assessments and recommendations inform final decisions made by HHMI leaders.
NHMRC of Australia funds many types of grants, including Investigator Grants, which support outstanding investigators at all career stages, and Ideas Grants, which provide support for innovative projects addressing a specific research question. NHMRC introduced a new grant program in 2019, which includes a new framework for “Track Record,” which they define as “the value of an individual’s past research achievement, relative to opportunity, not prospective achievements, using evidence-based components.” According to Glover, the emphasis on track record encourages the evaluation of researchers relative to opportunity; it assesses past, not prospective, achievements and importantly accounts for gaps in research careers.
For each Investigator Grant application, Glover said NHMRC peer reviewers use publications, research impact, and leadership to assess an applicant’s Track Record. She added that there is a greater value placed on research output and impact than input measures such as grants. Each element within the new Track Record assessment framework is scored using publicly available scoring systems supported by category descriptors. For example, publications are assessed from a 10-year list, which takes into account career distributions.
Similar to HHMI, Glover said the NHMRC asks its applicants to highlight their five best publications, within that 10-year timeframe. Research impact is assessed through the applicant’s description of verifiable outcomes that applicants’ research makes to knowledge, health, the economy and/or society. Importantly, assessment is not on the anticipated impact of a project, but focuses on the verified outcomes of previous research and how that work has been adopted for use or used to inform further investigation.
Glover said that contributions to leadership are also assessed within the most recent 10-year window, and include activities such as mentoring and other professional or institutional leadership roles. Using feedback from applicants and peer reviewers after implementing Track Record into their funding assessment, NHMRC plans to continually review and revise their assessment policies and practices in order to better inform peer reviewer training and ensure fair assessment.
DORA’s funder discussion group is a community of practice that meets virtually every quarter to discuss policies and topics related to fair and responsible research assessment. If you are a public or private funder of research interested in joining the group, please reach out to DORA’s Program Director, Anna Hatch (info@dora.org). Organizations do not have to be a signatory of DORA to participate.
Ashley Lakoduk is DORA’s Policy Intern