Each quarter, DORA holds two Community of Practice (CoP) meetings for research funding organizations. One meeting takes place for organizations in the Asia-Pacific time zone and the other meeting is targeted to organizations in Africa, the Americas, and Europe. The CoP is a space for funders to learn from each other, make connections with like-minded organizations, and collaborate on projects or topics of common interest. Meeting agendas are shaped by participants. If you are employed by a public or private research funder and interested in joining the Funder CoP, please find more information on our webpage or email us at info@sfdora.org.
The first 2025 meeting of the Asia-Pacific Funder Discussion Group convened on Thursday, March 13th, bringing together representatives from various regional research funding organizations. The meeting featured the presentation from Professor Adrian Barnett (Queensland University of Technology (QUT)), who has over 30 years of experience in health and medical research and a current focus on meta-research and shared his perspectives on utilizing lotteries to enhance fairness and efficiency in research assessment, particularly when traditional peer review faces inherent limitations. The ensuing discussion underscored the growing recognition of the unpredictability of scientific progress and the potential for randomized selection to complement merit-based evaluation, striving towards a more equitable and effective research funding system.
Addressing Selection Challenges: “When the reasons run out”
Prof. Barnett’s presentation began by referencing the philosophical notion of using lotteries “when the reasons run out” with historical examples of randomization in decision-making. Examples included a football match between Sheffield and Shropshire Wanderers in 1873 decided by a coin toss after two draws; places on medical degrees in Sweden and the Netherlands; military conscription in the USA and Australia; and the sophisticated system of stratified randomization in Athenian democracy through utilizing the Kleroterion. Prof. Barnett underscored how societies have been using lotteries for a long time to increase fairness, and the fundamental difficulty of perfectly ranking research proposals, echoing Immanuel Kant’s recognition of the limitations of reason.
A series of examples then showcased the impossibility of perfect ranking and the role of chance. From a large NIH scheme demonstrating a weak correlation between initial grant scores and subsequent research outcomes in terms of citations (10.7554/eLife.13323) to a NHMRC data indicated that a significant 29% of grants fell into a “grey zone”, where funding success was contingent on the specific reviewers assigned (10.1136/bmj.d4797). Finally, the poignant case of Katalin Kariko, who was the only one not funded in an NIH scheme with funding available for six out of seven applications, despite her later Nobel Prize-winning work on messenger RNA, powerfully illustrated the potential for overlooking groundbreaking ideas, the possibility of sexism playing a role in her initial rejection was also noted, with lotteries being a way to mitigate for this risk.
Inefficiencies of Current Funding and Experimentation with Lottery
The meeting also addressed the inefficiency of current complex funding application systems, where the effort required can detract from actual research. Prof. Barnett described the “Szilard point”, where the expenses incurred in obtaining a grant equal the value of the awarded grant, drawing a parallel to a Bond villain’s strategy to undermine research by creating numerous high-stakes, low-success-rate funding schemes. He cited specific examples such as the Axbury fellowship in Australia with 100-160 applications for a single $25,000 award and the Johnson & Johnson fellowship for women in STEM with 650 applications for six awards, illustrating the less than 1% success rates in some cases.
Transitioning to the potential of lottery systems, Prof. Barnett highlighted the growing number of funders adopting this approach, including the Health Research Council of New Zealand, VolkswagenStiftung, Austrian Science Fund (FWF), Swiss National Science Foundation (SNSF), Novo Nordisk Fonden, British Academy, UKRI/NERC, and Wellcome, among others. He elaborated on the Health Research Council of New Zealand’s exploratory grant scheme, which employed two simple yes/no criteria: “is the research potentially transformative?” and “is the proposal exploratory but viable?”. Proposals receiving two “yes” answers from a majority of reviewers were entered into a lottery. A survey of applicants revealed that 63% were in favor of this method, provided ineligible applications were excluded through initial peer review and the remaining pool was deemed roughly equal in merit. Notably, support for randomization was higher among those who won funding through the lottery, which is to be expected as people are usually biased to systems that favored them. The interest in lotteries is also supported by a survey of the community that indicates a license to experiment and pilot with lotteries.
Challenges and Opportunities of Lottery Systems
Prof. Barnett then addressed common arguments against using a modified lottery, such as the concerns that it rewards the less deserving or less enthusiastic, encourages the less talented to apply, and leads to applicants putting in less effort. He countered that the New Zealand experience showed applicants invested a similar amount of time in their applications due to the initial peer review stage.
He outlined several potential benefits of incorporating modified lotteries:
- Reduces cronyism and the perception thereof, especially in small scientific communities.
- Increases diversity in applicants and winners. Early data from the British Academy indicated an increase in non-white applicants after implementing a conditional lottery. The study is not yet ready, but the assumption is that there were more applicants due to a increase in perception of fairness.
- Reduces application and review costs.
- Acknowledges the inherent unpredictability of scientific discovery.
- Reduces the stigma associated with ‘failure’ when proposals are rejected.
- May mitigate the “Matthew effect” that may come from the well-funded researchers continuing to be funded.
- Could encourage applicants to propose more innovative ideas, which could move science faster.
- Creates opportunities for randomized trials of funding mechanisms, which are currently lacking despite the vast sums allocated through traditional methods that are still under evaluated. The randomization of medical school admissions in Sweden has for example allowed for studies demonstrating the significant impact of having a doctor in the family on life expectancy.
Prof. Barnett summarized how complex application systems are costly and potentially amplify biases and how lotteries may end up being the most scientific approach. You are welcome to watch back the presentation and retrieve Professor Adrian Barnett’s slides.
Q&A Session: Reviewer Perspectives and Lottery Weighting
The subsequent Q&A session explored reviewers’ perspectives, with Barnett acknowledging mixed initial reactions. Ideas for weighting lotteries were discussed, including the concept of a “golden ticket” that reviewers could award to guarantee funding for a particularly promising proposal. The possibility of allocating multiple tickets based on reviewer enthusiasm or granting extra tickets to applicants who have been unsuccessful in the lottery for several consecutive years was also raised. The importance of an initial peer review stage to determine eligibility and a basic level of merit was emphasized, the crucial question of evaluating the effectiveness of the lottery system compared to the traditional merit-based process, and public perception on lotteries was also discussed.
Looking Ahead: Q2 2025 Meeting and Call for Presentations
DORA announced upcoming programming updates, including the Q2 2025 meeting which will feature a presentation from the Global Research Council RRA Working Group members on the GRC RRA Working Group Dimensions report survey. The invite for this meeting will be updated to accommodate the new date (May 29).
Funders are also invited to consider submitting their initiatives for presentation in the Q3 and Q4 2025 slots, which are currently open.