Each quarter, DORA holds two Community of Practice (CoP) meetings for research funding organizations. One meeting takes place for organizations in the Asia-Pacific time zone and the other meeting is targeted to organizations in Africa, the Americas, and Europe. The CoP is a space for funders to learn from each other, make connections with like-minded organizations, and collaborate on projects or topics of common interest. Meeting agendas are shaped by participants. If you are employed by a public or private research funder and interested in joining the Funder CoP, please find more information on our webpage or email us at info@sfdora.org.
During the first 2025 meeting of the DORA Africa, Americas, Europe Funder Discussion Group meeting, we welcomed Dr. Ulrike Bischler of the Volkswagen Foundation, who has been working as a program director for three decades now and presented a deep dive into the Volkswagen Foundation’s experience with partial randomization as a component of their former “Experiment! – Search of bold research ideas” funding framework. This initiative, which ran from 2013 to 2020/21, aimed to support high-risk, high-gain projects within life and natural sciences, and engineering, receiving a significant number of applications (over 5,000 for 183 grants approved).
Addressing Selection Challenges: The Role of Partial Randomization
Dr. Bischler’s presentation tackled a fundamental question: Given that peer review is often considered the “gold standard”, what is the rationale for incorporating lotteries into research funding processes? She pointed out that historically, lotteries have served important purposes beyond mere chance, particularly in ensuring fairness and preventing undue influence. Examples include:
- Athenian Democracy: Lotteries were used for the allocation of political offices to ensure the independence of mandate holders and make bribery impossible.
- Republic of Venice: Lotteries helped in the election of the Doges to limit the influence of wealthy clans and achieve a broader power balance.
- Basel University (18th century): Selection of the rector and professorial positions was done via lottery from a merit-based list of three to encourage top candidates to apply without fear of losing face.
- Soccer Tournaments: Coin tosses were used as tiebreakers (before penalty shootouts), recognizing that rational choice might be impossible when there is no significant difference in performance.
The presentation also included a comparison of strengths and weaknesses of different selection processes:
- Peer Review: Offers distinction by quality and reasoning for feedback but can be biased and time-intensive.
- Double-blind Review: Shares the strengths of peer review and reduces personal bias but may still have topic-related biases and challenges in reviewer selection.
- Lottery Only: Is bias-free and time/labor-saving but lacks quality distinction and feedback.
- Partially Randomized: Aims to combine the strengths of both peer review and lottery systems.
The “Experiment!” Funding Model and the Introduction of Randomization
The Volkswagen Foundation’s Experiment! was a small grant scheme aimed to explore high-risk, high-gain ideas with uncertain outcomes in life sciences, natural sciences, and engineering. The framework employed a structured, three-step process for proposal review:
- Pre-selection by Program Directors: This initial stage involved checking formal eligibility and program compatibility, resulting in a shortlist for the external jury. The high application volume made this a time-consuming step.
- Review by an External Jury: A double-blind peer review process was conducted by an external jury. They performed a scientific quality triage, categorizing proposals as “top”, “entry in lottery”, or “not fundable”. Identifying and securing the availability of suitable reviewers was a persistent challenge.
- Supervised Lottery: A supervised physical lottery was conducted, with the lottery pool comprising proposals that the jury deemed “top” or suitable for “entry in lottery”. This allowed the foundation to create two distinct cohorts for comparison from the same applicant pool.
Key Findings and Lessons Learned on the Impact of Randomized Selection
Analysis of the outcomes of the “Experiment!” initiative provided several notable insights:
- Enhanced Diversity: A significant increase in diversity was observed among the funded projects, with more early career grantees, more women (potentially linked to the higher representation of women among early career researchers, even with double-blind review), and more minority subject areas being funded. Notably, the demographic profile of the applicant pool remained largely consistent.
- Acceptance within the Scientific Community: While some jury members initially had reservations, all ultimately favored the process. Importantly, the Volkswagen Foundation reported no complaints from applicants. The framing of the lottery as a complementary tool to scholarly discourse was considered crucial for its acceptance.
- Perceived Research Quality: A survey accompanying the research indicated initial concerns about a potential decline in research quality due to the lottery. However, a comparison of the jury-selected and lottery-selected cohorts revealed very similar research outputs and outcomes, including publications, patents, and funding/career effects. The Foundation’s assessment of outputs suggested no significant difference in quality between the two groups, particularly given the initial pre-selection by the jury.
- Relief for Peer Reviewers: Partial randomization was seen as a way to alleviate the burden and save time on peer review panels when faced with numerous high-quality proposals that were rationally indistinguishable.
The Q&A session addressed how to collect data about diversity in topics for curiosity-driven research, where the diversity data was collected from the field of research submitted by applicants. It was noted that while larger fields showed proportional selection between application and selection rates in double-blind reviews, smaller fields saw more proportional selection even if they didn’t apply as much when using partial randomization.
Updates from Other Funders
Following Dr. Bischler’s presentation, several attendees shared updates on their own initiatives:
- Sean Sapcariu (Luxembourg National Research Fund) highlighted that PEP-CV will integrate with ORCID, allowing users to demonstrate mentorship on their ORCID profiles. PEP-CV is a platform for peer mentoring exchanges around completing narrative-style CVs, helping early career researchers learn to write effective narrative CVs.
- Anna Hatch (Howard Hughes Medical Institute) reported their Transparent and Accountable Peer review (TAP) training will be offered again in 2025. The training uses preprints teach Early Career Scientists how to produce collegial and constructive peer reviews.
- Annalisa Montesanti (HRB Ireland) announced the revamped content about the HRB Narrative-style CV in the HRB website and the publication of users’ experience insights at https://www.hrb.ie/funding/hrb-narrative-style-cv/.
Looking Ahead: Q2 2025 Meeting and Call for Presentations
DORA announced upcoming programming updates, including the Q2 2025 meeting which will feature a presentation from the Global Research Council RRA Working Group members on the GRC RRA Working Group Dimensions report survey. The invite for this meeting will be updated to accommodate the new date (May 29).
Funders are also invited to consider submitting their initiatives for presentations in the Q3 and Q4 2025 slots, which are currently open.