Driving Institutional Change for Research Assessment Reform

October 21 – 23, 2019

Improving research assessment

Academic institutions and funders assess their scientists’ research outputs to help allocate their limited resources. Research assessments are codified in policies and enacted through practices. Both can be problematic: policies if they do not accurately reflect institutional mission and values; and practices if they do not reflect institutional policies.

Even if new policies and practices are developed and introduced, their adoption often requires significant cultural change and buy-in from all relevant parties – applicants, reviewers and decision makers.

We will discuss how to develop and adopt new research assessment policies and practices through panel discussions, short plenary talks and breakout sessions. We will use the levels of intervention described in the “Changing a Research Culture” pyramid (Nosek, 2019), to organize the breakout sessions.

We want your input on the agenda:

  • Looking at the brainstorming sessions, are there questions you would want to include?
  • Are there other topics we should consider for brainstorming sessions and what questions would be discussed?
  • What’s missing from the panel discussions, short talks and brainstorming sessions?
  • Webcast archive

    Items marked with were webcast. Watch now and join the conversation now with #AssessingResearch on social media!

    Monday, October 21, 2019
    5:00 pm Reception, Great Hall
    6:00 pm Dinner, Dining Room
    7:00 pm

    Opening Remarks, Cowan Auditorium

    • Boyana Konforti (HHMI)
    • Erika Shugart (ASCB and DORA) – slides
    7:20 pm

    Keynotes, Cowan Auditorium

    • Erin McKiernan (UNAM) – slides
      How the journal impact factor is used in review, promotion, and tenure in the United States and Canada
    • Paula Stephan (Georgia State University) – slides
      How we can change the research assessment culture
    • Bodo Stern (HHMI) – slides
      HHMI’s approach to evaluating scientists
    8:20 pm Joint Q&A, Cowan Auditorium
    Moderated by Stephen Curry (DORA)
    8:45 pm Evening wrap-up, Cowan Auditorium
    Stephen Curry (DORA) – slides
    9:00 – 11:00 pm Social, The Pilot Lounge
    Tuesday, October 22, 2019
    7:00 am Breakfast, Dining Room
    8:30 am

    How to develop new research assessment policies and practices
      Panel Discussion, Cowan Auditorium
    Moderated by Erika Shugart (ASCB and DORA)

    • Stephen Curry (Imperial College and DORA)
    • Frank Miedema (UMC Utrecht / Utrecht University)
    • Sandra Schmid (UT Southwestern Medical Center)
    • Miriam Kip (BIH QUEST Center for Transforming Biomedical Research)
    • Omar Quintero (University of Richmond)
    • Needhi Bhalla (University of California, Santa Cruz)
    9:30 am

    Breakout Sessions

    Participants will work in preassigned groups for all breakout sessions. Each group will have a designated discussion leader and note taker. To encourage open and honest discussion, breakout sessions will be held under the Chatham House Rule; information shared in breakout sessions may NOT be attributed to the speaker.

    Breakout topic 1. Infrastructure – using working groups to establish policies and practices

    1. How might institutional working groups be established? What different stakeholders need to be at the table?
    2. What information and support do working groups need to be positioned to evaluate research assessment practices?
    3. Are there any unintended consequences of using working groups to effect change? What would they be?
    4. How might working groups establish new policies?

    Breakout topic 2. User experience

    1. Many things can prejudice assessors: gender, race, ethnicity, institutional reputation, and more. How do organizations stay aware of and responsive to barriers the different groups might encounter?
    2. What parts of the research assessment process can lead to assessor bias?
    3. What nudges can an organization consider to influence the behavior of individuals submitting applications and those assessing them?
    4. How do organizations make it easy for different groups to understand and implement their policies and practices?
    10:45 am Break, Great Hall
    11:00 am Reports from Breakout Sessions, Cowan Auditorium
    Moderated by Anna Hatch (DORA)
    12:45 pm Lunch
    1:45 pm Discussion of Morning, Cowan Auditorium
    Moderated by Bodo Stern (HHMI)
    2:15 pm

    How to adopt new research assessment practices
      Short Plenary Talks, Cowan Auditorium
    Moderated by Boyana Konforti (HHMI)

    • Brian Belcher (Royal Roads University) – slides
      How conceptual clarity can improve how we assess research
    • George Santangelo (NIH Office of Portfolio Analysis) – slides
      Ways to measure impact without the JIF
    • David Mellor (Center for Open Science) – slides
      Lessons learned from open science initiatives about how to change research culture
    • Ruth Schmidt (Institute of Design IIT) – slides
      How to apply a user-design approach to research assessment reform
    • Lou Woodley (CSCCE) – slides
      How systems thinking applies to research assessment reform
    3:15 pm Break, Great Hall
    3:30 pm

    Breakout Sessions

    Breakout topic 3. Communities

    1. How can the researcher community encourage institutions to establish new research assessment policies and practices?
    2. Once alignment between policies and practices has been achieved, how can organizations build the support of the research community to make good practices normative?
    3. Are there other ways an organization could more clearly articulate to the research community how their priorities translate into practices?
    4. How can different stakeholder communities, including research institutions, funders, scholarly societies, publishers, and researchers, signal their priorities, policies, and practices to each other?

    Breakout topic 4. Incentives

    1. What are the incentives for funders and research institutions to align their research assessment policies and practices?
    2. How do incentives differ between assessors and those being assessed?
    3. How can funders and research institutions incentivize the behaviors they would like to promote?

    Breakout topic 5. Policy

    1. Is the vision, mission, and values of an organization accurately reflected in its policies? If not, what is the process by which policies can be modified?
    2. What are the benefits of publicly sharing an organization’s policies? What are the risks?
    3. What are the benefits of publicly sharing an organization’s practices? What are the risks?
    4. Is there a role for funders to influence how an organization’s policies are translated into its research assessment practices? Why or why not?
    4:45 pm Break, Great Hall
    5:00 pm Reports from Breakout Sessions, Cowan Auditorium
    Moderated by Stephen Curry (DORA)
    6:00 pm Discussion of Afternoon, Cowan Auditorium
    Moderated by Erin McKiernan (DORA and UNAM)
    7:00 pm Dinner, Dining Room
    8:30 – 11:00 pm Social, The Pilot Lounge
    Wednesday, October 23, 2019
    7:00 am Breakfast, Dining Room
    8:30 am

    Applying systems thinking to research assessment reform
    Introduction of Breakout Sessions, Cowan Auditorium
    Moderated by Anna Hatch (DORA)

    Research assessment reform involves multiple stakeholders, including funders, research institutes, libraries, publishers, and researchers. How can stakeholders work together to implement change in research assessment practices? What do different stakeholder groups need from each other to succeed?Participants will divide into groups of 6-8 people to discuss topics inspired by the ideas presented in the participant commentaries.

    9:00 am

    Breakout Sessions

    1. Finding the right balance between top-down and bottom-up approaches for research assessment reform
      Facilitators: Diego Baptista (Wellcome) & Connie Lee (University of Chicago)
      Room D113
    2. How might we use an institution’s stated values as a starting point to improve research assessment policies and align them with practices?
      Facilitator: Erin McKiernan (UNAM)
      Room D115
    3. Building trust in policies and practices—how do researchers know that research institutions and funders really mean what they say?
      Facilitators:  Dave Carr (Wellcome) and Prachee Avasthi (University of Kansas Medical Center)
      Room D116
    4. How can scholarly societies lead from the outside to influence research assessment reform?
      Facilitators: Erika Shugart (ASCB) and Brooks Hanson (AGU)
      Room D124
    5. What can departments and institutions do to improve the triage phase of assessment for faculty searchers?
      Facilitators: Lee Ligon (RIT) and Sandra Schmid (UT Southwestern Medical Center)
      Room D125
    6. Where do university rankings fit into research assessment reform?
      Facilitator: Stephen Curry (DORA)
      South Lounge
    7. How can departments, institutions, and funders evaluate contributions to team science?
      Facilitator: Sue Biggins (Fred Hutchinson Cancer Research Center) and Miriam Kip (BIH QUEST Center For Transforming Biomedical Research)
      North Lounge
    8. How might we improve equity and inclusion in academia (e.g. by examining how models of scarcity and exclusivity influence our current research assessment practices and concepts of rigor)?
      Facilitators: Needhi Bhalla (UC Santa Cruz) and Olivia Rissland (University of Colorado School of Medicine)
      Room C123
    9. What do preprints need to be more useful in evaluation?
      Facilitator: Jessica Polka (ASAPbio)
      Room N140
    10:30 am Break, Great Hall
    10:45 am Reports from Breakout Sessions, Cowan Auditorium
    Moderated by Boyana Konforti (HHMI)
    11:30 am Lunch, Dining Room
    12:30 pm Meeting Reflections and Next Steps, Cowan Auditorium
    Anna Hatch (DORA)
    1:00 pm Departures