A DORA Community Engagement Grants Report
In November 2021, DORA announced that we were piloting a new Community Engagement Grants: Supporting Academic Assessment Reform program with the goal to build on the momentum of the declaration and provide resources to advance fair and responsible academic assessment. In 2022, the DORA Community Engagement Grants supported 10 project proposals. The results of the Exploring the Current Practices in Research Assessment within Indian Academia project are outlined below.
By Suchiradipta Bhattacharjee and Moumita Koley — Indian Institute of Science (India)
Background: Standards for Scholarship
The state of research evaluation in India
Research assessment is an integral part of the vast research and academic ecosystem of India. Surprisingly though, how it is done is not well understood. Are they quantitative or qualitative or a mix of both? Researchers associated with the system have a vague idea but scholarly literature on this subject is almost non-existent. For the most part, it is definitely quantitative metrics driven, but there are also a good number of institutions following a more qualitative approach. Be it qualitative or quantitative, the underlying factor of assessment remains the same – how popular/established a scientist is in her field and the basis of this judgment goes back to research articles, journal prestige and citations. It is therefore reasonable to enquire if the assessment process has stagnated for decades. Moreover, it is also fair to ask whether the present system encourages novel knowledge creation, innovation or social impact?
Importance of the Project
Scientists respond to incentives just as everyone else does. Is the present assessment system incentivising scientists to engage in ground breaking research? Or just rewarding them to pick the low hanging fruits that may lead to a good number of papers in reputed journals? With India’s steady investment in R&D and commitment towards increased utilization of science and technology to solve societal challenges, we felt it is necessary to understand if the present system is rewarding incremental and over-exploration of research fields rather than breakthrough or socially relevant scientific research?
Inspiration for the Project
Over-reliance on readily-available journal-based indices to assess the quality of research is now a global phenomenon and problem. India is no exception, at least partially. JIF is a number that provides a false sense of superiority to the researchers – the higher the impact factor of a journal, the higher the reputation and quality of the journal. Similarly, researchers’ performance and achievements are synonymous with high h-index scores or number of citations received.
Against this backdrop, we wanted to explore the research assessment practices followed by Indian academics, and start the necessary conversation surrounding the importance of a robust research assessment process that will encourage transformative discoveries and breakthrough science.
Project Process and Outcomes
Project Processes
- Workshop cum panel discussion with national funding agencies: A one-day workshop-cum-panel discussion in hybrid mode was organized in collaboration with the Department of Science and Technology (DST), Government of India, to deliberate on the research assessment practices used by India’s national funding agencies. Participants were scientific administrators from the Department of Science and Technology, Department of Biotechnology (DBT), Council of Scientific & Industrial Research (CSIR), Science and Engineering Research Board (SERB) and the Indian Council of Medical Research (ICMR). SCOPE Framework was used to understand the current practices and their strengths and weaknesses. The findings emphasized the need for an India-centric research evaluation framework that integrates the disciplinary contexts. More details of the event can be found here.
- Workshop on Assessment Practices in Indian Research Ecosystem: A two-day joint workshop was organized by the DST-CPR, IISc and the Indian National Young Academy of Sciences (INYAS) with INYAS members and alumni as participants. The participants deliberated on the strengths and weaknesses of the current research assessment practices in India. The discussions and recommendations of the workshop can be accessed here.
- Personal interviews were conducted with representatives from research institutions, academia, and funding agencies. The respondents ranged from early career researchers to senior professors who are also members of various research evaluation committees and scientific administrators of Indian funding agencies (DST & DBT). The purpose of these interviews were – (a) to understand research assessment practices in various institutions and funding agencies within the Indian research ecosystem and their strengths and weaknesses, and (b) to understand the perspective of researchers at different levels of their careers on how responsible the current research assessment practices are.
- Surveys: With the objective of collecting and collating more information about how research assessment is being carried out in institutions, we also initiated two surveys – one for the STEM researchers and another for academicians and researchers in the agriculture discipline. A total of 30 responses were received through the two surveys.
Challenges Encountered and Their Possible Solutions
- Institutional bureaucracy has been a challenge as workshops with institutions/ research councils, funding agencies, etc., have taken much longer than anticipated to be cleared through the official process. Moreover, as things are again gradually moving offline with the decline in covid-19 cases, the institutions are actively trying to clear their backlogs in holding outreach activities, making it harder to get their availability as expected.
- Shift from online to hybrid workshop mode: As the Covid-19 cases were declining as the project activities picked up pace, the identified stakeholders were more interested in a physical workshop for better engagement and participation. Because of this unanticipated development, one workshop was modified to hybrid/physical.
Since physical workshops need a higher budget, we co-organized them with the Department of Science and Technology, Ministry of Science and Technology, Government of India to share the cost and access the physical infrastructure required to host the same.
Outcomes of the Project
Outreach and engagement
One of the major objectives of the project was to initiate a discussion about the necessary reforms in assessment practices across the research ecosystem in India. We have been able to engage a multitude of stakeholders in the Indian research ecosystem – early and mid-career researchers, faculties from research and academic institutions, senior academicians and researchers, science administrators, and grants management teams at funding agencies.
The workshops were a success in that regard as the science administrators acknowledged the necessity of reassessment of assessment practices and taking definitive steps towards it. However, considering the magnitude of the research ecosystem in India, such activities need to be expanded to include more data points. This will help to capture a clearer picture of the research assessment system.
Output
- Developed project website for outreach purposes
- Published a popular article – “Does the Way India Evaluates Its Research Doing Its Job?”
- Workshop report (with INYAS) – “Research Assessment In India: What Should Stay, What Could Be Better?”
- Workshop report (with funding agencies) submitted and currently under consideration in the Journal of Science Policy and Governance.
Findings – Research assessment practices in India
Within the institutions, the variability is much higher. For example, state and central universities are largely dependent on quantitative metrics, while institutions like IITs have a mix of both, though leaning towards a more qualitative assessment. IISc has adopted a completely qualitative, peer-reviewed process with internal and external (including those from outside India) peer reviewers.
Funding agencies in India have adopted a more qualitative approach, especially beyond the initial screening of applicants. The degree varies between different departments (DST, DBT, SERB, CSIR, ICMR, ICAR, etc.) and their efficiency in the process. The primary benchmark for research assessment is the judgment of research proposals and CVs based on expert committee opinion. However, the challenges identified are diversity of committee members (in terms of experience, inclusivity, disciplinary diversity, industry, and civil society representation), capacities in handling large numbers of proposals (for funding agencies), the personal bias of the committee, lack of awareness about open science practices, and societal impacts of research.
On the other hand, the assessment practices of the agriculture research system (Indian Council of Agricultural Research, ICAR), CSIR, Universities, and a majority of research institutions of national importance are primarily based on quantitative measures. Research institutions, especially for internal promotions and grant allocation, depend on quantitative metrics, with a few qualitative aspects. Still, various activities are considered in the process, not just numbers/ impact factors of publications. In the case of universities, evaluation criteria for promotions are very much focused on publication-based metrics.
Future Prospects
Plans for continual improvement and adaptation
This project at this point stands at an interesting junction. The initial findings are exciting, especially since quite a few institutions and funding agencies employ a more qualitative approach in research assessment. Expanding the understanding of their approaches, challenges and opportunities will be helpful not only to recommend how they can be improved; but have the potential to be case studies for the research ecosystem globally who are looking forward to adopting a more qualitative approach of research assessment. However, to reach such a stage the whole study needs to be expanded with substantial numbers of respondents. The current study design also needs some refinement, though the basic structure can remain the same.
Future plans
We plan to apply for funding to expand this study and the initial finding will serve as the background material for a possible larger project proposal.
About the authors
Suchiradipta Bhattacharjee is an STI Senior Policy Fellow hosted at the DST- Center for Policy Research, Indian Institute of Technology, Delhi.
Moumita Koley is an STI Post-Doctoral Policy Fellow at the DST-Center for Policy Research, Indian Institute of Science, Bangalore.