The Chinese Academy of Sciences

The major R&D Outcome-Oriented Evaluation System (2011-2021/22) represented a move away from a strong reliance on simple quantitative indicators towards recognizing and rewarding significant research contributions. Its key elements included:

• Focus on Major Research Outcomes: The primary emphasis was on the quality, impact, and significance of research outputs rather than just the quantity of publications or other easily quantifiable metrics. Institutes were expected to define and strive for "major breakthroughs" and contributions, focusing on their unique positioning, three major basic, strategic, and prospective science and technology innovative achievements, and five key potential directions (what was called the ‘One-Three-Five’ plan). Six types of potential major research and development outcomes were identified, including solving major scientific problems, creating new research fields, achieving breakthroughs in key technologies, providing solutions, achieving remarkable socio-economic benefits, and providing significant and influential advice.
• Five-Year Evaluation Cycle: The system operated on a five-year cycle, aligning with national five-year plans and the CAS's strategic planning.
• Multi-Stage Assessment: The evaluation process involved several stages:
  • Expert Diagnostic Assessment (mid-term): International experts were invited to diagnose the institutes' status and to evaluate the quality and technical value of their main research areas to help improve internal management and clarify core advantages. This was a formative assessment aimed at guiding the institutes.
  • Overall Performance Evaluation (end-term): Domestic experts provided qualitative opinions on an institute's performance compared to its five-year targets. Quantitative indicators (funds, projects, staff, outcomes, patents, awards, international exchanges) were provided as references for these experts. The results of this evaluation were directly linked to incentive resource allocation. Superior major breakthroughs merited significant financial competitive rewards.
  • Monitoring of Key Performance Indicators (KPIs): Annual monitoring of KPIs was conducted to observe and track the institutes’ research performance.
• Limited Role of Simple Quantitative Metrics: While quantitative data was collected and used as a reference for expert panels, the final evaluation was heavily reliant on expert judgment of the significance and impact of the major research outcomes.

Introduced in 2022, the new Mission-Oriented Evaluation System signifies a further evolution in the CAS’s approach to research evaluation. It retains the focus on “major research output”, indicating that the philosophy of prioritizing impactful research contributions remains central, and shifting towards a more continuous monitoring and a broader consideration of factors.

Recognizing that a primary focus on quantitative metrics such as the number of publications, citations, and honorific titles had considerable adverse effects on the progress of science and technology in China, CAS adopted the Major R&D Outcome-Oriented Evaluation System around 2011-2012. This shift, aligned with the ‘Innovation 2020’ CAS strategy, aimed to prioritize high-quality and high-impact research outcomes that would lead to significant contributions in areas such as solving major scientific problems, achieving breakthroughs in key technologies, and providing solutions with remarkable social or economic benefits, rather than simply focusing on the quantity of publications. The focus for the new system was on serving society, the country and the economy, in addition to science itself.

In 2022, the Mission-Oriented Evaluation System was adopted, reflecting a desire for more continuous oversight and a broader understanding of institutional performance. The annual monitoring system allows for more frequent feedback and enables institutes to make adjustments more readily than under a five-year cycle. This can foster a culture of continuous improvement and greater responsiveness to evolving needs.

The evolution of research evaluation systems at the CAS, exemplified by the adoption of the Major R&D Outcome-Oriented System currently being furthered by the Mission-Oriented System, reflects a dynamic and adaptive approach to research management. The initiation of CAS’s own evaluation system in 1993 was a direct response to the perceived inadequacy and misunderstandings arising from external evaluations. Since then, new systems were developed to better align with the CAS’s evolving mission and national needs.

The CAS Headquarters played a central role in proposing and authorizing the changes in evaluation systems, reflecting its management responsibilities. The development process involved learning from international experiences, including German Max Planck Institute Assessment, and how the UK’s Research Excellence Framework (REF) defined impact. The development of each system included the definition of specific indicators and methodologies tailored to the objectives of that phase. For instance, the initial systems in the early 90’s heavily relied on quantitative indicators, while later systems incorporated peer review and expert assessments.

The results of evaluations have been linked to significant consequences, such as funding allocation and the appointment of institute directors. This creates a strong incentive for institutes to respond to the evaluation criteria and adapt their practices. The shift to influencing extra funding in the mission-oriented system represents a nuanced change in how evaluation results are used to manage performance.

The engagement with the central government regarding the reforms, and the adoption of the major R&D outcome philosophy at the national level, indicates a strategic effort to align CAS’s evaluation practices with broader national science and technology policy. The concept of focusing on major research outputs has had a broad impact, being recognized and adopted in other parts of the Chinese research system.

The Major R&D Outcome-Oriented Evaluation System was developed and launched around 2011, following the announcement of the ‘Innovation 2020’ strategy and the ‘One-Three-Five’ plan. The system was in place for approximately ten years, encompassing two full five-year evaluation cycles (roughly 2011-2015 and 2016-2021/22).

The Mission-Oriented Evaluation System started implementation in 2022, and it is in its early stages of implementation, with the first couple of years of annual monitoring completed.