Introduction
The San Francisco Declaration on Research Assessment (DORA), published in May 2013, does not mention the term ‘open scholarship.’ And yet DORA and open scholarship are becoming increasingly entwined[1]. DORA’s ambition is to improve research evaluation practices but the practicalities of implementation make it impossible to separate the evaluation of research from questions about who and what research is for, who gets to be involved, and how it should best be carried out, all of which have to take account of the power dynamics that shape the scholarly landscape. Equally, progress towards open scholarship, which aims to make the products and processes of academic work as accessible to as many stakeholders as possible, requires changes in the ways that researchers and their research outputs and practices are assessed, incentivized and constructed. Here we examine the growing interactions between DORA and the open scholarship movement. By clarifying the alignment of the values and principles that underpin both endeavors, we see that they raise vital questions about equity and inclusion in research that must be central to reform within research organizations and the wider scholarly community.
What is open scholarship?
Unlike open access, for which there are three well-known definitions laid out in the Budapest, Bethesda, and Berlin statements and declarations, there is no universally accepted definition of open scholarship (sometimes referred to as open research or, more narrowly, as open science). It is commonly defined in terms of sharing scholarly products and processes, as in this statement from the University of Melbourne:
“Open Scholarship is the practice of applying principles of openness throughout the research and scholarly environment. This results in transparent and accessible knowledge that is shared and developed through collaborative networks. Practicing Open Scholarship means sharing research outputs (such as research protocols, methodology, code, data, and publications) as early as possible in the research process in a way that enables access and reuse by others.”
However, Chan and colleagues have challenged the view of open scholarship as “a set of technical infrastructure, workflow, protocols, and licensing conditions that can be universally applied regardless of context, history, and human agency.” From a global perspective that consciously embraces equity and inclusion they argue that “there is no single or universal concept of Open Science that is sufficient to encompass the diversity of knowledge traditions and practices from around the world.” It is clear therefore that there are many dimensions to open scholarship, but a core motivating idea is recognition of the value that comes from opening up the creation and sharing of knowledge in ways that are beneficial to the widest array of stakeholders, including the academic community, business interests, politicians, community leaders, and indeed all interested citizens. Beyond changing the ways that the various products of scholarship are shared, open scholarship challenges the present norms and power structures of academia.
How do DORA and open scholarship align?
Ideas of openness and transparency are woven into DORA’s recommendations for evaluators to be explicit about criteria for hiring, promotion, and funding decisions. The core principle of DORA is that scholarship should be evaluated “on its own merits,” rather than relying unduly on journal metrics. Just as importantly, although we recognize that research articles (and books and book chapters) remain a key focal point for research evaluation, DORA strongly advocates extending evaluation to include many other types of research output, including data, review articles, reagents, software, peer review, expert advice, training, new techniques, and technologies. This emphasis on the diversification of outputs synergizes strongly with the aim of open scholarship to make not just the final report of a piece of work openly accessible, but also the processes and materials (e.g., reagents, protocols, data, code) that go into it and come out of it.
This synergism between DORA and open scholarship goes further. For example, it is easier to evaluate a piece of work on its own merits if it is openly accessible; evaluators may resort more readily to third-party metrics if tasked with assessing work in journals to which their institutions do not subscribe. The open availability of associated data and code not only enhances the reliability of published work by exposing it to worldwide scrutiny, but also adds to the potential impact of research within and beyond the research community by facilitating use and reuse. DORA strongly believes that such impact, which open scholarship clearly aims to enhance, should be evaluated as part and parcel of research assessment. The infrastructure of open science, such as the use of persistent digital identifiers to outputs such as data and code, can help to ensure that they can be cited and recognized in research evaluation processes (though of course the value of such identifiers is only fully realized if the linked outputs are openly available).
The recent rise in the use of freely available preprints in disciplines beyond physics (e.g., bioRxiv, medRxiv, EarthArXiv, SocArXiv, or the Humanities Commons) is also a spur to journal-independent assessment of research because preprints oblige researchers to assess papers without any journal association. They have the further benefit of enabling researchers to cite work that is available to read on CVs and in funding applications well before it has been published in a peer-reviewed journal. For too long now, scholars have been prepared to delay publishing the results of publicly funded research because they know that our systems of evaluation place such a high premium on journal prestige or metrics. In doing so, the scholarly community has sometimes put itself in a position where it is acting against the public interest. DORA’s emphasis on taking the broad view of the value of research and scholarship is very much aligned with the desire of open scholarship to maximize the value and impact of new knowledge.
One of the key goals of open scholarship is to increase participation in research, not just by sharing all the various products and processes of scholarship with stakeholders from outside the academy, be they policy makers, community groups, or members of the public, but also by thinking anew about how to enable greater public involvement in the direction and prioritization of publicly funded research.
There is also a deeper and more important issue here, which has risen up the policy agenda in recent years: How inclusive is the academic community? Increasingly, researchers, university leaders, and policy makers in many countries are facing the problem of underrepresentation within the scholarly community. The historical exclusion and silencing of women, people of color, and other groups, which is maintained through daily experiences of sexism, racism, homophobia, and ableism, means that academia is not truly representative of the community that it frequently claims to serve. These historical biases, which persist in hidden or implicit forms despite the advent of equality legislation, present structural problems that still propagate through research assessment practices. Their impact can be seen today, for example, in the lack of equitable career advancement of women and underrepresented minoritized scholars. As a result, in clarifying its objectives earlier this year, DORA has emphasized equity and called for “broader representation in the design of research assessment practices that directly address the structural inequalities in academia.” The responsibility for opening up the academic community beyond its traditional membership lies within those who currently hold the power in the system, including tenured faculty and administrators. Although universities still often reflect the inequalities of the societies in which they are embedded, as progressive civic institutions they should be in the vanguard of challenging them. Inequalities within the scholarly community need to be resolved to legitimize the claims of academics to decide on prioritization of research that aims to tackle national and international challenges. Thus, questions of how to promote equity and inclusion are intimately bound with the need to reform research and research assessment practices so that they are fit for the 21st century.
Given the current dominance of Europe and North America in the global research landscape, broader questions of global equity also arise from efforts to open up scholarship and to reset international norms for evaluating research quality and impact. For example, business models for open access publishing that have been introduced for some disciplines in wealthy countries (e.g., article processing and data-storage charges) are unlikely to be viable for scholars in institutions or countries where research funds are more limited. Arguably, funder-led initiatives in Latin America, such as SciELO, which provide cost-effective open access solutions sensitive to institutional and national needs, have not always gotten the recognition their innovation deserves.
A further problem that drives global inequity is the dominance of university league tables driven by metrics based on journals, prestige, and income. These rankings largely favor research-intensive institutions in the global North and have driven a homogeneous view of what excellence looks like. University rankers therefore pose a threat to the rich diversity of institutions across the world; many universities and researchers in the global South serve vital regional needs, but are increasingly subject to pressures to conform to standards that they have had no voice in setting.
To address this, DORA has assembled an international advisory board to absorb perspectives from every continent. We believe not only that the products and practices of scholarship should be accessible from anywhere in the world, but also that the global community must be involved in developing the norms of open scholarship; for interesting perspectives, see work by Denisse Albornoz and others on the OCSDnet Open Science Manifesto, and by Leslie Chan and colleagues arguing for science that “can support cognitive justice and situations where everyone contributes knowledge, regardless of their country, social class, gender and language.” It would be tragic and self-defeating if moves to open scholarship or to reform research assessment reinforced rather than dissolved historical hierarchies that are rooted in discrimination and Eurocentrism.
DORA remains focused on enabling better and more equitable research evaluation, but clearly our work increasingly overlaps with open scholarship initiatives. We may be traveling along separate paths but ultimately DORA and the open scholarship movement have the same destination in mind: quality research that enriches all of humanity.
Anna Hatch is DORA’s Program Director
Ginny Barbour is DORA’s Vice-Chair
Stephen Curry is DORA.s Chair
[1] Examples of the growing entanglement of DORA and open scholarship can be seen in recent policy documents from the European Union, the Wellcome Trust, Charité Universitätsmedizin Berlin, and University College London.