Poor data and code sharing undermine open science principles

“Formalistic policies and prevalent policy-practice gaps in high-profile medical journals undermine data and code transparency, necessitating a supportive publication ecosystem that empowers authors to uphold scientific responsibility and integrity.”

This is the message that emerged from a study titled “Formalistic data and code availability policy in high-profile medical journals and pervasive policy-practice gaps in published articles: A meta-research study”, prepared by a group of 28 researchers at various Chinese universities and published in Accountability in Research: Ethics, integrity and policy on 25 March.

In the study the authors noted: “Nowadays, scientific research credibility is increasingly questioned due to the widely discussed ‘reproducibility crisis’, which underscores the inability to reproduce many published findings.

“In response, data and code sharing has emerged as a vital solution as it facilitates independent verification and critical scrutiny of scientific results, fostering transparency and reproducibility to enhance both the credibility of individual studies and trust in the broader scientific enterprise.”

In contrast, poor data and code (DAC) sharing undermines open science principles. It is the “stringency of DAC availability policies in high-profile medical journals” and the “policy-practice gaps in published articles” that the study evaluates.

Three of the study authors, Zhiyi Chen, Wei Li and Jingxuan Zhang, in a joint statement sent to University World News, explained further:

“The study highlights critical gaps in data and code sharing practices and policies in high-profile medical journals, revealing significant challenges to achieving transparency and reproducibility in scientific research.”

Study outcomes: the ‘critical gaps’

According to the study article: “Although many medical journals claim to embrace and support open DAC sharing initiatives and ethics, the adoption of data-sharing policies remains far from achieving open science goals, with no significant progress observed over the past decade, despite the implementation of institutional or stakeholder-driven data-sharing statements.

“Even in leading medical journals, the proportion of articles indicating an intention to share data remains relatively low, with less than 10% reporting such intentions in surgical and orthopaedic journals.

“Furthermore, even in journals with policies mandating DAC sharing, 98% of research data and 99.5% of code [is] not shared, highlighting a stark gap between declared policies and actual publication practices,” the article emphasised.

“Despite these profound challenges, there remains limited understanding of how journal policies are implemented to promote DAC sharing and how authors comply with DAC-sharing practices in their research.”

Pervasive policy-practice gaps

The study reported that the analysis of 3,191 articles from leading high-profile medical journals – The BMJ, JAMA, NEJM and The Lancet – revealed pervasive policy-practice gaps, with 33.7% lacking DAC availability statements, 23.3% refusing DAC sharing (58.4% without justification), and 39.0% of publicly shared DAC being inaccessible or invalid.

The study pointed out that only 0.5% of articles achieved full computational reproducibility. Over 90% of analysed articles exhibited policy-practice gaps, with missing availability statements, unjustified refusals to share DAC, and invalid repository links being common issues, according to the study.

The study showed that among 931 Q1 (top tier) medical journals, only 9.1% mandated both DAC sharing and availability statements, with 70.6% lacking mechanisms to verify authenticity and 61.2% allowing publication despite invalid sharing.

The research also indicated that journal impact factors were positively correlated with the stringency of availability statement policies but not with sharing policies, indicating a focus on formal compliance rather than substantive practice.

“Despite the growing emphasis on open science, the study reveals that most high-profile medical journals adopt formalistic DAC policies that lack enforcement mechanisms, leading to significant policy-practice gaps,” Chen, Li and Zhang noted in their statement.

The study explained further: “Although DAC availability policies are intended to promote openness, transparency, and better science, our findings indicate that these goals are not being fully achieved. Compared to the motivation behind declaring DAC availability policies, actual efforts to mandate or enforce these practices on authors have been deprioritised.

“Our meta-research revealed that journal policies fail to ensure scientific integrity in open science practices. Among medical journals, fewer than one-tenth mandated that authors share DAC alongside a clear availability statement,” the study noted.

Bridging the gap between policies and practices

These results offer “several take-away messages for higher education policymakers to foster transparency, reproducibility, and scientific integrity”, the authors statement noted.

For higher education policymakers, the study emphasises the importance of fostering a culture of transparency by embedding open science principles into academic and research ecosystems.

“These findings underscore the need for systemic reforms to address the disconnect between policy and practice,” the authors told University World News.

“To address the problems identified in the paper, a systemic overhaul of data and code sharing practices is essential, requiring collaboration among journals, authors, institutions and policymakers.

“Institutions should mandate training on DAC sharing and open science practices [such as research ethics and reproducibility] for students and researchers should be integrated into graduate and postgraduate curricula, ensuring they are equipped with the skills to meet stringent DAC requirements.

“Policymakers should also incentivise responsible DAC sharing through recognition programmes, such as open science badges or career advancement credits, to encourage compliance and reward transparency,” which would further motivate researchers,” the authors statement pointed out.

“Furthermore, higher education institutions should advocate for standardised DAC sharing policies across journals and disciplines, aligned with frameworks like the Transparency and Openness Promotion guidelines and FAIR principles… with clear consequences for non-compliance.” This, the authors said, “will reduce ambiguity and ensure consistency”.

“Additionally, universities should establish robust oversight mechanisms, such as appointing data editors or creating dedicated offices for data management, to ensure adherence to best practices,” the statement suggested. This would empower researchers to meet stringent DAC requirements.

“By implementing these measures, policymakers can help bridge the gap between journal policies and actual practices, fostering a culture of accountability, transparency, and reproducibility in research,” the statement emphasised. This will also ensure that DAC sharing becomes a cornerstone of scientific integrity and public trust in academia.

“Ultimately, higher education institutions must lead by example, embedding open science into their core values and practices.”

Challenges and solutions

Chen, Li and Zhang said: “Implementing measures to address the challenges in data and code sharing will face several significant obstacles.” These challenges include resistance from stakeholders, technical and logistical barriers, and a lack of standardised enforcement mechanisms.

Journals, authors and institutions may resist change due to entrenched practices, concerns about increased workload, or fear of exposing errors in research. Authors may hesitate to share DAC due to fears of data misuse, breaches of participant privacy, or competitive disadvantages.

Additionally, many journals lack the technical infrastructure or expertise to verify DAC sharing, and the absence of standardised policies across disciplines creates inconsistencies and confusion.

“To address these challenges, a multifaceted approach is necessary,” the authors suggested.

First, “addressing authors’ concerns is critical. Journals and institutions should provide clear guidelines on protecting participant privacy, mitigating data misuse, and ensuring proper attribution”.

Second, “addressing barriers like privacy concerns or competitive disadvantages through innovative solutions, such as synthetic datasets or secure sharing platforms, is equally important [and] can alleviate privacy concerns while maintaining data utility”.

Third, “technical and logistical barriers must be overcome by investing in infrastructure and expertise, such as reliable repositories and automated tools for verifying DAC accessibility, [which] is also essential to support researchers”.

“Finally, collaboration among stakeholders is vital,” the authors stressed in their statement.

“Policymakers, journals, and institutions must work together to develop enforceable policies, provide training on DAC sharing, and allocate resources for oversight.

“By addressing the challenges systematically, the measures proposed in the study can be effectively implemented, fostering a culture of transparency, reproducibility, and scientific integrity.

“This holistic approach will not only enhance research credibility but also prepare the next generation of researchers to uphold the principles of open science,” Chen, Li and Zhang concluded.