Why I let other people read all of my research notes

Open notebook science (ONS) is about nothing less than making the entirety of your research public. It has the potential to provide an answer to some of the major problems we are facing in science and especially in cognitive neuroscience.

We are living in the age of the reproducibility crisis. In a survey from two years ago, 90% of researchers attested that there is at least a slight reproducibility crisis in science.

In my field, cognitive neuroscience, the shock came when a large consortium attempted to replicate the results of 100 studies in social and cognitive psychology in an Open Science Collaboration published in 2015. They were only successful in 36% of cases.

Reasons for failure to replicate experiments range from misconduct and wrong methods to inadequate sample sizes and publication bias. Some of these have been shown to be effectively reduced when experiments are pre-registered (that is, when researchers submit a study protocol to a journal before a study begins).

My rather radical approach is to make every aspect of my work public, including my notes about ideas and certain versions of my code. What does this mean in concrete terms? As a cognitive neuroscientist, my work involves running experiments and analysing data. When I analyse data, I use an electronic notebook (for instance, an R Markdown or Jupyter) to create documents with code embedded in the text that explain my analysis.

Every change I make to analysis documents or to experimental scripts is tracked by a version control system called git, which is publicly shared on GitHub. This is accompanied by general posts on ideas and progress of the projects I am currently working on.

As a result, the whole process of my work can be scrutinised. If people want, they can understand both the intellectual and the technical progression of my work. The standard I want to reach here is that other researchers are able to replicate and build upon my work without any further help.

Being able to understand the evolution of a project as well as having access to all the files are key for long-term comprehensibility of research. Paired with pre-registration and adequately powered experiments, this is the highest level of reproducibility.

Speeding up dissemination of research

Apart from ‘just’ improving the quality of scientific output, what is in it for me? I can convince sceptics about my work, can answer questions by referring to posts and I will be prepared to deal with the infamous reviewer two – a reference to the twitter joke about the reviewer who criticises a good piece of research because it isn’t the piece they would have written. Also, it will make it easier for people to collaborate with me.

Importantly, it allows other people to catch my mistakes before publication. One of the biggest advantages is the fast dissemination of my work. Social media enables me to quickly share my results with people researching similar topics. Quite often, we do work that only a small handful of people are directly interested in. This is the target audience for my open notebook and it means that my non-significant results don’t only end up in a file-drawer.

Good science

What are the possible downsides? Writing notes so that other people can understand them is undeniably time consuming. However, the most important future collaborator is yourself and it is important that you understand your own work and the decisions you made. Thorough note-keeping is therefore important for good science, whether or not you decide to make them public.

You might also fear that your work gets scooped. While this might be true to some extent, ONS timestamps any changes made to documents which will help to defend my work. In addition, I can start off privately and go public when I think it’s appropriate. Don’t forget, even if my work is completely public, people still need to cite me.

Beyond that, data sharing, even though highly recommended, can be tricky because we have a duty to protect our participants as well. Only fully anonymised data with the appropriate permission from the participants can be shared.

In my experience, people find my work and approach helpful and inspiring, which is rewarding. For me personally, the ‘open’ part in ONS is not the most difficult part. It is keeping a meticulous record of what I do and organising code that other people can understand alongside getting work done.

Therefore, the amount of notes and the timeliness of their publication is up to those researchers who use it and this will vary. Under no circumstance should adopting ONS be another source of stress and pressure. To avoid any confusion, ONS, although it will radically change the way we do science, should not replace but complement peer-review publication in the same way pre-prints do.

Jörn Alexander Quent is doing a PhD in biological science at the MRC Cognition and Brain Sciences Unit at the University of Cambridge, United Kingdom, where he is a Gates Cambridge Scholar.