Grant: CIHR Fellowship

Some research findings are unreliable. When scientists repeat an experiment, they often obtain different results from what has already been published. The individual biases of researchers and the incentive to publish impressive findings may drive this situation. To reduce researcher bias, experimental designs that blind both participants and experimenters have become the gold-standard across various disciplines. Data analysts, however, are rarely blinded. When data analysts are not blinded, they may make analytic choices-either intentionally or without explicit awareness-that favour the results they desire. Our experiment tests whether blinding data analysts makes published research less biased and more reliable. We will compare published findings between researchers who conduct a blinded data analysis and those who conduct a non-blinded analysis. To establish these groups, we have partnered with an organization that manages a longitudinal health and lifestyle dataset that over 200 researchers request to access each year. We will provide half of the researchers with the full dataset, as is typically done (i.e., the control group). The remaining researchers will receive only 10% of the data at first (i.e., the 'blinded' group). Researchers in the 'blinded' group will develop an analysis plan based on this partial dataset and be asked to upload it to a permanent and publicly accessible repository. At this point, we will provide the full dataset for the researchers to run their 'blinded' analysis. We anticipate that the blinded group will publish results that are more reliable and less biased. If this is true, our study will have discovered an underused method to reduce bias across research disciplines. Moreover, we will have established a method of implementing blinded data analysis that is scalable across health research. In our increasingly data-driven world, it is important that researchers get the right answers, not simply the answers we wish to be true.