Jamison Chang, MD, medical officer, is a board-certified internist with over 15 years of clinical experience with a broad range of disease entities in both the ambulatory and hospital settings. After completing his residency and chief residency at UNC Chapel Hill, he obtained additional training in nephrology as well as a master’s degree in clinical research (MS-CR). These experiences allow Dr. Chang to meld clinical pragmatism with scientific rigor to help plan and conduct high quality clinical trials. 

“Stop being so negative about things” or “if you had a less negative attitude, things would go better for you.”  No matter the setting, there tends to be a strong distaste for negativity within our culture.  One notable exception to this is in pursuit of scientific progress.   Here, at least in theory, a negative finding (something does not work) should garner as much attention as a positive finding (something works).  Ideally, a scientific discipline then takes the balance of both supporting and refuting evidence to decide on the current state of knowledge.  This is science at its best.  When the scientific community falls short of this ideal by failing to consider the entirety of scientific evidence, the quality of scientific research can fall.  Lower quality research can result in poorly informed policies, wasted resources and in the extreme cases, harm.  While not the only scientific discipline affected, clinical research has well-documented cases of not devoting equal attention to both negative and positive findings.  The remainder of this post will discuss supporting evidence for this claim, attempt to explain why this may be occurring, explain the effects of this phenomenon on the clinical research enterprise, and then offer a few solutions on how we can start righting the ship.

The asymmetry in reporting positive and negative outcomes was recently highlighted by Aaron Carroll, MD, a professor of pediatrics who recently published an editorial in the NY Times (Sept 24,2018) titled “Congratulations Your Study Went Nowhere.”  Dr. Carroll cites a recent study in Psychological Medicine whose purpose was to explore possible biases in medical research related to antidepressants.  This group evaluated 105 antidepressant studies registered with the FDA. Half of these studies were “positive” and half were “negative” according to the FDA. Whether a study is generally declared positive or negative is based whether or not the study achieves or does not achieve the primary outcome or primary goal.  In these depression studies, a common primary goal is often to determine if there is an improvement in depression using commonly accepted scales. Notably of the 105 trials reviewed, 98% of the positive trials were published while only 48% of the negative trials were published.  Studies may also look at other outcomes, so called secondary outcomes or endpoints. In the case of depression trials, some examples of secondary outcomes may include hours of sleep or change in weight.  Clinical trials are unable to provide similar levels of statistical certainty regarding secondary vs primary outcomes.  Rather, these secondary outcomes are used to generate hypotheses for future trials.   Despite this well-accepted convention, the study in Psychological Medicine noted 10 of 25 trials considered negative by the FDA were reported as positive by researchers who appeared to shift the focus from a negative primary outcome to a favorable secondary outcome.   Dr. Carroll also cites a 2004 JAMA study where researchers reviewed more than 100 trials approved by a scientific community in Demark that resulted in 122 publications and over 3000 outcomes.  Half of these outcomes on whether drugs worked were not reported and two-thirds of cases of possible harm were not reported.

This is not to cast a negative light on scientists or entities that do not fully report outcomes in clinical trials.  While there may be instances of deliberately not reporting certain findings, underreporting of outcomes likely derives from a collective “Eh” regarding negative trials from the clinical research collective.  Biomedical journals, grant funding agencies and biomedical scientists seem to lean more favorably toward studies that demonstrate an effect versus those that don’t show one.  Pick up any major medical journal and this phenomenon will be readily apparent.  Go to any major medical conference and you will witness hundreds of posters showing positive results to every 10 showing negative results.  Journals more often bolster their reputation by publishing breakthrough articles that more often demonstrate the effectiveness of a new therapy rather than a lack of effectiveness.  There is something inherently more attractive about reporting positive results than negative results in the current clinical research environment.  Unfortunately, this is doing a disservice to the quality of clinical research as a whole and potentially limiting our ability to further improve the health and well-being of patients.

Selective reporting has major implications for the current clinical research enterprise.  Starting with the most obvious implication: if more positive results are reported than negative results, new therapies or devices may actually be less effective in practice than in published literature supports.  There are major financial implications to this, with both insurers and payers utilizing resources on these therapies that might be better apportioned elsewhere.  Underreporting of negative trials and/or outcomes also greatly hinders one of the most critical aspects of scientific research: learning from both the past successes and the failures of other scientists. If this knowledge is not widely available to the scientific community, we are more likely to repeat the same mistakes, utilizing scarce resources inefficiently and hindering future scientific progress.

Given the high stakes of underreporting the results of negative trials and outcomes, how might we go about addressing these issues?  Many governing bodies including the Food and Drug Administration (FDA) in the US have mandated registering clinical trials at ClinicalTrials.gov.  The requirements have evolved over the years but most of the requirements focus on disclosing details of the design of the trial including the primary endpoints, secondary endpoints and analysis plan.  The intent here is for more transparency so that stakeholders can validate whether a trial was carried out properly.  Inadequately conducted clinical trials can lead to erroneous conclusions about the effectiveness (or lack thereof) of a product. In other words, trials may be falsely negative (the new therapy may be effective but errors in trial conduct obscured this effect) or falsely positive (the new therapy does not work but improper trial conduct results in it appearing better than it actually is). The FDA further tightened reporting requirements in 2017 with the FDA Final Rule 42 CFR 11 (the National Institutes of Health (NIH) published similar regulations).  The results of these regulations are encouraging. From 2007 to 2017, major university reporting when from around 28% to 78% with a 20% improvement (58 to 78%) between 2015 and 2017 (1). Increased regulation has been helpful but other changes are cultural.  Fundamentally, we need to celebrate and encourage the reporting of important negative results as we do positive results.  We should implore journals to publish negative results so that the clinical research community can learn from and improve upon what has been done previously.   Reporting negative results may seem less newsworthy and to some, boring, but if that is the price of better science and better treatments for patients, maybe we should consider boring over bling?

The scientific method is an enduring achievement that continues to benefit us.  However, the magnitude of this benefit is significantly curtailed when this method is not employed as it was intended.  In the case of clinical research, there appears to be an inherent bias toward popularizing and publishing things that “work” in lieu of things that didn’t “work”.  This asymmetry of accentuating the positive not only potentially leads to erroneous conclusions about current therapies but also impacts the direction and success of future biomedical research.   We need to urge culture change within the clinical research space.   “Accentuating the negative and not just the positive” might be an appropriate mantra for this culture change as we move forward emphasizing the need to place both positive and negative findings on equally footing.  In this way, we maximize our ability to obtain the best possible answers to our research questions and hopefully deliver the greatest benefits to patients.

Acknowledgements: Many thanks to Dr. Aaron Carroll whose editorial in the NY Times helped clarify my thinking on this issue.  Thank you to Dr. Jack Modell for further clarifying the important issues.

1. Piller C and Bronshtein T.  Faced with public pressure, research institutions step up reporting of clinical trial results. STAT Jan 19, 2018.