Cancer Research Replication Problem

An eight year project that tried to reproduce the results of key cancer biology studies has finally concluded and its findings suggest that similar to research in the social sciences, cancer research has a replication problem. The effort to replicate nearly 200 preclinical cancer experiments that generated buzz from 2010 to 2012 found that only about a quarter could be reproduced. ¹ Researchers with the Reproducibility Project Cancer Biology aimed to replicate 193 experiments from 53 top cancer papers published from 2010 to 2012. But only a quarter of those experiments were able to be reproduced. ²

Substantial inefficiency in preclinical research may be hampering the drug development pipeline later on

The researchers couldn't complete the majority of experiments because the team couldn't gather enough information from the original papers or their authors about methods used, or obtain the necessary materials needed to attempt replication. What's more, of the 50 experiments from 23 papers that were reproduced, effect sizes were, on average, 85 percent lower than those reported in the original experiments. ¹ The lessons of the project suggest that substantial inefficiency in preclinical research may be hampering the drug development pipeline later on, says Tim Errington, who led the project. As many as 19 out of 20 cancer drugs that enter clinical trials never receive approval from the US Food and Drug Administration. Sometimes that's because the drugs lack commercial potential, but more often it is because they do not show the level of safety and effectiveness needed for licensure. All of this should not come as a surprise. Way back in 2005 John Ioannidis reported in that most published medical research findings are false. His statistical analysis and logic were impeccable and his paper has never been seriously refuted. Furthermore, he has had a tremendous impact: the paper has been viewed more than 2.5 million times. ³ In 2009, Dr. Marcia Angell, former editor of the New England Journal of Medicine, wrote that, “It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgment of trusted physicians or authoritative medical guidelines. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as an editor.”⁴

---

As researchers have a harder time generating useful results, they become increasingly desperate and prone to confirmation bias and fraud

Science has become less reliable, Ioannidis asserts, because competition among researchers for publications, grants, tenure and other rewards has intensified. As researchers have a harder time generating useful results, they become increasingly desperate and prone to confirmation bias and fraud. “Much research is conducted for reasons other than the pursuit of truth,” Ioannidis writes. False positives and exaggerated results in peer reviewed scientific studies have reached epidemic proportions in recent years. Many studies that claim some drug or treatment is beneficial have turned out to not to be true. Even when effects are genuine, their true magnitude is often smaller than originally claimed. ⁵ All of this means there is a lot of medical misinformation. The Institute of Medicine estimates that only 4 percent of treatments and tests are backed up by strong scientific evidence: more than half have very weak evidence or none. The solutions that Dr. Ioannidis suggests: more access to complete data sets, more independent investigations, and more transparency.

References

1. Tara Haelle, “A massive 8-year effort finds that much cancer research can't be replicated,” sciencenews.org, December 7, 2021
2. Timothy M. Errington et al., “Reproducibility in cancer biology: challenges for assessing replicability in preclinical cancer biology,” .org, December 7, 2021
3. John P. A. Ioannidis, “Why most published research findings are false,” PLOS Medicine, August 30, 2005
4. Dr. Jason Fung, “The corruption of evidence based on medicine— killing for profits,” lewrockwell.com, June 18, 2018
5. John P. A. Ioannidis, “An epidemic of false claims,” Scientific American, June 1, 2011
6. Shannon Brownlee, Overtreated, (New York, Bloomsbury, 2007), 92