June 3, 2020 -- More than 140 scientists and doctors are challenging the validity of an influential study that found the antimalarial drugs hydroxychloroquine and chloroquine were more likely to cause death of COVID-19 patients in hospitals. After the study's results were published in The Lancet on May 22, the World Health Organization temporarily suspended its own clinical trial of hydroxychloroquine.
An open letter to the observational study's authors and The Lancet's editor-in-chief, posted May 28, lists 10 concerns. The writers, who identify themselves as "clinicians, medical researchers, statisticians, and ethicists from across the world," say the researchers failed to account sufficiently for things that may have influenced their results, including how severe different cases were, and say the study lacked an ethics review and possibly had errors in the underlying database.
They also say the study's authors are being too secretive about their data sources and methods, even though The Lancet signed a pledge to support data sharing during the coronavirus pandemic.
"This paper has had a really negative impact on clinical trials," said James Watson, PhD, a statistician at the Mahidol-Oxford Tropical Medicine Research Unit in Thailand and the lead signatory on the open letter. "A lot of decisions [about hydroxychloroquine] have been made on the basis of very poor evidence. This drug could be harmful, it could be beneficial, it could do absolutely nothing at all, but we need a randomized trial."
The Lancet study is based on data from the medical services company Surgisphere of about 96,000 hospital patients diagnosed with COVID-19 from December 20 to April 14 from around the world. Every patient was discharged by April 21, unless they had died by then. The majority of patients, 81,144, did not receive antimalarial drugs. The remaining 14,888 patients began to receive chloroquine or hydroxychloroquine within 48 hours of their positive diagnosis, either alone or with an antibiotic.
After controlling for numerous things, including age, race, sex, and ailments like cardiovascular and lung disease, the authors found that patients on the drugs were twice as likely to die in the hospital as patients who did not receive them (18% death rate for patients who received hydroxychloroquine, 16.4% for chloroquine, 9.3% for those who did not receive either). Patients who also received antibiotics had even higher mortality rates.
"We were fairly reassured that, although the study was observational, the signals were robust and consistent across all regions of the world in diverse populations," lead author Mandeep R. Mehra, MD, of Brigham and Women's Hospital in Boston, previously told Medscape Cardiology about the results.
Severity-of-Disease Levels Needed
Watson's central critique is that The Lancet authors did not properly separate patients by how severe their disease was. The way the study authors determined disease severity, the level of blood oxygen, and rates of suspected infection did not identify significant differences between people who received the drugs and those who did not. Watson feels that another parameter, the ratio of blood oxygen pressure to oxygen-enriched air, would have revealed differences between the two groups.
Antimalarials are generally provided to the sickest COVID-19 patients, Watson says, after other options have failed. The difference in death rates, he suggests, could appear because patients who received hydroxychloroquine or chloroquine were sicker than others to begin with.
The study authors say that they adjusted for 35 potential variables and stand by their work while recognizing its limits. "The authors leveraged the data through Surgisphere to provide observational guidance ... the authors have underscored the importance and value of randomized clinical trials and articulated that such trials will be necessary before any conclusions can be reached," Mehra said in a statement. The study's second author, Sapan S. Desai, MD, is CEO of Surgisphere.
Previous critiques posted online to the scientific forum PubPeer noted differences between the official number of deaths reported in Australia and the observational study's count. This past weekend, The Lancet published corrections to the paper, changing one hospital's location from Australasia to Asia and including some raw data that was omitted in the initial publication. The correction notice states: "There have been no changes to the findings of the paper."
Susan Ellenberg, PhD, a biostatistician at the University of Pennsylvania's Perelman School of Medicine in Philadelphia, says she sees no reason to halt clinical trials of the effectiveness of hydroxychloroquine for COVID-19 treatment on the basis of The Lancet study. "When you're looking for treatment effects, whether it's effectiveness or safety, your best answer is going to come from randomized trials," she says.
Ellenberg notes that the World Health Organization's decision to pause enrollment in its hydroxychloroquine study is temporary, and that enrollment could resume based on a review of its own safety data. She is hopeful the medical community will have a more solid understanding of effective treatments for COVID-19, based on the results of randomized trials, in the next 2 or 3 months.
Another concern highlighted by the open letter is that average daily doses of hydroxychloroquine in the observational study were 100 mg higher than FDA recommendations, although 66% of the study's data was from North American hospitals. In a tweet thread predating the open letter, Watson also said the wide range of antimalarial doses -- the average dose for chloroquine was 765 mg per day, for example -- make the results hard to interpret.
The study data is reported at a global level, with some details provided by continent but not for individual hospitals. The 671 hospitals that gave data to Surgisphere are not named, and the authors did not make the statistical code behind the study publicly available for others to use and critique. "The authors have not adhered to standard practices in the machine learning and statistics community," the open letter states.
The open letter asks for aggregated patient hospital-level data to be made public and for an independent analysis of it to be overseen by the World Health Organization. In his statement, Mehra says he and his coauthors will do "an independent academic review of the data."