Gideon Meyerowitz-Katz is an epidemiologist and writer based in Sydney, Australia. His work covers chronic disease, the pandemic response, and more recently, error detection in science. In this op-ed, he discusses issues with research that have become increasingly apparent during the pandemic.

There are no two ways about it: Science is flawed. We’re not talking about the philosophical leanings of science or the origins of white coats and linoleum-floored laboratories, but about the nuts and bolts of the process by which we determine whether things are true or false.

In the decades before the pandemic, scientists spent endless hours wrestling with the painful fact that much of the knowledge base of science and medicine is reliant on research that is flawed, broken, or potentially never occurred at all.

Science has a gap between its mechanics and outputs. The mechanics of science are fine. The machines always get bigger and more efficient. New tools are always developed. Techniques become more sophisticated over time, and more knowledge is acquired.

The outputs of science are not. The culture of academia demands publication and warrants little retrospection about potential errors — this means that mistakes are rarely corrected, and even outright fraud is often left undetected in academic literature.

Enter the pandemic

And then along came a pandemic, and the gaps in science widened to an inescapable chasm. While biomedical research has had obvious and immediate success in COVID-19 mitigation, it has been accompanied by an enormous tidal wave of garbage, which instantly overwhelmed our garbage mitigation mechanisms.

From fraud to wasteful research to papers so error-filled that it is amazing that they’ve been published, the pandemic has produced a tidal wave of woeful scientific output that has, nevertheless, had staggering consequences for people’s lives.

Take ivermectin. It is an amazingly successful antiparasitic medication that has treated literally billions of people in the time since it was invented, and it has almost eliminated some parasitic diseases from the world.

It has also been globally promoted as a cure for COVID-19 by a group of passionate fans. It is likely that more ivermectin has been taken to prevent or treat COVID-19 than any other single medication, except perhaps dexamethasone.

And yet, we do not know if ivermectin is actually useful in the treatment of COVID-19 at all.

A recent review from the Cochrane collaboration — long considered the gold standard in medical research — concluded that ivermectin should not be used for the treatment or prevention of COVID-19 outside of well-conducted clinical trials, which is a stark contrast to the hundreds of millions of doses still being taken for those exact reasons.

How? 

In early 2020, people were desperate for any kind of treatment for COVID-19. A melange of partial evidence emerged.

This included: a laboratory study showing that the drug acted as a strong antiviral in a petri dish, a study in a French nursing home where the residents took ivermectin to treat a scabies outbreak and seemed to subsequently enjoy higher survival rates, and preprint reporting that ivermectin reduced the mortality from COVID-19 by 90%.

All three were weak evidence in different ways. Single in vitro studies are very poorly predictive of eventual clinical outcomes, and the nursing home paper was an accidental and uncontrolled observational study — what if the residents had never been exposed to SARS-CoV-2 in the first place?

The clinical study was entirely fabricated and later withdrawn from the preprint server, subsequent to great scandal.

The ivermectin story somehow got even worse from there. In late 2020, studies started popping up showing what can only be described as simply incredible results for the medication — a 90% mortality benefit or a 100% reduction in cases when used as a prophylactic.

After nearly a year, myself and other data sleuths demonstrated that many of these studies probably never happened, but the damage was well and truly done long before the first fake paper was retracted.

A meta-analysis of ivermectin, which is usually considered the gold standard of research practices, found a huge benefit for the drug. However, the paper has not been corrected, even though the studies underlying its results were found to be likely fraudulent.

In any other discipline — media, government, private enterprise — such an analysis would be taken down with apologies immediately. Instead, the paper is allowed to stand as a testament to the general disinterest of the scientific world in correcting errors.

This story could’ve been told very differently. Imagine a world where the initial laboratory paper came with a disclaimer, where the fraudulent preprint was looked on with skepticism immediately, and where the positive trials were assessed for fraud before they were even published.

Instead, at every stage, the process of highlighting concerns with data is ignored, with peer-review being the only flimsy barrier to publication for terrible research.

When we most needed effective fact-checking, our grand institutions of scientific research instead reviewed studies in a matter of days, if not hours, and posted fraudulent studies online to be shared across the world.

It’s tempting to say that research into ivermectin is uniquely flawed, but that’s clearly not true — realistically, it would be remarkable if a broken system produced only one failure.

Beyond ivermectin

Trials of favapiravir, another repurposed COVID-19 medication, have recently been retracted due to data concerns.

There are now nearly a dozen studies looking at whether vitamin D has a benefit in COVID-19 that have been corrected or withdrawn entirely over the last 18 months.

The website Retraction Watch keeps a running tally of the pandemic-related studies that have been retracted. As of publication, the figure is 199 and growing every week.

Even worse, those are just the papers that people have looked into. Errors in science are rarely noticed because there is simply no reward for pointing out other people’s mistakes.

If we were to start looking at all of the useless, wasteful, terribly done research, we might expand that number to thousands, or even tens of thousands of papers.

There are published ecological studies of ivermectin — where researchers compare entire countries’ drug use and COVID-19 mortality. These studies use mass drug administration protocols as their measure of the number of people who received ivermectin during the pandemic. This is despite those protocols mostly being disrupted or canceled early in 2020.

One study of vitamin D was retracted from the SSRN preprint server after it became clear that the authors had incorrectly labeled it as a randomized trial, though they had not randomized the participants at all. It has since been republished largely unchanged, with no mention of the previous retraction at all in the final paper.

Good science does exist

None of this is to say that there is no good science. The vaccine trials alone are perhaps the most impressive scientific work that has ever been done, with efficacious immunizations developed, tested, and trialed in under 1 year.

The RECOVERY and SOLIDARITY clinical trials, which looked at repurposed drugs to treat COVID-19, have almost certainly saved millions of lives during the pandemic.

The problem is that large, well-conducted clinical trials are far from the norm. In a recent systematic review of hydroxychloroquine for COVID-19, the median number of people enrolled per arm in clinical trials was 59 — one study looked at just two patients.

Without even carefully assessing these studies, we can say that most of them were probably a waste of time.

Indeed, if you look at the meta-analytic model from this review, virtually our entire knowledge of hydroxychloroquine for COVID-19 comes from just two studies, which recruited about 70% of all the people whom this drug had ever been tested on.

This is despite nearly 300 trials of the drug registered on clinicaltrials.gov, and the highest research spend of any single medication in the early pandemic.

If all of those tiny trials had been linked together, they may have achieved something useful, but instead, we’re left with two good studies and a smattering of largely pointless research.

All of this is, perhaps, the predictable outcome of a system that pushes publication above all else and punishes error-checking with disdain, scorn, and lawsuits. Publishing a terrible study can earn you praise and promotions; at worst, it might end up a line on your CV somewhere.

Checking studies for errors publicly earns you a steady payment of hate mail and death threats, and it nets you none of the citations, publications, and awards that academia regards as important.

Where does all this leave us?

Science has some enormous issues. Unless we can find a way to reward error-checking with actual money, we will continue to accept that a worrying proportion of our research output — the studies that we use to make life-and-death decisions — is either fake or incredibly problematic.

While it is tempting to think of this as a tedious problem among eggheads, that couldn’t be further from the truth.

It is not unlikely that you or your family have personally been impacted by bad research during COVID-19 — maybe you were given hydroxychloroquine during a hospital stay or took some metformin just in case. Perhaps you live in a place that reopened schools based on a study with mathematical errors or were told that masks constituted child abuse due to a paper that was later withdrawn.

Overall, there is a real impact of bad science in our everyday lives that the pandemic has thrown into stark relief.

Worse still, we know another pandemic is coming eventually. If we don’t fix these issues now, the next time a new disease spreads through our world, we will be doomed to repeat the mistakes of COVID-19. And that is perhaps the most worrying thought of all.

Source: Read Full Article