September 1, 2009

New JAMA Study Adds To Picture of How Researchers Skew Results to Get Favorable Outcomes

Following hot on the heels of my previous post about researcher malfeasance comes a new a new study just published in JAMA which illuminates the persistence of the discredited technique where researchers redefine study endpoints to come up with results that satisfy the research's sponsors.

The study is Comparison of Registered and Published Primary Outcomes in Randomized Controlled Trials Sylvain Mathieu. JAMA. 2009;302(9):977-984.

It's goal was to see whether requiring researchers to register their study designs publicly made a significant difference in how the results of the research was reported.

The reason for requiring registration was the tendency of drug company researchers to change the "endpoints" they reported on if the originally selected endpoints did not come up with data favorable to their company's drugs.

To give you an example of how this worked. A drug company would commission a big expensive study to see if Drug A prevented heart attacks. The results four years later would show that Drug A had no impact at all on heart attacks. At this point the researchers would then hunt through their data until they found evidence that Drug A decreased the incidence of hangnails in female study participants aged 45-50. The final publication of the study would be a research paper title, "Drug A Effective Against Hangnails." The more important finding that the drug was useless for the purpose the company had tested it for would never be published.

To counter this abuse, it was proposed to require that researchers register their studies when they began, making it clear what their primary endpoints were supposed to be. In our example, the primary endpoint would be heart attack reduction.

This latest study finds that only 45.5% of the studies published after the supposed adoption by the mainstream research community of the requirement to register studies were adequately registered.

Of those, 31% "showed some evidence of discrepancies between the outcomes registered and the outcomes published." In addition, "The influence of these discrepancies could be assessed in only half of them and in these statistically significant results were favored in 82.6% (19 of 23)."

The researchers in this study conclude, "Comparison of the primary outcomes of RCTs [random controlled trials] registered with their subsequent publication indicated that selective outcome reporting is prevalent."

In short, researchers are ignoring the registration requirement and journals continue to publish studies that ignore or obfuscate the study's original primary endpoints.

And, even more importantly, exactly how badly the results are being massaged and tweaked is impossible to tell from the way these studies are published.

When will peer reviewers start getting serious about reviewing these studies and demanding that researchers live up to the ethical standards that are, supposedly, shared by the research community? For example, that researchers divulge to peer reviewers the original reason the study was undertaken and publish the results whatever they might be.


Anne said...

Is there anyway we can know which studies changed the outcome goals? I wonder what percent of studies are just buried when the results are not what the researchers or pharmaceutical company wants.

Scott S said...

The concept of "peer-reviewed" is largely dead, as far as I'm concerned. Not only are the medical journals no longer published by medical associations but by for-profit publishers (such as Elsevier, Wiley/Blackwell, and Wolters Kluwer to name a few of the giants) and the editorial staff are increasingly paid journalists, not scientific or medical researchers, but the execution of clinical trials for clinical trials is increasingly being managed by paid third-party (for-profit) companies who are supposed to provide objective management. My personal feeling is that 2 things need to happen before these issues can be resolved: #1) all medical students should be required to attend a semester-long ethics course and #2) the "publish-or-perish" mentality in many academic institutions needs to be revised. But I won't hold my breath waiting on either!!

Unknown said...

Off topic, but check out this idea for an apparently continuous BG monitor.

Anonymous said...

What's scary is that in the UK NICE

and the PCTs (kinda like local government HMOs) insist that doctors use "evidence based" treatments.

Yet the quality of "evidence" is often so poor and biased - for example well controlled diabetics are generally eliminated from studies, so there is no "evidence" that we exist, and doctors can actually be disciplined for suggesting a low carb diet for that reason.

All power to the likes of you, Michael Eades and an increasing band of bloggers for doing what the Peer Reviewers should have done pre-publication.