New JAMA Study Adds To Picture of How Researchers Skew Results to Get Favorable Outcomes

Following hot on the heels of my previous post about researcher malfeasance comes a new a new study just published in JAMA which illuminates the persistence of the discredited technique where researchers redefine study endpoints to come up with results that satisfy the research's sponsors.

The study is Comparison of Registered and Published Primary Outcomes in Randomized Controlled Trials Sylvain Mathieu. JAMA. 2009;302(9):977-984.

It's goal was to see whether requiring researchers to register their study designs publicly made a significant difference in how the results of the research was reported.

The reason for requiring registration was the tendency of drug company researchers to change the "endpoints" they reported on if the originally selected endpoints did not come up with data favorable to their company's drugs.

To give you an example of how this worked. A drug company would commission a big expensive study to see if Drug A prevented heart attacks. The results four years later would show that Drug A had no impact at all on heart attacks. At this point the researchers would then hunt through their data until they found evidence that Drug A decreased the incidence of hangnails in female study participants aged 45-50. The final publication of the study would be a research paper title, "Drug A Effective Against Hangnails." The more important finding that the drug was useless for the purpose the company had tested it for would never be published.

To counter this abuse, it was proposed to require that researchers register their studies when they began, making it clear what their primary endpoints were supposed to be. In our example, the primary endpoint would be heart attack reduction.

This latest study finds that only 45.5% of the studies published after the supposed adoption by the mainstream research community of the requirement to register studies were adequately registered.

Of those, 31% "showed some evidence of discrepancies between the outcomes registered and the outcomes published." In addition, "The influence of these discrepancies could be assessed in only half of them and in these statistically significant results were favored in 82.6% (19 of 23)."

The researchers in this study conclude, "Comparison of the primary outcomes of RCTs [random controlled trials] registered with their subsequent publication indicated that selective outcome reporting is prevalent."

In short, researchers are ignoring the registration requirement and journals continue to publish studies that ignore or obfuscate the study's original primary endpoints.

And, even more importantly, exactly how badly the results are being massaged and tweaked is impossible to tell from the way these studies are published.

When will peer reviewers start getting serious about reviewing these studies and demanding that researchers live up to the ethical standards that are, supposedly, shared by the research community? For example, that researchers divulge to peer reviewers the original reason the study was undertaken and publish the results whatever they might be.