January 27, 2016

The New England Journal of Medicine promotes research secrecy

Charles Murray comments: "NEJM editors: Bullshit. If your data can't be shared, along with coding documentation, you've got something to hide"

Another Facebook commenter: "My favorite line: "other researches might "even use the data to try to disprove what the original investigators had posited." This is so stunning I have no words to express it.

NEJM and JAMA are the two most prestigious American medical journals so what they do and say is widely noted.  But both journals still publish a lot of rubbish.  See e.g. here, here, here, and here.

So I was not totally surprised at the latest NEJM article excerpted below. They advocate abandoning one of the basic  safeguards of science:  data transparency.  Science exists on  trust.  If a scientist reports a set of findings, other scientists will always believe what he says.  But, with various levels of self-awareness, scientists will sometimes misrepresent their findings. And that is no mean problem.  Around two thirds of research findings reported in leading journals have been found to be "unreplicable".  In other words, other researchers doing  the same thing fail to confirm the original finding. So two thirds of what is reported is apparently wrong.

Not all the erroneous findings are conscious and deliberate fraud.  Most commonly, the problem is that the author takes a rosy view of what is in his data.  There is something in his data that suits his preconceptions so he reports that and ignores other information in his data that is contrary to his expectations.

So to be sure that his data has been fully and dispassionately analysed, a scientist has long been held responsible for making his data available to other analysts.  It's a basic safeguard.  And if a scientist refuses to make his raw data available that basically tells you all you need to know: His work is faulty and he knows it is faulty.

So, now that we know how serious the unreplicability problem is, some journals are taking steps to circumvent it -- such as asking authors to pre-register their hypotheses to defeat data dredging.  But NEJM is doing the opposite.  They want to make it harder to check on the soundness of a research report!  That is so amazing that I am inclined to apply the usual suspicion:  I am inclined to suspect that the authors of the article (Dan L. Longo  and Jeffrey M. Drazen) have things in their own research past that they don't want to see the light of day.

And the reasons they give for what they advocate are so specious as to border on the hilarious.  They fear, for instance, that other scientists might "use the data to try to disprove what the original investigators had posited".  But why fear that?  That possibility is what data openness is all about. 

And there is no doubt that closer scrutiny of many findings WILL "disprove what the original investigators had posited".  I have only once requested raw data from another researcher  -- a request that was refused -- but that was mainly because there was very often enough information in the statistics provided to show that the conclusions did not follow from the data.  I did and reported that often in my 20 years as an active social science researcher  and I still do it often on my blogs. I pointed to an example of it just yesterday.

So NEJM is in the position of defending crap science.

But why?  Are there any non-personal motives involved?  I suspect that it might have something to do with the battering Warmists have taken over their refusal to release their data.  And the reason for Warmist secrecy is plain.  The classic case was Michael Mann's "hockeystick" picture of climate history.  When he did inadvertently let  details of his data and methods leak out, skeptics showed that his procedures were so faulty that just putting random numbers through Mann's computer program would produce a "hockeystick".  Since we in fact live in a era of exceptional temperature stability (with year to year temperature averages differing by only hundredths of one degree), the Warmist claim that we live in an era of dangerous warming was always going to need heroic lies to support it.

And academics do generally support Warmism (It gives them a golden shower of research grants) so I think our medical authors may be wading in to give some skin of defensibility for the chronic Warmist secrecy.

The aerial view of the concept of data sharing is beautiful. What could be better than having high-quality information carefully reexamined for the possibility that new nuggets of useful data are lying there, previously unseen? The potential for leveraging existing results for even more benefit pays appropriate increased tribute to the patients who put themselves at risk to generate the data. The moral imperative to honor their collective sacrifice is the trump card that takes this trick.
However, many of us who have actually conducted clinical research, managed clinical studies and data collection and analysis, and curated data sets have concerns about the details.

The first concern is that someone not involved in the generation and collection of the data may not understand the choices made in defining the parameters. Special problems arise if data are to be combined from independent studies and considered comparable. How heterogeneous were the study populations? Were the eligibility criteria the same? Can it be assumed that the differences in study populations, data collection and analysis, and treatments, both protocol-specified and unspecified, can be ignored?

A second concern held by some is that a new class of research person will emerge — people who had nothing to do with the design and execution of the study but use another group’s data for their own ends, possibly stealing from the research productivity planned by the data gatherers, or even use the data to try to disprove what the original investigators had posited. There is concern among some front-line researchers that the system will be taken over by what some researchers have characterized as “research parasites.”

More HERE

Go to John Ray's Main academic menu
Go to Menu of longer writings
Go to John Ray's basic home page
Go to John Ray's pictorial Home Page
Go to Selected pictures from John Ray's blogs