Peer-censorship and scientific fraud

The major scientific journals are often regarded as the touchstones of scientific truth. However, their reputation has been tarnished with yet another major scientific fraud unfolding over South Korean researcher Hwang Woo-suk’s peer-reviewed and published Stem Cell research. Should the publication of these results be viewed as simple ‘mistakes’, a crime by a deviant individual, or a broader conspiracy aided by lax reviewing and journal oversight? Blogs were apparently instrumental in uncovering the inconsistencies in Hwangs publications. Here I look at peer-censorship in environmental sciences and its role in concealing scientific waste and fraud, and uncover the emerging solutions from pre-print archives and blogs.

The paper here: “Refereed Journals: Do They Insure Quality or Enforce Orthodoxy?” by Frank J. Tipler makes an interesting introduction to the topic of peer censorship. This contains many gems and instances of peer censorship as experienced by Nobel laureate and equivalent scientists.

Today, the refereeing process works primarily to enforce orthodoxy. I shall offer evidence that “peer” review is NOT peer review: the referee is quite often not as intellectually able as the author whose work he judges. We have pygmies standing in judgment on giants.

Censorship occurs when a few people manage to create an embargo on dissenting results, usually via the peer review process itself. When this is the case, all (excepting possibly the reviewers) are the worse for it, as interesting, critical, and potentially exposing scientific papers have not been published. Does peer-censorship occur in climate and environmental research field? Here is a comment made on the science policy blog Prometheus by Hans von Storch, “Reflections on the Challenge”, (emphasis is mine).

The debate about the hockey-stick is technically not really relevant. We have archived our main goal, namely that the premature claims that the issue of millennial temperature reconstructions was mostly solved have been broadly rejected. One or two years ago it was hard to publish results which were inconsistent with the MBH reconstruction; now everybody agrees that there may be more to it. The jury is still out and I expect that consensus will settle on something with significant larger variations in the shaft of the hockey-stick.

Having said this – the debate about the hockey-stick is most significant when it comes to the culture of our science. Posting the hockey-stick as key evidence in the SPM and Synthesis Report of the IPCC was simply stupid and evidence for what Bray calls post-sensible science – as science which is encroached by moral entrepreneurship. Or post-normal science. We have more cases of this type of claims-making, which is usually a mix of “good” political intentions and personal drive for the limelight. Have we, as a community, become better in rejecting such claims? I am afraid, we have not.

Posted by: Hans von Storch at November 24, 2005 10:56 AM

Here is another similar experience recounted by Robert Kaufmann on RealClimate in response to, “Naturally Trendy?”, #60 , (emphasis is mine).

I would like to pick up on a comment made by per (#58) about testing GCM’s against real-world data. As an outsider to the GCM community, I did such an analysis by testing whether the exogenous inputs to GCM (radiative forcing of greenhouse gases and anthropogenic sulfur emissions) have explanatory power about observed temperature relative to the temperature forecast generated by the GCM. In summary, I found that the data used to simulate the model have information about observed temperature beyond the temperature data generated by the GCM. This implies that the GCM’s tested do not incorporate all of the explanatory power in the radiative forcing data in the temperature forecast. If you would like to see the paper, it is titled “A statistical evaluation of GCM’s: Modeling the temporal relation between radiative forcing and global surface temperature” and is available from my website http://www.bu.edu/cees/people/faculty/kaufmann/index.html

Needless to say, this paper was not received well by some GCM modelers. The paper would usually have two good reviews and one review that wanted more changes. Together with my co-author, we made the requested changes (including adding an errors-in variables” approach). The back and fourth was so time consuming that in the most recent review, one reviewer now argues that we have to analyze the newest set of GCM runs – the runs from 2001 are too old.

The reviewer did not state what the “current generation” of GCM forecasts are! Nor would the editor really push the reviewer to clarify which GCM experiments would satisfy him/her. I therefore ask readers what are the most recent set of GCM runs that simulate global temperature based on the historical change in radiative forcing and where I could obtain these data?

Comment by Robert K. Kaufmann — 22 Dec 2005 @ 12:33 pm

The above experiences by respected researchers in the field show that censorship of unpopular views is real and present. While the strangulation of science by peer censorship is bad enough, by ensuring that dissenting the voices require a much higher level of persistence and scholarship than the consenting voices, it has an enabling relationship with scientific fraud. A notion to be disabused is that peer review is adequate for the scientific community to police itself. Most recently on ClimateAudit in, “The Hwang Affair: A Chronology”, http://www.climateaudit.org/?p=476.

I’ll comment some more on Nature and Science procedures. The unveiling of the fabrication was not done by their peer reviewers or even by Western scientists. All of the heavy lifting in detecting the fabrication was done in Korea, mostly reported on blogs. I obviously think that disclosure is one of the best and easiest ways of making fraud harder. From my experience with paleoclimate, neither journal applies “best practices” in respect to archiving data and methods and both journals have been unresponsive or ineffective in responding to requests for data. So they are vulnerable to criticism. More on this later.

Filed under: Disclosure and Diligence — Steve McIntyre @ 8:46 am

While the importance of disclosure in uncovering fraud is clear, concealment is a symptom of many syndromes other than fraud. Superficial explanations for peer-censorship — conservatism, or laziness — do not speak to the causes. Protecting conservative publications as a means to gain and maintain personal power is a more cogent motivation. A contemporary treatment based in understanding of power relations would probably argue that power is entwined with journals and reviewers in a flawed and inefficient process exploited by ambitious individuals. These are uncomfortable facts of life that scientists prefer to ignore and the general public is largely unaware.

To a large extent the problem has been exacerbated by journals having exclusive control over so many aspects of science: distribution, protection of intellectual property, quality control, archival procedures and commentary. Splitting up these functions would (and is) doing a great deal to break up this control. The scientific community would do well to make use of alternatives that are increasingly available.

For example, pre-print services such as http://arXiv.org provide an archive of unpublished but citable texts, providing the protection of intellectual property of the authors. If these were the main form of primary deposition of manuscripts (as they appear to be becoming in Physics and Mathematics) then the onus would be on the broader community to evaluate the value of an article, and not place faith exclusively in passing peer review. Journals could compliment pre-print archives by providing a range of value added services, such as degrees of review, replications, and commentary to meet a variety of needs for quality control.

Some journals have adopted new peer review processes utilizing pre-print archives. “Peer Review or Peer Censorship?” by William A. Dembski, here describes a two-step process for submissions to the journal Progress in Complexity, Information, and Design (PCID):

(1) Having met basic scholarly standards and being relevant to the study of complex systems, the paper will be accepted into the ISCID Archive. (2) Once in the ISCID Archive, the paper will be accepted for publication in PCID provided at least one fellow of ISCID signs off on it.

Finally the burgeoning of blogs will continue. They will be the place for people to turn for depth of opinion on scientific worth. I don’t ascribe to the Utopian vision of every scientist with a blog, but clearly blogs support a range of modalities from simple ‘What’s New’ abstract services, to ‘cornering the market’ on commentary over an area of expertise. Together with Trackback ping technology to allow deep relevant cross-referencing between blogs, and the support contributors, blog modalities have immense value to other scientists in the field, for education and outreach to the general community, to aid understanding of complex and nuanced topics, to uncover waste and fraud, and in combination with pre-print archives, sidestep attempts at peer-censorship.