Perspectives

One of the more bizarre but dramatic episodes in scientific publishing took place earlier this year when the journal Nature disowned an important paper it had published on contamination of native Mexican maize by genetically modified corn from the United States. The paper, "Transgenic DNA Introgressed Into Traditional Maize Landraces in Oaxaca, Mexico" (Nov. 29, 2001), made for hot news around the world as the first scientific demonstration that such a transgression had actually occurred. SPP covered the political and scientific roots of the controversy in its #13 offering. In this companion piece, we look at the highly curious peer review aspects of the story.

Nature had a rough time handling this paper and wound up frustrating and infuriating almost everyone connected with the controversy, as well as embarrassing itself. It set kind of a precedent in the scientific editing process when it instructed the authors of the paper, Berkeley microbiological ecologists David Quist and Ignacio Chapela, to return to the lab well after the paper appeared so that they could perform additional experiments to verify that they really demonstrated what they had claimed. It was a precedent-setting act because journals never, to anyone contacted by SPP anyway, insist on additional research by authors of a paper once the paper is actually published.

Yet, that is what Nature did despite the fact that the paper passed original review by three initial referees. This is a story worth attempting to make some sense of, even though the lessons are clearly of the evolving kind.

One thing worth thinking about concerns the nature of peer review on subjects swollen with political and commercial agendas. How, the ancient question goes, can journals best quarantine bias, sloppiness, and malice from its peer review processes? And did Nature succeed in safeguarding peer review integrity in this latest case? The other question has to do with whether Nature actually advanced the cause of publishing in this prickliest of scientific and technological realms-genetic engineering.

A deeper perspective on these questions emerges from a narrative of the events leading up to and following publication, as revealed through some correspondence SPP carried on with a couple of principals-Nature executive editor Maxine Clarke, and one of the coauthors of the paper, David Quist. Outside peer review is a moderately studied subject in policy scholarship (the Journal of the American Medical Association devoted its June 5, 2002, issue to the subject). But the issue here boils down to whether a journal should assume the function of an author's research director after the author's paper sees print. The answer is probably yes when the journal needs to correct a mistake it made by appointing referees who proved not to be quite up to their job. In Nature's case, the journal was required to resort to multiple referees to right the ship.

This corner concludes that Nature, through all the stress and sweat it underwent, through its questionable selection of referees, did in fact advance the dialectic about sifting good science from the bad. In this case the standards defended were those of molecular biology. Whether Nature harmed its reputation is almost a moot point because it is generally presumed to be the best scientific journal in the world and publication there is always the goal of researchers on the move. Even Nature's critics in this case seek to be published in Nature.

Nevertheless Nature was vilified by pro-Quist and Chapela (QC hereafter) forces for siding with industrial interests. One criticism that stung Nature more than any other was the suggestion that an advertising brand partnership with a handful of biotechnology companies was a force leading the journal to ostensibly side with those who attacked Quist and Chapela. Nature, of course, finds odious any charges that it sides with anyone when it comes to publishing research.

But this caper also shows that it and other journals probably need to open the process a whole lot more, especially identifying referees to make them far more accountable, and by widening exposure of pre-published manuscripts to a broader community of assessors. There is already too much covert sniping and undermining in various fields of science and, of course, probably too many tie-ins between industrial interests and academic researchers, as has already been documented for some areas of clinical research. Also, journal publishing is much too interlaced with other media of technical communication and propaganda, many of which themselves promote powerful commercial interests. It is the quality journals that must adhere to standards of constancy and thus ought to take a hard look at the virtues of sanctimonious but subversive anonymity of referees. Who are these people anyway, the public might ask.

To review the paper's controversy ever so briefly, QC submitted a manuscript early last year to Nature, reporting the sensational finding that genetically modified corn had transgressed into native maize varieties in Mexico. This was shocking news because the Mexicans revere their maize; it is the native source of the world's corn and they don't like to see its genes messed with by products slotted with altered genes. Genetically modified (GM) corn seed is outlawed for planting in Mexico, as are field trials of new GM varieties. The U.S. biotech industry would like to set up shop in Mexico but can't as long as GM corn is seen as a contaminant. The industry believes it is not a contaminant but instead a gift from God (who, in this context, is the science they do) and actually makes for fuller and more biologically diverse species. Ecologists and organic farmers as a whole dismiss that perception. It is a monumental debate.

Here now is the publication story, based on a log kept by co-author David Quist. Early in 2001 Nature received the QC paper and began the traditional peer review process by sending it off to three referees for their assessments. Two of the reviewers immediately gave approval, but the third wanted more evidence. He or she said proper controls were lacking: QC should have done tests on presumably uncontaminated maize samples collected from periods prior to the development of GM corn. QC said all right, did the controls, and in early October sent in the additional data that were then given approval by all three reviewers.

QC had already gone public with their findings that became the talk of all factions in the fractious biotech world. Nature knew it and decided to give the paper extra attention. Not only would QC light the skies with the paper, as Q and C were hoping, but Nature, too, would share in the glow. (Some scientific journals compe quite briskly for dominance, none more than Nature and its American rival Science.)

Nature then sent the revised paper to another set of reviewers. The journal was already fidgeting over the prospects of publishing the paper. One of Nature's editors said to QC in a letter that "intense media interest" placed the paper in a special category that would put it "under close scrutiny." It should therefore "contain all of the data and methodological details needed to satisfy knowledgeable and potentially skeptical readers."

The new set of reviewers all sanctioned the revised paper and it was published on November 29 as a "Letter to Nature," "after," according to Quist, "eight months and at least four rounds of peer review."

Within two weeks Nature began receiving severe and rigorous criticisms of the QC paper. The critics said essentially that QC used faulty techniques and went absurdly far in drawing any conclusions from what they purportedly found. In private, and in comments circulated through the pro-industry Internet site Agbioview, QC's critics used far more loutish language. QC's supporters on the whole were silent during this time, but began organizing themselves once they were aware of the heavy artillery directed at QC and the mischievous role the pro-industry website AgBioView was playing in undermining QC.

Nature, itself under a shelling, then began to have second thoughts about what it did and asked QC for more "urgently needed" data to further substantiate their claims. None of the critics took any issue with the claim that some transgression of native maize had at least been strongly suggested. The main objections were directed at QC's more sweeping declaration that transgression was more widespread. Such evidence was lacking in the chromosomal detail demanded by the critics.

Nature, convinced by the critical responses that the paper was faulty, told QC to go back to the lab and perform an additional but routine test known as the Southern blot, on the earlier samples. QC had three weeks to do it, submitted some results that they themselves believed were not convincing, and asked for a little more time as they continued working away at reducing background "noise" in the analysis.

The story becomes somewhat tangled at this point, but on March 18 QC informed Nature that they did finally confirm "hybridization" of the test samples of maize and prepared a new report for the assessment by reviewers. They felt they had satisfied all relevant objections to their original findings and therefore hoped Nature would include those results in their response to the criticisms that were to be published in Nature's "Brief Communications" section, which itself is peer reviewed.

Nature gave QC's new results to, in its words, "an independent reviewer," a molecular geneticist who found their data less than compelling. Nature told QC, therefore, that they could not agree to publish their new findings. QC were convinced that the referee did not really address the new data but essentially repeated the critique of earlier results. To QC the process seemed rigged against them.

Nature, likely aware that its own reputation was at risk, backed down and allowed QC to include in their response letter the new results. On April 10, the full set of communications was published along with a box by editor Philip Campbell saying that the original paper never really merited publication because of insufficient data. As some apparent attempt at grasping a Golden Mean however, he did make the point that Nature was not retracting the paper and that it remained a citable publication.

The whole episode was a unique event in the history of technical publishing. No journal editor SPP spoke with had ever recalled a journal asking for additional data after a paper already had been published. Usually it is quite enough for a paper, once accepted, to be published but to await criticisms that the authors are free to rebut. After that, the arguments are usually advanced by further laboratory work by the same and other scientists. That was what happened earlier this year when Science published a roundly criticized paper on cold nuclear fusion by Oak Ridge National Laboratory physicists. Its editor, Donald Kennedy, said in an editorial that Science's job was to publish innovative research, however novel and unpopular, and leave the rest up to the community to refute it or verify it.

But why did Nature walk those extra miles?

SPP engaged Nature's executive editor Maxine Clarke over this whole squabble in a series of e-mail exchanges over the last few weeks and, while cautious and conscious of preserving the confidentiality of her journal's editorial processes Clarke did imply that the QC event was a highly unusual event in Nature's 133-year history.

"The high policy implications of this paper," she says, "did mean that we were very careful in considering it. It went through several rounds of review. It was also one reason why we decided to publish QC's response to the technical criticisms even though our usual policy is not to publish responses if the referees find them unconvincing as they did in this case.

"I also think that there is some truth in the point that when a journal publishes a paper in which there is a very high degree of public interest and scientific controversy, scientists with strong extreme interests scrutinize published papers more intently than they would otherwise do for other types of papers because they are very motivated to find any flaws which can be used to undermine or support the conclusions of the paper."

"What was different for Nature in the QC case was, one, that we allowed QC to add new data as they claimed it would settle things once and for all and two, the editorial note of clarification [by Nature's editor in chief Philip Campbell] published with the response. Usually, if a response is judged inconclusive, it is not published at all. If it had turned out that the original paper was definitely artifactual...we would have asked the authors to retract that part of it, or if they would not, imposed a retraction by publishing one ourselves. But because the matter is still unclear, we took the course we did which seemed to us to be the most open and the most useful to our readers."

Clarke calls the criticisms of the QC paper "on the extreme end of the distribution level of response." She says "these things are always time-consuming to deal with when they are happening, but they are all different and hence are both stimulating and 'learning experiences' for us. Nobody is suggesting that there is no transgenic maize in Mexico, whatever the flaw in QC's...technique. I hope these experiments are repeated by another lab and published in a peer-reviewed journal soon, so we can see whether the conclusions do in fact stand up or not."

It is not the nature of a journal such as Nature to admit to any but the most normal of human frailties. Clarke, speaking for editor-in-chief Campbell, believes Nature performed its duties fairly and well during an especially untidy political episode. At least two of QC's critics, Nicholas Kaplinsky at Berkeley, and Matthew Metz, previously at Berkeley, had run-ins with them during an especially vitriolic fight on campus over a research agreement with Syngenta, the Swiss biotechnology firm. Clarke is quick to point out that ramification.

"One very pertinent fact in the QC affair," she says, "is that all the protagonists, published in Nature at least, are or were at Berkeley, and the affair is part of the bigger argument over Syngenta's sponsorship of the research institute there. I think this fact is muddying the waters or generalizing the QC situation."

In the end, everyone got in their swings. Chapela and Quist got their findings published in the journal they were aiming at, their opponents gave them the bashing they thought Q and C deserved, QC got their new findings reported and thus their data solidified, and Nature learned something more about how to be a better journal. And everyone wound up unhappy.

What has especially shaken Nature were the contents of letters it published on June 27 charging the journal acting on behalf of the biotech industry in discrediting the QC paper. Nature is part of the multi-journal Nature Publishing Group, the unit that manages the financial functions of the publishing enterprise. As such it is always seeking added revenue, such as selling to industry sponsorship of the many side ventures promoted by the publications. These include conferences, special issues, reprints, and web portals. How, Nature's critics suggested, could Nature truly claim editorial independence while its advertising arm was constantly seeking closer financial ties with industry?

Clarke acknowledges at least part of the problem. "We were subject to accusations and comments that haven't previously been made against us," she says. "particularly in respect of our own commercial conflict of interest. Although we are well used to other sorts of criticism, it is usually more directly about the scientific issue at hand and our handling of it, rather than us being accused of having vested commercial interests that drove our decision-making.

"I think it had not really occurred to the editors, before QC came along, that anyone would seriously think that we, the editors, had such an interest because of NPG's sponsorship program in other contexts. I am sure we will think carefully about how to communicate our undoubted editorial independence more effectively in the future."

Nature Editor Philip Campbell has called the episode "embarrassing" and a "technical oversight," which is about as self-effacing as a journal of such standing can be. But Nature adheres, nevertheless, to the conviction that in the end it did the right thing. As Clarke puts it, "Our on the record position is that we have been honest throughout and we are not ashamed of that. In fact, we are proud of it. It is really about the kind of thing we do all the time, listening to the scientific community and our readers, and trying to 'do the right thing' for science and its communication."

As for what the whole hassle means for peer review, one factor does emerge as something to think and talk about when science and politics coil through the processes of technical communications in the way they did in the QC mêlée. And that involves questions of thoroughness and openness.

Nature essentially admits it could have done better by its initial referees (leaving out the debate over whether their initial judgment in favor of publication was right after all). Some fields of science, especially physics, often send pre-publication manuscripts via the Internet out to a broad range of researchers for early evaluation and debate. These are not official referees but they do contribute to an all-important screening process that does feed back to the authors and the publishing journal.

Certainly, widely circulated pre-publication versions of the Nature paper would have opened the referee process up more widely. The outstanding model of collective, non-paper peer review is the pioneering arXiv.org site at Cornell University developed by physicist Paul Ginsparg in the mid-1990s while he was at Los Alamos National Laboratory. Web-based peer review and publishing is a big story in itself for what it means for the future of paper publication and its impact on advertising and exclusivity. Moreover, the whole trend toward pre-publication circulation conflicts with the desire of journals to restrict advance notice of important results for the sake of exclusivity. It also, as Ginsparg has so often pointed out, threatens the economics of independent journals with their obvious dependence on paper and revenue through reprints, advertising, and the like. These are highly debated issues in scientific publishing today.

In any case, Nature came close to asking that QC retract their paper but pulled back and merely regretted publishing it. It stands as a citable paper. Whether the ecologists and molecular biologists finally do decide to work together and assemble richer sets of data on gene transgression in Mexico is the question here. The last word is that Mexican government laboratories have confirmed by more rigorous methods QC's findings. But this paper is currently undergoing peer review.

Return to Perspectives