J. Marvin Herndon's Peer Review Folly

Peer Review Folly: Independent Critique and Recommendations

J. Marvin Herndon
Transdyne Corporation

Can a major government self-appraisal really be objective? One independent scientist’s critique shows what’s wrong with peer review and how to fix it. But does anyone pay attention?

Introduction


From time to time a government might conduct a comprehensive review of a particular process and/or procedure and the report becomes the new “gold standard” for implementation of the particular process and/or procedure. But can self-appraisals really be objective? How often does a government really seriously question the very processes and procedures it employs? Even when public comment is solicited, generally only those insiders with vested interests make statements. Moreover, public comment is generally restricted to pre-report opinions that may or may not be given much weighed during government review. After the report is published, in my experience, the new “gold standard” is “cast in concrete”; comment and criticism is neither invited nor acted upon. But after publication, I submit, is the time for independent critique and recommendations.


On December 16, 2004, an individual in the White House to whom I had complained about the inequity of “peer review” sent me a copy of the Office of
Management and Budget’s “Final Information Quality Bulletin for Peer Review: December 15, 2004”. On December 26, 2004, I sent to the White House my critique of that Bulletin and my recommendations for systemic changes, which were neither appreciated nor implemented. Six years later, the U. S. Government still conducts peer review according to that Bulletin, and American science continues to decline. Have responsible officials never heard the phrase: “Find out what’s wrong and fix it”?


Critique of Bulletin


This Bulletin appears to have been crafted by individuals who either are extremely naïve of human nature or choose to ignore human nature. The Bulletin
appears to be predicated upon the tacit assumption that in all instances peer reviewers will provide honest, truthful reviews. The tacit assumption that peer reviewers will always be truthful leads to a principal flaw of this Bulletin, namely, the failure to provide any instruction, direction, or requirement either to guard against fraudulent peer review or to prosecute those suspected of making untruthful reviews. The long-standing failure of the federal government to require and to aggressively enforce truthfulness in the making of peer reviews (1) encourages and rewards those who make deceitful peer reviews and (2) can be expected to have and to have had a deleterious impact on America’s scientific capability, adversely affecting America’s technology, and, concomitantly, weakening America’s economic and military capability.


The Bulletin states, “… the names of each reviewer may be publicly disclosed or remain anonymous (e.g. to encourage candor).” The Bulletin approves the application of anonymity and even appears to promote some alleged virtue of its use, while being completely blind to its downside. If anonymity did in fact encourage candor and truthfulness, anonymity would be of great value in our legal system. There is in fact a historical record of the use of anonymity in courts: Anonymous testimony was used in the Spanish Inquisition and in nearly every totalitarian regime. In each instance, the results were the same: individuals denounce others, for a variety of reasons. Anonymity, instead of being a positive element, as implied by the Bulletin, is instead an extremely negative element, encouraging and rewarding the worst aspects of human nature and human behavior.


In the selection of participants in peer review, the Bulletin urges “due consideration of independence and conflict of interest” but, at the same time: (1) the Bulletin provides a highly restricted financial definition of conflict of interest, ignoring personal, professional, or scientific conflicts of interest, (2) the Bulletin falsely accords some individuals a conflict-of-interest-free-status, specifically, “…when a scientist is awarded a government research grant through an investigator-initiated, peer-reviewed competition, there generally should be no question as to that scientist’s ability to offer independent scientific advice to the agency on other projects”, (3) the Bulletin acknowledges that “the federal government recognize that under certain circumstances
some conflict may be unavoidable…” and, (4) the Bulletin completely prevents the avoidance of conflicts of interest by approving the use of anonymous reviews, a practice which is wide-spread in federal government agencies that support scientific research.


The Bulletin fails to provide any instruction, direction, or requirement either to guard against fraudulent peer review or to prosecute those suspected of making untruthful reviews. At the same time, the Bulletin approves of the use of anonymous reviews that are free from accountability and from civil recourse. The Bulletin gives tacit approval to circumstances that allow conflicts of interest and prevents the avoidance of conflicts of interest. With that combination, the Bulletin encourages and gives free rein to any criminal or quasi-criminal element that seeks to attain unfair advantage through deceit and misrepresentation in the anonymity-protected peer review system.


The Bulletin states that “reviewers may be compensated for their work or they may donate their time as a contribution to science or public service”. All instances with which I am familiar, specifically, NASA and the National Science Foundation, do not pay reviewers for their time. Reviewers do reviews for a variety of reasons, such as to curry favor with agency officials or to exercise control over their competitors, but the bottom line is that time is money and agencies often get for their non-compensation a hastily done, superficial review.


The most serious short-coming of the Bulletin is its failure to recognize or to admit the debilitating consequences of the long-term application of the practices described above. The application of anonymity and freedom from accountability in the peer review system and the openness to conflicts of interest, gives unfair advantage to those who would unjustly berate a competitor’s formal request for research funding. The perception – real or imagined – that some individuals would do just that has had a chilling effect, forcing scientists to become defensive, adopting only the consensus approved viewpoint and refraining from discussing anything that might be considered as a challenge to other’s work or to the funding agency’s programs. That is not science!


If a foreign power or a terrorist group had set out to slowly and imperceptibly undermine American science, I doubt that it could have devised a methodology for the purpose any more effective than the practices set forth in the Bulletin. Used for decades, these practices have diminished American science to the present point of approaching third-world status.


Recommendations for Systemic Changes in the Administration of Peer Review


Like the Critique, the following recommendations are specifically limited to the administration of peer review as applied to reviews of scientific grant/contract proposals that are not subject to secrecy considerations related to national security and defense.


Generally, conflicts of interest, in the broadest definition, should never be permitted and should never be tolerated in peer review. Federal regulations permitting same, such as those under which NASA operates, should be changed.


Anonymity should never be used in peer reviews. Reviewers should sign and swear their reviews “under penalty of perjury” and should be held accountable for the truthfulness of their reviews.


Reviewers should always be compensated, and compensated well, for making reviews.


An independent “ombudsman agency” should be created to address conflicts and disagreements between the individual or organization submitting the research proposal and the agency to which the proposal was submitted. The “ombudsman agency” should be empowered to bring potentially unlawful activity to the attention of the Department of Justice for investigation and possible prosecution.


Recipients of federal research grants/contracts should incur the responsibility of making some pre-determined number of reviews under the conditions described above. Like defense attorneys selecting a jury, each grants/contract recipient should be accorded a number of “pre-emptive strikes” so as to be able to remove himself/herself from certain specific proposal reviews.


An individual or organization submitting a research proposal should be presented with an official list of names that the funding agency proposes to be peer reviewers. Like prosecuting attorneys selecting a jury, the individual or organization submitting the proposal should be accorded a number of “pre-emptive strikes” so as to be able to remove certain specific proposed reviewers.


Implementation of the above recommendations will begin to correct the longstanding debilitation of American science. The above recommendations are intended to correct the short-comings displayed in the Bulletin. Additional managerial improvements are possible, but are not specified in the present document.


Download Bulletin:
http://cio.energy.gov/documents/OMB_Final_Info_Quality_Bulletin_for_peer_bulletin(2).
pdf

Return to Home Page NuclearPlanet.com