 
J. Marvin Herndon's Peer Review Folly |
Peer Review Folly: Independent Critique and Recommendations
J. Marvin Herndon
Transdyne Corporation
Can a major government self-appraisal really be objective? One independent scientist’s critique shows what’s wrong with peer review and how to fix it. But does anyone pay attention?
Introduction
From time to time a government might conduct a
comprehensive review of a particular process and/or procedure and the report
becomes the new “gold standard” for implementation of the particular process
and/or procedure. But can self-appraisals really be objective? How often does a
government really seriously question the very processes and procedures it
employs? Even when public comment is solicited, generally only those insiders
with vested interests make statements. Moreover, public comment is generally
restricted to pre-report opinions that may or may not be given much weighed
during government review. After the report is published, in my experience, the
new “gold standard” is “cast in concrete”; comment and criticism is neither
invited nor acted upon. But after publication, I submit, is the time for
independent critique and recommendations.
On December 16, 2004, an individual in the White House to
whom I had complained about the inequity of “peer review” sent me a copy of the
Office of
Management and Budget’s “Final Information Quality
Bulletin for Peer Review: December 15, 2004”. On December 26, 2004, I sent to
the White House my critique of that Bulletin and my recommendations for systemic
changes, which were neither appreciated nor implemented. Six years later, the U.
S. Government still conducts peer review according to that Bulletin, and
American science continues to decline. Have responsible officials never heard
the phrase: “Find out what’s wrong and fix it”?
Critique of Bulletin
This Bulletin appears to have been crafted by individuals
who either are extremely naïve of human nature or choose to ignore human nature.
The Bulletin
appears to be predicated upon the tacit assumption that in
all instances peer reviewers will provide honest, truthful reviews. The tacit
assumption that peer reviewers will always be truthful leads to a principal flaw
of this Bulletin, namely, the failure to provide any instruction, direction, or
requirement either to guard against fraudulent peer review or to prosecute those
suspected of making untruthful reviews. The long-standing failure of the federal
government to require and to aggressively enforce truthfulness in the making of
peer reviews (1) encourages and rewards those who make deceitful peer reviews
and (2) can be expected to have and to have had a deleterious impact on
America’s scientific capability, adversely affecting America’s technology, and,
concomitantly, weakening America’s economic and military capability.
The Bulletin states, “… the names of each reviewer may be
publicly disclosed or remain anonymous (e.g. to encourage candor).” The Bulletin
approves the application of anonymity and even appears to promote some alleged
virtue of its use, while being completely blind to its downside. If anonymity
did in fact encourage candor and truthfulness, anonymity would be of great value
in our legal system. There is in fact a historical record of the use of
anonymity in courts: Anonymous testimony was used in the Spanish Inquisition and
in nearly every totalitarian regime. In each instance, the results were the
same: individuals denounce others, for a variety of reasons. Anonymity, instead
of being a positive element, as implied by the Bulletin, is instead an extremely
negative element, encouraging and rewarding the worst aspects of human nature
and human behavior.
In the selection of participants in peer review, the
Bulletin urges “due consideration of independence and conflict of interest” but,
at the same time: (1) the Bulletin provides a highly restricted financial
definition of conflict of interest, ignoring personal, professional, or
scientific conflicts of interest, (2) the Bulletin falsely accords some
individuals a conflict-of-interest-free-status, specifically, “…when a scientist
is awarded a government research grant through an investigator-initiated,
peer-reviewed competition, there generally should be no question as to that
scientist’s ability to offer independent scientific advice to the agency on
other projects”, (3) the Bulletin acknowledges that “the federal government
recognize that under certain circumstances
some conflict may be unavoidable…” and, (4) the Bulletin
completely prevents the avoidance of conflicts of interest by approving the use
of anonymous reviews, a practice which is wide-spread in federal government
agencies that support scientific research.
The Bulletin fails to provide any instruction, direction,
or requirement either to guard against fraudulent peer review or to prosecute
those suspected of making untruthful reviews. At the same time, the Bulletin
approves of the use of anonymous reviews that are free from accountability and
from civil recourse. The Bulletin gives tacit approval to circumstances that
allow conflicts of interest and prevents the avoidance of conflicts of interest.
With that combination, the Bulletin encourages and gives free rein to any
criminal or quasi-criminal element that seeks to attain unfair advantage through
deceit and misrepresentation in the anonymity-protected peer review system.
The Bulletin states that “reviewers may be compensated for
their work or they may donate their time as a contribution to science or public
service”. All instances with which I am familiar, specifically, NASA and the
National Science Foundation, do not pay reviewers for their time. Reviewers do
reviews for a variety of reasons, such as to curry favor with agency officials
or to exercise control over their competitors, but the bottom line is that time
is money and agencies often get for their non-compensation a hastily done,
superficial review.
The most serious short-coming of the Bulletin is its
failure to recognize or to admit the debilitating consequences of the long-term
application of the practices described above. The application of anonymity and
freedom from accountability in the peer review system and the openness to
conflicts of interest, gives unfair advantage to those who would unjustly berate
a competitor’s formal request for research funding. The perception – real or
imagined – that some individuals would do just that has had a chilling effect,
forcing scientists to become defensive, adopting only the consensus approved
viewpoint and refraining from discussing anything that might be considered as a
challenge to other’s work or to the funding agency’s programs. That is not
science!
If a foreign power or a terrorist group had set out to
slowly and imperceptibly undermine American science, I doubt that it could have
devised a methodology for the purpose any more effective than the practices set
forth in the Bulletin. Used for decades, these practices have diminished
American science to the present point of approaching third-world status.
Recommendations for Systemic Changes in the
Administration of Peer Review
Like the Critique, the following recommendations are
specifically limited to the administration of peer review as applied to reviews
of scientific grant/contract proposals that are not subject to secrecy
considerations related to national security and defense.
Generally, conflicts of interest, in the broadest
definition, should never be permitted and should never be tolerated in peer
review. Federal regulations permitting same, such as those under which NASA
operates, should be changed.
Anonymity should never be used in peer reviews. Reviewers
should sign and swear their reviews “under penalty of perjury” and should be
held accountable for the truthfulness of their reviews.
Reviewers should always be compensated, and compensated
well, for making reviews.
An independent “ombudsman agency” should be created to
address conflicts and disagreements between the individual or organization
submitting the research proposal and the agency to which the proposal was
submitted. The “ombudsman agency” should be empowered to bring potentially
unlawful activity to the attention of the Department of Justice for
investigation and possible prosecution.
Recipients of federal research grants/contracts should
incur the responsibility of making some pre-determined number of reviews under
the conditions described above. Like defense attorneys selecting a jury, each
grants/contract recipient should be accorded a number of “pre-emptive strikes”
so as to be able to remove himself/herself from certain specific proposal
reviews.
An individual or organization submitting a research
proposal should be presented with an official list of names that the funding
agency proposes to be peer reviewers. Like prosecuting attorneys selecting a
jury, the individual or organization submitting the proposal should be accorded
a number of “pre-emptive strikes” so as to be able to remove certain specific
proposed reviewers.
Implementation of the above recommendations will begin to
correct the longstanding debilitation of American science. The above
recommendations are intended to correct the short-comings displayed in the
Bulletin. Additional managerial improvements are possible, but are not specified
in the present document.
Download Bulletin:
http://cio.energy.gov/documents/OMB_Final_Info_Quality_Bulletin_for_peer_bulletin(2).
pdf
