Variability of Reviewers' Comments in the Peer Review Process for Orthopaedic Research.

Abstract

Retrospective analysis of peer review comments. To assess the likelihood that comments provided by peer reviewers of one orthopaedic journal would be similar to comments of reviewers from the same journal and a second journal. The consistency of the peer review process in orthopedic research has not been objectively examined. Nine separate clinical papers related to spinal deformity were submitted for publication in major peer-reviewed journals and initially rejected. The exact same manuscripts were then submitted to different journals. All papers were returned with comments from two to three reviewers from each journal. Reviews were divided into distinct conceptual criticisms that were regarded as separate comments. Comments were compared between reviewers of the same journal and to comments from reviewers of the second journal. When comparing comments from reviewers of the same journal, an average of 11% of comments were repeated (range 0% [0/12] to 23% [3/13]). On average, 20% of comments from the first journal were repeated by a reviewer at the second journal (range 10% [1/10] to 33% [6/18]). If a comment was made by two or more reviewers from the first journal, it had a higher likelihood (43% [6/14]) of being repeated by a reviewer from the second journal. When an identical manuscript is submitted to a second journal after being rejected, 80% of peer review comments from the first journal are not repeated by reviewers from the second journal. One may question if addressing every peer review comment in a rejected manuscript prior to resubmission is an efficient use of resources. Comments that appear twice or more in the first journal review are more likely to reappear and may warrant special attention from the researcher. Level IV.