Just a small number of impartial, complacent, or incompetent referees can significantly undermine the ability of the scholarly peer-review system to select the best scientific papers.
SFI External Professor Stefan Thurner and collaborator Rudolph Hanel, both of the Medical University of Vienna, created a model of a generic specialist field where referees fall into one of five categories ranging from always correct to impartial.
After running the model with 1000 scientists over 500 time-steps, Thurner and Hanel found that even a small number of bad referees can significantly reduce the quality of published papers. When just 10 percent of referees do not behave "correctly" the quality of accepted papers drops by one standard deviation. If the fractions of rational, random and correct referees are about 1/3 each, the quality selection aspect of peer review practically vanished altogether.
They say that the presence of some bad referees in the peer-review system will cause it to perform not much better than throwing a coin.
Thurner argues that science would benefit from the creation of a market for scientific work. He envisages a situation where journal editors search preprint servers for the most innovative papers before approaching authors with an offer of publication. The best papers, he believes, would naturally be picked up by a number of editors.
"Papers that no one wants to publish remain on the server and are open to everyone – but without the 'prestigious' quality stamp of a journal," Thurner explains.