Peering into the online future

Download as PDF

Ask someone off the street what they think of Shakespeare and you’re likely to get a mixed response. For many people the practical experience is one of stultifying English lessons, baffling encounters with Ye Olde Talke and something weird involving men in tights. Yet in the hands of experts, Shakespeare unfolds into a rich and vibrant creative experience, with a remarkable ability to resonate with the issues of the day.

Bearing that in mind it’s perhaps apt that it’s the Shakespeare Quarterly that is making waves with their experiment in open, online peer review — and that their experiment should be a success. When the prestigious science journal Nature trialled open peer review in 2006, the results were not encouraging. Of more than 1300 articles sent for review during the trial period, the authors of only 71 agreed to allow their submissions being displayed for public comment. Nearly half received no comments at all, while of the feedback that was received, over half were to only 8 of the papers. Editors felt that getting comments from sufficiently qualified researchers was “like pulling teeth” and found little use in the comments that were made; there was a feeling that the highly competitive nature of much of the research submitted was a significant factor in the unwillingness of scientists to participate. But whereas Nature’s competitive environment proved red in tooth and claw, with Shakespeare it seems that “the play’s the thing”, with participants noting an improvement in both the quality and extent of feedback they received.

Trials and experiences like those of Nature or the Shakespeare Quarterly have not come out of the blue. Ever since the arrival of the internet, individual groups of researchers and publishers have been using the online environment to experiment with new means of research communication.

Some — like the well-known “e-print” repository arXiv.org — used the Web to enhance and facilitate existing practices, in arXiv‘s case the longstanding tradition of preprint exchange in the high-energy physics community. In some fields, this preprint exchange forms the dominant part of cutting-edge research exchange, without editorial peer review as an entry condition. As arXiv founder Paul Ginsparg noted, researchers have “learned to determine from the title and abstract (and occasionally the authors) whether we wish to read a paper, and to verify necessary results rather than relying on the alleged verification of overworked or otherwise careless referees.” Arxiv’s minimal moderation procedures are surprisingly successful at filtering out the minimal quantity of junk papers that are submitted.

Other innovators have taken advantage of the Web’s ability to serve as a virtual meeting place. Long before VoIP and videoconferencing became mainstream technology, a small group of philosophers at the Institut Jean Nicod in Paris conceived of a website for online conferences. Interdisciplines offers invited “speakers” the opportunity to present articles centred around a particular topic of discussion. Past events have included themes as diverse as art and cognition, the function of mirror neurons, and the origins of suicide terrorism. But the articles are only the start — released over a period of time, each then becomes open to discussion, with the comment threads following them often providing as much interest as the articles themselves. Authors often revise their articles in response to (or even collaboration with) other participants, and when the proceedings of Interdisciplines conferences have been published in books or journal special issues, it has been common for the most interesting parts of the discussion to be published alongside them.

A similar dialogue-focused review process has been pioneered by a new Open Access journal for biology research. Biology Direct takes a radically different approach to traditional model of peer review being the gatekeeper for publication. Acceptance is not conditional on satisfying the reviewers — rather, it is conditional on the authors managing to convince three different members of the editorial board to provide or obtain reviews in the first place. So long as three different reviews are received it is up to the authors to decide the fate of their own paper. They may submit a revised version or withdraw their paper entirely, but they may also choose to ignore some or all of the reviewers’ advice. Published papers are accompanied by the complete author-reviewer correspondence, so that readers can decide for themselves the merits of the arguments, while also being able to identify clearly the contribution of the reviewer to the final article.

The editors of Biology Direct have spoken of their hope that this model could “help to revitalize the culture of scientific debate that is waning in the uneven duel between the omnipotent, anonymous reviewer and helpless author.” This is not the only asymmetry that the online environment can help overcome. As Andrew Dayton noted perceptively in his article on “open discourse”, the traditional scientific paper is by its nature a very one-sided affair. The authors’ claims are set firmly on the page, and the only means of querying them is to publish an article of one’s own, with all the hassle of review and no guarantee that it will read by the same people who read the original article. This is a big contrast with the highly conversation-focused environment of blogs, wikis and social networking sites (and increasingly, online newspapers and magazines), where readers’ comments and feedback are a respected and integral part of the way the system works.

Such openness to unsolicited feedback reflects another side of the online environment — its capacity to source the views and expertise of large numbers of people to create a more effective collective body of work. The most famous example is Wikipedia, but there are plenty of other examples of this “wisdom of the crowds” for the peer review process to take notice of. When CBS News announced it had obtained military memos criticizing George W. Bush’s service in the Texas Air National Guard, it was political bloggers whose discussions and queries soon pieced together the evidence that these documents were a fraud. More recently, a claimed proof of the P ≠ NP conjecture was scrutinized, found interesting, innovative and — sadly, ultimately — flawed, entirely within the scientific blogosphere. The collective feedback of individuals with personal interest or curiosity in a topic provides a powerful resource for scientific publications to engage with.

ICST’s own submissions management system, e-Scripts, opens a window to this kind of voluntary, curiosity-driven engagement. Titles and abstracts of submitted articles are posted for all registered members to see, and participants can bid to review papers that are of interest to them. Benefits through ICST’s Science Rewards Program are offered to such volunteer reviewers, but the primary aim is that by appealing to self-motivated reviewers who are interested in the topic of a submitted paper, both speed and quality of of review will be improved. It’s a key departure from the traditional behind-closed-doors process of referee selection which is so prone to abuse. The question we invite readers to consider, given the success of the Shakespeare Quarterly’s open review trial, is whether this goes far enough. Would authors, reviewers and readers be happy to see ICST’s publications adopt an even more radical approach to peer review? And what out of all the different options out there would the ICST community most like to see tried?

A little over 40 years ago, physicist John Ziman wrote a famous article on the importance of conversation and collective effort in creating a connected and reliable body of scientific knowledge. Nevertheless, he considered journal referees an integral part of the system: elsewhere, he wrote of how the review process provided articles with “the imprimatur of scientific authenticity”. In the online world, where the “wisdom of crowds” has the potential to far more effectively identify errors and misdemeanours than the limited scrutiny of two or three individuals, perhaps open and immediate scientific conversation will become the primary means of review — not merely the follow-up to what gets past the traditional journal referee.