One major characteristic of the social Web is that it is very evaluative. People are gathering online to engage with content in ways that have the effect of passing judgment. These judgments can be either explicit or implicit. Explicit evaluation might include voting, ranking, annotating and commenting. Implicit evaluation might include tagging, bookmarking, downloading and viewing.

Dario Taraborelli, Research Fellow in the Department of Psychology at University College London, was kind enough to alert me to his thought-provoking blog posting, Soft peer review? Social software and distributed scientific evaluation. It's a very interesting read, as are the comments. Essentially, Dario suggests that social software tools, in this case online reference management systems such as Connotea and CiteULike, are gathering collective evaluative metadata in several forms. Taken as a whole, these constitute soft evaluation systems. This is in contrast to the hard evaluation processes of traditional peer review.

According to Dario, the components of soft evaluation systems include semantic metadata (tags), popularity (bookmarks), hotness (user activities within a specific time frame), and collaborative annotation. He comments, "My feeling is that academic content providers (including publishers, scientific portals and bibliographic databases) will be urged to integrate metadata from social software services as soon as the potential of such services is fully acknowledged."

I'm intrigued by this idea. I'm also trying to sort it out. Metrics based on user behavior, especially behavior on Web sites, are always a tricky thing. I'm speaking as someone who has labored for years to draw sensible conclusions from the Web analytics data for my library's various sites. In a soft evaluation system, for example, the number of tags or bookmarks for a particular item might or might not be a reflection of its popularity. You can engage in these activities without actually reading the content. The "toread" tag is a good example of this - you never know if the reading actually gets done. And abuse of the system is unavoidable. But taken as a whole, across large, distributed populations, the aggregated data would be meaningful. It is the potential scale of such a system that can add value to current evaluative criteria.

A combination of hard and soft peer review is a very intriguing concept for the future of scholarly content evaluation.