In a recent blog post, Lance raised the question of whether a conference should be judged (?) by its best or its worst papers. To me, this is part of a broader issue that views the conference (and journal) system as mostly playing a role in allocation of attention. As there are “too many” scientific papers, the question of which ones should get the attention of “the community” is a critical one. This view of attention as a resource is a general phenomena in our information-overload world, as explained by the nice introduction to the wikipedia article on Attention Economy:
Herbert Simon was perhaps the first person to articulate the concept of attention economics when he wrote:
- “…in an information-rich world, the wealth of information means a dearth of something else: a scarcity of whatever it is that information consumes. What information consumes is rather obvious: it consumes the attention of its recipients. Hence a wealth of information creates a poverty of attention and a need to allocate that attention efficiently among the overabundance of information sources that might consume it”(Simon 1971, p. 40-41).
He noted that many designers of information systems incorrectly represented their design problem as information scarcity rather than attention scarcity, and as a result they built systems that excelled at providing more and more information to people, when what was really needed were systems that excelled at filtering out unimportant or irrelevant information (Simon 1996, p. 143-144).
In recent years, Simon’s characterization of the problem of information overload as an economic one has become more popular. Business strategists have adopted the term “attention economy” (Davenport & Beck 2001), and some writers have even speculated that “attention transactions” will replace financial transactions as the focus of our economic system (Goldhaber 1997, Franck 1999). Information systems researchers have also adopted the idea, and are beginning to investigate mechanism designs which build on the idea of creating property rights in attention (see Applications).
My natural tendency is indeed to view the issue of allocation of attention (between scientific papers, as well as elsewhere) indeed as a mechanism design problem. Looking from afar, the journal and conference is a pretty impressive mechanism — better than what humanity has for most resource allocation purposes. Looking more closely, this mechanism was designed under a completely different set of technological constraints from those we have now. In particular the original main motivation of journals and conferences was dissemination of information, which today is better handled by the web, e.g. the ECCC or arXiv. Personally, I can cleary see the need for conferences as providing direct human interaction, but how journals still survive escapes me.
I believe that the near future will bring significant changes in how science uses conferences and journals. Blogs are already making a real impact: notice for example the attention (here, here, here, here, here, and here) that Mark Braverman (a young postdoc) got for his last result, before any journal or conference has assessed it.
Today, it is hard to separate the social issue of how science should organize itself, from the direct effect that the current mechanism has on our own job offers, tenure decisions, and promotions. It should be no surprise that the same mechanism that is used to allocate attention among scientists is also used to allocate other resources (our salaries) among them: the criteria are probably quite similar.
Hi Noam,
Welcome to the blogosphere! I look forward to reading your blog.
You say:
Personally, I can clearly see the need for conferences as providing direct human interaction, but how journals still survive escapes me.
I see journals as fulfilling the need for checking correctness and completeness of proofs. Do you anticipate a “Web 2.0” environment where papers in the arXiv, ECCC etc. get vetted and commented on rigorously by general readers?
I agree that journals still perform some useful functions such as verifying correctness, or allowing possibly-fruitful interaction with a good editor. I just don’t think that they are reasonable mechanisms for these functions now. I’m not sure whether a web-2.0-ish mechanism is the perfect answer, but I can see various web-2.0-ish mechanisms that may be better than the current system. (E.g. maybe only “interesting” results really need to be verified completely, and these will automatically be? Wouldn’t you consider Braverman’s new result more verified at this point than most journal papers?)
I agree journals will have to evolve, but don’t see the current system as a good substitute – yet. Even if you only take the papers from last year’s STOC, FOCS, and SODA (which are a small subset of the number of theory papers that get used by very good papers later) — only a very small fraction would have been commented on rigorously in the blogs. I think a good compromise for now maybe journals having very fast turnaround — less than 3 months — like Science and Nature do, and which Theory of Computing accomplishes most of the time. (I am an editor for ToC, but my respect for ToC’s turnaround is independent of this fact.)
I … don’t see the current system as a good substitute – yet.
I’d put the emphasis on yet.
and as a result they built systems that excelled at providing more and more information to people, when what was really needed were systems that excelled at filtering out unimportant or irrelevant information
A great quote from Simon. Even today, over a decade later, the IR community still uses precision and recall as their main tool for measuring the quality of an IR system. Implicit in their definition is more information is better, for example, the same answer repeated four times is better than a single answer.
In contrast a system such as Google the goal is to return the pertinent web link as number one and if not, at most within the first three results provided.
[…] This “meta-research” layer would be the critical element in helping the readers allocate their attention. There are various formats to draw attention to the key results: awards, invited talks, or […]
[…] I have to admit that I was intrigued by this unusual mode of marketing: the sole desired reward is attention — a link to the website. Of course, the spam filter on this blog routinely removes many […]
[…] It may be artificially triggered by the limited size of the conference, but it really reflects a competition for the limited attention that humans […]
[…] find the good stuff there — you might as well just put your paper on the arXiv (see also my post on the attention economy). I personally also don’t quite trust the referees for correctness, not in other journals […]
[…] that few non-Turing-Award-winners will get their work noticed this way, so some mechanism for allocation of attention is still needed. Possibly related posts: (automatically generated)Evalautaion of CS papers for […]
[…] whole “scientific/academic establishment” should aim to promote. Inherent here is the competition for the attention of other researchers who have to decide to spend time and effort reading your paper rather than […]
[…] future more flexible system evolve where dissemination is solved by arXiv-like web repositories and selection handled by a flexible and non-exclusive system that ranges from the informal recommendations of […]
[…] for paying some attention to my blog. As when I started writing in the blog, I still think that attention is the currency of our times, I hope that as a group we will deserve even more of […]
[…] listening. If you have been listening for the last two years, I’m grateful for your attention — I sing like a […]