After having looked at some of the debate raised by Moshe Vardi’s questioning of CS conferences (on Lance’s blog post as well as on my own post), I was struck by the fact that everyone was concerned with publishing papers; hardly anyone worried about reading these papers. This seems to be the general attitude in the theoretical CS community and there is a risk of it becoming even more so in the AGT community. People write papers just so that they get published, not really making much effort to have them be read; conferences and journals are being created to cater to the publishing needs of authors rather than to satisfy any desire of having more information to read.
CS journals lost their prestige and appeal simply since we stopped reading them. This is not a complaint about the reading habits in CS but rather about the publishing habits: what readers found in CS journals was usually not very helpful: lots of boring, mediocre, and badly written papers (and this in addition to the non-timeliness and expense of journals). The presumed added value of peer-reviewed verification of correctness was only meaningful to papers that were not previously seriously read by the community — usually those that were not interesting anyway. Since the authors of the most important papers do want people to read them, the best publications moved elsewhere — in the case of CS to conferences. For a long time we did read proceedings of CS conferences. The danger I see for many conferences now is the that people read and listen less and less to results presented there too. Many conferences have very few attendees that are not presenting their own papers — it did not use to be that way and this is a sign of rot.
I have two general recommendations for catering to possible readers of our scientific results:
Publish Less Papers
We publish too much (as recently noted by Mark). I am not against writing and sharing small steps towards solving a problem, but the pre-mature packaging as a “””publication””” is usually bad. Algorithms and theorems are often not really fully understood by the authors, and are thus not properly crystallized, simplified, or generalized. Models are not fully polished, justified, discussed or compared to other related models. Proofs are rarely made crisp. We also feel compelled to “market” our results often sacrificing scientific honesty, not to mention the amount of time wasted on the whole packaging effort. No wonder few people bother reading such papers. There are other ways to get preliminary results in the open (while maintaining credit) such as technical reports, the arXiv, or giving workshop talks. I really like the tradition in the field of economic theory of circulating a working paper that keeps being improved until it is deemed ready for real publication in a journal (where the top ones are actually widely read, I understand).
The funny thing is that the pressure for publishing more papers rather than better papers is not really external — it is a psychological trick we play on oursleves. In most places it is easier to get a job or tenure with ten first rate papers than with twenty second rate ones. It does not make any sense for hiring, tenure, grant, or promotion committees to count papers — if you insist on “counting”, then at least make sure you count impact: citations, the h-factor, or just publication in the absolute top venues. Conferences and journals should insist on evaluating papers from the point of view of the reader: simplicity, full context, clear writing, crisp proofs, polished models, all these are critcal to a paper being useful to its readers. They can help reducing the sheer number of papers by simply accepting less of them: top ones should be more selective, and less-than-top conferences should consider becoming publication-less workshops.
Help Identify the Important Ones
All these papers are “out there”: in conference proceedings, journals, technical reports, on authors’ websites, the arXiv, and other places. No one can read even a fraction of the papers in his field. Wouldn’t it be nice to have someone point to those that we should be advised to read? Someone to summarize a topic? This “meta-research” layer would be the critical element in helping the readers allocate their attention. There are various formats to draw attention to the key results: awards, invited talks, or tutorials in conferences. In the long term, a book that summarizes an area, in the middle term, a “hand-book” written by many authors, or in the shorter term, lecture notes. Wouldn’t it be nice if more people wrote surveys or expositions? I particularly liked the new physics review site (as reported by Suresh). Blogs can point papers out too (Oded Goldreich does and I try to do so too).
Elsewhere, Noah Snyder is arguing that beginning graduate students should choose their reading less on the importance of the material and more on its readability.
I think it does not help that the STOC/FOCS system encourages hastily written papers, and that papers that do not succeed in presenting their ideas clearly and simply are thought of as more deep and therefore are more likely to be accepted.
I had a related thought earlier today: for my own sake as a researcher, one of the most important things is, “Am I learning new techniques?”
I see people press themselves to think “Can I write a paper about something which hasn’t appeared in the literature before?” whereas I feel the community (and individuals) would be better served by understanding its own body of literature and publishing instead under the guideline, “This problem seems not to be solvable using any adaptation of previous techniques – how can I solve it, and how can I make the new techniques as lucid as possible to the reader?”
Unsurprisingly, I largely agree with this post and comments. Too many journal papers look like the author is showing that (s)he can do something technical, with not much motivation, and little attempt at elegance or generality. That sort of stuff can go in the arXiv or working paper/preprint, to ensure priority, and conference talks should promote it and help to refine it. But the “archival” version should be written properly. I have probably missed out on learning some good stuff because I just couldn’t stomach the poor style of presentation of many papers.
In my experience with journals in the combinatorial area, I have found a large bias toward technical difficulty and solving of hard concrete problems, whereas generality, elegance and making moderately hard things trivial or automatable gets less credit. Papers that summarize a field well get much less credit, as do books it seems. But these are much more valuable than yet another application of standard techniques, especially to fairly trivial problems. They are much, much harder to write. I am about to start work on a book in the analytic combinatorics area, and I don’t expect to get too much else done during that time.
Of course value judgments are involved, and I am sure that many things I consider nontrivial Noam would consider trivial (probably not many vice versa though). There are strong pressures exerted by deans, etc, which encourage hasty submission. I myself have been explicitly told I need to publish more to advance in my current institution. My best paper (in SIAM Review) is a survey and worth 5-10 normal papers, but around here it isn’t seen that way. The lower down we go on the university food chain, the more counting of papers seems to happen (perhaps because the evaluators are less confident of their own judgments on quality than they are at top places). I do feel the pressure but don’t want to send to a write-only venue. The more senior academics have a big part to play if the culture is to be changed.
I think referees need to be part of the solution. It is not unreasonable to reject a paper because it is poorly motivated and the notation is hard to follow, even if it is all correct and reasonably new and technically interesting. The revised version will no doubt be published eventually and be better for the experience.
[…] Readerless publications « Algorithmic Game Theory […]
Note that the last two commentors are at odds. The former says that one should find a problem that current techniques do not solve and the latter states that a problem should be well motivated. I believe that there could be a gaping hole in current techniques, but still not a good motivation to study the problem except that it is not covered by current techniques. So rejecting a paper purely on having no (practical) motivation does not seem right to me.
Hi asterix
You misunderstand what I mean by “motivation”. No “practical” application is required. A paper that applies a new technique to a toy problem can be very valuable. I simply mean that one must give some argument as to why one has chosen to publish on this topic rather than another. If the underlying problem is not important, and the techniques are not likely to (eventually) help solve something important, perhaps the paper should not be published in a journal (especially if it is poorly presented). In other words, papers that give no insight are not much use.
Another brief comment: the shift in power from readers to authors will probably increase with the open access, pay to publish model so common now in fields like biology. The consequences of this for mathematics and related fields have been discussed by John Ewing in the AMS Notices a while ago:
Click to access tx080300381p.pdf
It should not be a surprise to anyone to hear this, but the current grant system (at least in the US) definitely seems to encourage publishing quantity rather than quality. The grant is evaluated (whether at the middle or at the end) by number of publications; it is very hard for the grantees to judge quality.
Note that math, which many inside CS seem to want to emulate, has very low funding and so maybe this pressure is not an issue. Do we really want to trade places?
In its new ERC grants (http://erc.europa.eu/index.cfm?fuseaction=page.display&topicID=23 ), the European union was smart enough to emphasize quality over quantity by using the simple trick of allowing proposals to list only their top 10 papers in the last 10 years. One would hope that other grant bodies find their own way to reward quality rather than quantity — it is not that difficult.
This comment relates to an older post about conferences and jouranls. Econmists often present their work many times before they submit their paper.
This process provides many comments, which eventually lead to a better paper. This is not about the writing itslef, but it helps the author also organzing his thoughts and polishing the story of the paper.
Usually in CS papers are being presented after accepted. At least the majority of the times and then comments of the audience are only helful for further discussion.
The CS theorist often hears about a problem and runs to solve as long as it is “hard” enough. It is not necessaarily a bad thing, but leads to high competition and competition leads to publishing as fast as possible. So in this case it is not the quantity that leads the reseacher.
[…] I am in favor of having large conferences where everyone meets and everyone talks as is common in other disciplines, I would like to see these as an additional format to the top CS […]
“My best paper (in SIAM Review) is a survey and worth 5-10 normal papers, but around here it isn’t seen that way.”
I don’t know whether other publications are 5-10 times worse, but often much worse, in any case. Just like the “THE Book” like proofs are often much more important than all this “complicated mist”.
Leaving things “complicated” means “we haven’t understood them”. So, why rush to publishing this? Why not to let this stay for a while? So, why we strive for “Readerless publications?” READERLESS — a great and very tight title for what is happening now.
Something is wrong in all the mechanics of the TCS …
[…] From my point of view, this reduction in the role of referees makes the journal superflous: information that is unfiltered for quality, becomes useless since it is impossible to find the good stuff there — you might as well just put your paper on the arXiv (see also my post on the attention economy). I personally also don’t quite trust the referees for correctness, not in other journals and certainly not here — correctness is established after the “community” has chewed on the paper for a while (which will only happen if it is interesting). I find it hard to believe that any favorable reputation will be attached in the long run to publication in such places — they’ll mostly be a sink for readerless publications. […]
[…] in this uncharted territory this culture seems to have served the field well, despite its many shortcomings. At this point AGT seems to have become rather faddish (in the appropriate communities, that is), […]
[…] incentive to “publish” whether or not our research is worthwhile and whether or not anyone will ever read it. The “real” reason for publication is dissemination: let other researchers learn […]
I see people press themselves to think “Can I write a paper about something which hasn’t appeared in the literature before?” whereas I feel the community (and individuals) would be better served by understanding its own body of literature and publishing instead under the guideline, “This problem seems not to be solvable using any adaptation of previous techniques – how can I solve it, and how can I make the new techniques as lucid as possible to the reader?”
thanks
killing games