As picked up by Lance, Moshe Vardi uses his editor’s letter in the May CACM to start up a debate on the future of conferences in CS. Moshe asks:
Is the conference-publication “system” serving us well today? Before we try to fix the conference publication system, we must determine whether it is worth fixing.
While I agree with the view that the conference system in CS is cracking and needs an overhaul or replacement, I feel somewhat uncomfortable with the undertones of the question. Certainly no one is opposed to conferences per-se. It seems that the real issue is “how much should we value conference publications relative to journal publications”, with an implication that this will be used in hiring or promotion decisions. Indeed Moshe points specifically to the following decision:
in 1999, the Computing Research Association published a Best Practices Memo, titled “Evaluating Computer Scientists and Engineers for Promotion and Tenure,” that legitimized conference publication as the primary means of publication in computer research.
In my view, the issue of evaluation of candidates is not the correct starting point for thinking about our conferences. Top conferences in CS started gaining higher prestige than journals simply because the average scientific quality there was better (i.e. the CRA decision was the result, not the cause, of the higher prestige of conferences). I have no doubt that we see questions about conference publications arising now simply since publication in many conferences is no longer a signal of high quality. This is being recognized by the community and will certainly be taken into account by future promotion and hiring committees. Instead our question should be how do we best use conferences (and journals, and the Internet, and anything else) to advance our field. Conferences should provide the best service that they can, and so should journals. The community will certainly come to appreciate the venues that offer the highest signal of quality, be it a conference or a journal.
I seems to me that many of the changes in CS conferences are not really driven by an attempt to advance computer science, but more by attempts t0 advance computer scientists. Oded Goldreich puts it bluntly:
My impression is that FOCS and STOC do not function any more as forums devoted to the presentation and exchange of ideas (but rather function as “weight-lifting competitions”)
I’m not sure that I agree with Oded’s point of view regarding FOCS or STOC, but the situation is certainly worse in other conferences.
Personally, I do have some preliminary ideas about what kind of changes I would like to see in the conference system: fewer conferences with proceedings, smaller PCs (unlike IJCAI’09), less accepted papers, and in general more “less”. I also don’t see any reason to limit in any way the overlap between a journal publication and a confernce publication. If needed, the EC system of allowing a confernce submission not to be publihsed in the proceedings as to allow future publication in economics journals seems good. The point of this post though is not to put forward these suggestions but rather to steer the debate away from focusing on evaluation of candidates.
Turing's Invisible Hand 
If the problem is that conferences are acting too much like beauty contests and too little as places to exchange ideas, then fewer conferences and fewer acceptances seem to me like the opposite of a solution. Too-low acceptance rates lead to conferences in which the fads and cliques crowd out the less-glamorous but more fundamental research.
I tend to think that, on the average, with good program committees, more-glamorous should correlate well with more fundamental.
> changes I would like to see in the conference system: > fewer conferences with proceedings […]
With all due respect, this seems ironic given that you had just announced two new conferences in algorithmic game theory. In fact, my immediate reaction when I saw your previous post was, with EC, WINE and SAGT, isn’t it a bit too much?
I have to agree — I was just announcing the conferences — I have no involvement with either.
Re: “more-glamorous should correlate well with more fundamental”: I strongly disagree.
My feeling about FOCS/STOC and its tendency to encourage faddism is that it’s often a lot like a little kids’ soccer game: all of the players are clustered tightly around the ball, kicking haphazardly at it, and the knot of players and ball ends up taking a random walk that only occasionally gets near the goal. We could make better progress if we were more spread out.
I’m certain that most PCs try to aim at “fundamental” and not at “glamorous” — who would profess otherwise? You may disagree with the emerging consensus (as I sometimes do), but I can’t see the PCs having different goals.
The faddism accusations have been following STOC/FOCS for as long as I remember. Frankly I don’t understand: first, even the “fads” are always just a small fraction of conference papers. I can’t think of any fad (since “cryptography” in the 1980s) that ever monopolized more than 10%-15% of STOC/FOCS papers in even a single year. Second, I think that if many independent researchers become interested in a certain new area at some time — then this is good for science — not bad.
I also don’t understand the opinion of Noam that “less”
is better and that problems can be fixed by “good”
program committees. Don’t we currently have that?
If we want fewer papers (say in STOC/FOCS) then we may
as well call them workshops given the diversity and
size of the theory community these days.
In my view, the selectivity of the conferences is
forcing many interesting ideas and people away and
artificially elevating certain types of work over
other type of work. In that sense they are failing in
enabling exchange of ideas.
The question is why should I go to FOCS/STOC? Certainly not to hear the few papers in my narrow area that got in. To hear results in my sub-area I would go to workshops or specialty conferences. The reason for going to FOCS/STOC is to get the wider picture — the most important and interesting new results in other fields of theory. Frankly, my threshold of interest for results in these other areas (be it quantum stuff, computational geometry, cryptography, or data structures)is high. The main service that FOCS/STOC can give me is point out a few such results that I’d be advised to look at. These talks should get more than 20-25 minutes, so I can hear a reasonable background as well as a new idea.
It appears then that your idea of FOCS/STOC is
to act as a filtering mechanism so that people in
a particular area can learn about big results/ideas in another theory area. That is fine but
it is not clear to me that we need to have two/three
such conference a year and enormous investment in
time by authors and program committees just to
identify the very big results. Those are easy to
spot and we can have plenary talks and tutorials
once in a while to summarize these.
It seems like CS is the only academic discipline which insists on having conferences as its main publication venue.
Why don’t we move to publishing in online journals?
The argument often tends to be something along the lines of “journal reviews take a long time”. This is because all our reviewers are busy reviewing conference submissions. Can’t we enforce the same deadlines on journal reviewing as we do for conference reviews?
As TCS is rapidly growing, it seems impossible to hold a conference which captures “the important work” in all of the different areas. As is, the works across the different areas are getting too specialized for people in different disciplines to care.
Why can’t each area have some main online journal?
In physics, for example, conferences are genuinely used to exchange ideas, and get people to read (and cite) your latest journal submission. Why can’t we have something similar (and meet in more attractive places than Philadelphia, Austin, Atlanta, DC, etc)?
As an assistant professor, it often seems to me like the community is being taken hostage by the nostalgia of its founders, who seem to be the only ones enjoying and advocating for the conference system.
With so much inbreeding in our top conferences, and the almost unanimous agreement that the entire publication process is broken, we’re heading towards having our accomplishments dismissed by other academic disciplines.
The question: “Why don’t we move to publishing in online journals?” should be aimed straight at *journals* not conferences. People can certainly publish in journals right now — they choose to publish in conferences. It does not make sense to change conferences so they won’t compete with journals — change journals so that will compete with conferences!
TCS says: Why can’t we … meet in more attractive places…
We are still looking for volunteers to host FOCS 2010 in October/November 2010. You have it in your power to change the level of attractiveness of our conference locations.
Noam mentions IJCAI’09 in a negative context (due to its gargantuan PC). However, it is best to think of the IJCAI area chairs as the PC, whereas the “PC” are the reviewers. The difference is that the reviewers (=PC) in IJCAI get to bid for their papers and are given a presumably optimized assignment, rather than being assigned papers, often without much thought, by an overloaded PC. There are many reviewers since there are many submissions (around 1290 in IJCAI’09, to be exact). Despite the huge volume of papers, I can say the AAAI/IJCAI reviewing process is remarkable in its smoothness, the depth of the discussions among the reviewers, and the quality of the feedback; in my opinion it is much better, in fact, than conferences with small PCs and far fewer submissions (for example, if I recall correctly Noam remarked in a previous post that the reviews from STOC/FOCS, with their small PC, are usually useless).
[…] looked at some of the debate raised by Moshe Vardi’s questioning of CS conferences (on Lance’s blog post as well as on my own post), I was struck by the fact that everyone was concerned with publishing […]
thanks this post. I made some adjustments
With all due respect, this seems ironic given that you had just announced two new conferences in algorithmic game theory. In fact, my immediate reaction when I saw your previous post was, with EC, WINE and SAGT, isn’t it a bit too much?