I’ve enjoyed reading the lively discussion sparked by Ariel’s post endorsing a name change for EC. This seems like a good time to talk about some intriguing reforms that are actually taking place with this year’s conference. There are at least three noteworthy innovations in the EC’12 call for papers:
- The CFP designates three focus areas: theory and foundations, artificial intelligence, experimental and applications. The authors of each paper must designate it as belonging to at least one of the areas. Multiple focus areas are allowed. The SPC and PC have members in each focus area, and the labeling of papers by focus area will guide the assignment of reviewers.
- There is the possibility of holding parallel sessions, if the number of submissions of sufficiently high quality exceeds the number of papers that could be presented in a single-session conference.
- The submissions are in single-column format with a limit of 18 pages. (Hooray! No more squinting at 9-point font when reviewing EC submissions.)
Concerning the first two of these points, I thought it could be beneficial to have an open discussion on this blog about their rationale and their implications for the process of reviewing, selecting, and presenting papers. A few days ago I had an Internet chat with this year’s PC chairs, Panos Ipeirotis and Kevin Leyton-Brown, as well as Tim Roughgarden who was one of last year’s PC chairs. In the process, I also got the scoop on another innovation that hasn’t yet been publicly announced: authors of work accepted for publication during the past year in other venues (journals or conferences) may submit proposals for poster presentations of their work. The submission deadline for such posters will be later than the Feb. 6 deadline for full paper submissions; expect more details to be forthcoming.
The remainder of this post is a summary of my chat with Panos, Kevin, and Tim.
The parallel session option
EC has always been a single session conference. This clearly has some advantages, but of course it places a constraint on the number of papers that can appear in the conference. It has become increasingly difficult to obey this constraint while accepting all of the papers that were deemed by the program committee to be worthy of presentation, as the number of submissions has grown sharply in recent years. Kevin wrote, “We wanted to ensure that we accepted all the papers that we thought were worth featuring at EC. I’m not sure how many that will be. Maybe we’ll be able to do a single track after all, but we wanted to allow the quality of papers to drive the schedule, and to go parallel if that was what it takes. We imagine not being parallel in all sessions.” Tim added, “Last year, due to a number of constraints, we had to reject papers that had strong support from the PC. This was the hardest part, for me, of being co-PC chair last year. Last year, there was no option of parallel sessions. Since EC was part of FCRC, the hours available to present EC papers was unusually limited. I also insisted that every accepted paper should receive a talk of at least 20 minutes. (I think 25-30 minutes would be even better.) Taking the intersection of these constraints, the maximum number of papers we could accept was 48. (It wound up being 49, due to two similar accepted papers sharing a common talk slot.) Meanwhile, there were at least 55-60 submissions that had very strong support from the PC.”
Focus Areas
Q: Are there informal targets for the distribution of papers by focus area?
A: (Kevin) Absolutely no targets. Panos and I were very explicit that we’d be OK with having only a handful of papers get in from one of the areas. They’re a device for assigning reviewers. The main purpose, though, is communication and commitment. That is, we wanted to credibly signal to people outside the “core theory” area of EC that they ought to submit high-quality work to the conference, and that if they do, they’ll be reviewed by people who value the kind of work that they do. Making separate SPCs for the different areas was a way of communicating how we defined those communities, and of making a visible promise that these people would look after the papers. Allowing multiple tags per paper tries to reflect the fact that these “areas” aren’t partitions of the topics studied by the EC community, or indeed of the members of this community.
My sense is that in past years, reviewing of different areas has been done fairly and well. However, that’s not always how it’s been perceived by researchers from non-theory disciplines. In some cases, they think they’ve been rejected by theorists for not taking the right approach, when they’ve actually been rejected by members of their own community, for the right reasons. But this new process will make clear what’s going on. Rather than serving as a device for imposing quotas on the papers that are submitted, I hope it changes the perceptions about the conference enough before submission, that we end up with a wider range of strong submissions.
(Tim) I think such signalling is a good idea. Last year, Yan and I did all the reviewing assignments by hand, and put in a lot of work to ensure that submissions were reviewed by the PC members who were most likely to appreciate them. But the process is not transparent to the authors. With different tracks, it should be obvious to authors that their submission will be assigned to knowledgeable reviewers.
Q: The CFP says that a paper submitted to two focus areas will be handled by the most qualified reviewers in the union of the two areas. Does that mean it’s possible that all reviewers of a paper will come from a single focus area, although the paper was submitted to two areas?
A: (Kevin) Yes, that’s possible. It may be that we’ll do assignments by hand, but maybe not. What we promise is the union.
Q: Anything else you want to tell us about the focus areas?
A: (Kevin) We don’t intend to identify focus areas in the program. For example, we envision a session on ad auctions that might include papers that are empirical, theoretical, AI. There’s no reason even to indicate which papers these are. The goal is explicitly not to fragment EC into subconferences.
(Panos) Plus a paper may belong to multiple areas. The SPC is partitioned by focus area, but the PC is not. We have PC members that have been nominated by SPC members across multiple focus areas.
Poster session(s)
(Panos) We have a new idea for something separate from the reviewed paper track. The idea is to let people submit accepted work (e.g., accepted in journals, or maybe another conference) and present in poster sessions.
(Kevin) Not rigorously peer reviewed. With a later deadline, too.
(Panos) We will do lightweight checking that it is relevant, and that it is accepted for publication since the last EC. (Full acceptance in the case of journal articles, not conditional.)
Concluding Thoughts
I’d like to thank Panos, Kevin, and Tim for volunteering their time to discuss these issues. Blog readers, what do you think about these reforms? How do you feel about EC going to parallel sessions? Has there been a pro-theory bias in the reviewing of EC submissions in past years? Will you be submitting your work to the new poster session?
This is a great post Bobby, I love the model of the blogger as a reporter 🙂
IMHO the idea of having three focus areas, and separating the PC/SPC, is awesome for exactly the reason mentioned: more transparency in the reviewing process. In the past there certainly have been some complaints from the AI side of the fence about a theory bias (which, importantly, can also be for specific research topics rather than methodology), and the new system would presumably solve this problem if it existed, and correct the perception if the problem did not exist.
I have to say though that I am less excited about the idea of parallel sessions. So far EC has been outstanding in terms of the exposure that one gets for an accepted paper, and this is one of its big advantages over, say, AAAI/IJCAI, where one usually gets a smaller and more homogeneous audience in game theory or social choice sessions.
By the way, notice that author responses have also made a comeback. I guess we’re going to have author responses in even years and not have them in odd years 🙂
Thanks for the interesting discussion. It’s great to see that EC is not static and some of the innovations seem to be in the right direction. However, I would like to comment on one implication of adding parallel sessions.
One lesson that economic theory has taught us is the value of competition. Competition seems to drive up, in our field and related ones, the quality of the papers. I am sure that we all heard people saying things like “the results we have now are good enough for conference X, but we can substantially strengthen the paper by obtaining that result, and then we can send it to the more competitive conference Y”. I’m refraining from specifying X and Y to avoid the usual discussion whether X is indeed better than Y and what does “better” means. The issue is that Y is perceived by the speaker as better and that the competition encourages researchers to invest more and submit more polished papers simply by raising the bar.
Increasing the number of papers has other obvious advantages (parallel sessions has their obvious disadvantages), and maybe small increase will produce better results. But in general the situation described by Tim, that a number of good papers had to be rejected, sounds to me as a good one. It would have been worrying if this was not the case: rejection signals submitters that they should produce better papers. (Being too competitive is of course equally bad, but this is currently not a real concern for EC, in my opinion.)
Having a focus area seems like a great idea, and hopefully will encourage submissions from people working in related areas that don’t traditionally submit to EC.
The idea of being flexible with parallel session is also very good, mainly since it allows choosing the number of papers that will be presented based on the quality of submissions rather than artificial (but very real) constraints that Tim mentioned. This is one of the advantages that Journals usually have over conferences, and seems to be a nice solution.
It would be interesting to see how to poster session turns out. Allowing to submit published work removes barriers for Economists, but since it’s not in their culture, it’s not clear how many Economists would submit to the poster session. Econ grad students may be more motivated. Especially the ones who will go on the job market the following year.
What about double-blind reviews?
Did you get an amswer?
From a strategic point of view, having two parallel sessions may help to “steal” more potential listeners from AAMAS.
I really want to thank all of you for offering so many helpful infos. Keep up the good work.