WINE 2017, a follow up

Guest post from Nikhil Devanur:

Pinyan Lu and I were the PC chairs for WINE 2017, and we decided to conduct an experiment. We asked the PC members to score the submissions in the same way they would score an EC 2017 submission. In fact, we sent them the instructions that were given to the PC for EC 2017 and asked them to follow the same guidelines.

Right off the bat, there is the question of how well these instructions were followed. We sent these instructions multiple times, and yet some PC members seem to have ignored it initially. This would invariably come up during discussions. Someone would ask, “Is this EC scale, or WINE scale”, and every now and then there would be a “Doh! I forgot”.

With that aside, I want to present the results of this experiment. Moshe Babaioff kindly gave me the statistics for EC submissions, so we can compare. The key quantity of interest is of course, how do the submission qualities differ. We all know EC gets stronger submissions, but by how much? We sorted the papers into the following buckets, and compared the percentage of submissions in each bucket. The scoring scale for EC was from 1 to 7. Here’s the result.

Average score	EC	WINE
6+	6%	0%
5.5 to 6	9%	3%
5 to 5.5	17%	14%
4.5 to 5	14%	11%
4 to 4.5	11%	19%
3 to 4	27%	38%
1 to 3	16%	15%

This table doesn’t tell the whole story, so I did something else: I added 0.5 to the average of each WINE paper, and then calculated the CDF. Here’s what that looks like.

Average score	EC	WINE + 0.5
6+	6%	3%
5.5+	16%	17%
5+	32%	28%
4.5+	47%	48%
4+	58%	65%
3+	84%	89%
1+	100%	100%

Now you can see that the two columns are very close to each other. What this tells me is that there is about a 0.5 to 1 point difference between EC and WINE submissions, on a scale of 1 to 7. (My guess is that this would hold even after taking into consideration a bit of grade inflation in the experiment, which is hard to measure.)

Other than this experiment, here is some feedback that I wanted to give to the community.

1. Historically, WINE has been focused on the CS theory side of AGT, but there is no good reason to continue this. We made a conscious effort to change this and attract all AGT work, including what is typically done in AI and OR communities, as well as experimental work. We said this explicitly in the CFP, and I tried to have the PC represent this as well. I think we got a more diverse set of submissions, especially from the OR community. Several members from OR were attending WINE for the first time, and the feedback from them was that we should publicize WINE more to the OR community. I hope this trend continues, and we see virtually no difference between EC and WINE in terms of topics of relevance.
2. WINE has been using Springer for publishing the proceedings and we were not quite happy with them. One, we had ordered some number of hard copies of the proceedings, with the idea that they would be made available on a shared basis at the conference location. These were never delivered. Springer was also supposed to sponsor a monetary reward for the best paper, which they never sent. (We gave the reward out of other sponsorship money.) I would recommend seriously considering alternate publishers, including open access publishers.
3. This has been said before, but one of the most painful aspects of organizing WINE is the need to be budget balanced each year. There is no rolling bank account, and if for some reason the balance goes red in a year, that money has to come out of the PC chairs/local organizers!! This leads to a lot of anxiety and conservative actions in terms of how the sponsorship money is used.

There were other interesting proposals that came up in the WINE business meeting, but I chose not to include any of those so that this blog post is as short as possible. You can watch a recording of this (as well as that of all the talks) here:

http://lcm.csa.iisc.ernet.in/wine2017/

Finally, I want to give a shout out once again to the local organizers of WINE 2017. I got uniformly and overwhelmingly positive feedback from many of the attendees that this was one of the most enjoyable conferences they had attended. The local organizers (Y. Narahari and his group at the IISc, Bangalore) get all the credit for this.

Advertisement