Yes, Penn Integrates Knowledge appears ungainly. The acronym, PIK, rolls off the tongue easily enough but brings to mind the image of a dental instrument.

To more serious matters. On proposal review, an important thing to keep in mind is the message space. One is not constrained to use a small message space like a summary score or single ranking. I believe that some of the concerns raised about the Merrifield and Saari proposal can be addressed through the choice of the message space.

rakesh

One possibility I have thought about would be to introduce a comment phase before voting, in which each referee can circulate anonymized comments to the group looking at a particular proposal. This would give the opportunity to make sure that others have the same important information that you do, such as “I happen to know that X did this in 1977 — here’s the reference” to avoid some of the issues raised above. This would make the process rather more like a conventional review panel, but without the airfares, and with the workload distributed more equitably.

On the reward for predicting strong citations, aside from the time-lag issue, my experience has been that some of the best proposals are those for which I cannot predict the likely citations: if it pays off, it will be thousands, if it turns out to be a dead end, very few. Referees will therefore tend to err on the side of caution with such applications and opt for “money in the bank” applications that are bound to return a healthy crop of citations, thus elminating the very proposals we should be funding because the answer is truly unknown in an interesting way.

A subclass of peer prediction mechanisms actually gives reviewers an explicit way to report that they believe to be in the minority. In these Bayesian Truth Serum mechanisms, reviewers make two reports: 1. the actual score of the paper (e.g. accept or reject) and 2. a prediction of the scores given by other reviewers about the same paper. In the binary setting with “accept” and “reject”, for example, this prediction report would be the answer to the question “What is your belief that another randomly-chosen reviewer of the same paper voted for acceptance?”

Peer prediction also handles the randomness problem Ariel mentioned by using scores instead of rankings, so if a reviewer receives a bad batch of papers it simply reports lower scores on all of them. I think it would be very interesting to extend current peer prediction mechanisms to the NSF setting where the set of reviewers and papers coincide.

I can’t remember any more details. However it seems this is a never-ending topic of discussion. Maybe we should have a “petit challenge” for the algorithmic economics community to work on this problem. It is not as frivolous as it sounds.

In case this is of interest: the (pitifully small) amounts of funding doled out in NZ by the sort-of-equivalent of NSF are done this way: reject 80% based on a CV and 1-page abstract, then reject 50% of the remaining applications after asking them to put in a full application. As far as I can tell, the decision at the first step is done by Borda scoring plus some discussion in secret. The amazing thing is, I can’t seem to convince those in charge to release the total score to the applicant. Knowing how far away you are from the cutoff should be useful information, not contravene any privacy rules, and not pose an undue reporting burden. This only adds to my suspicion that the secret discussion part is not as above board as we would like. Personally, I would be happy to see more randomization used – as far as I can tell, only a few applications are clear accept or reject at the first stage, and a lot of energy is used trying to make fine distinctions based on very little information.