Reputation for Human Computation

Like many, I am fascinated by the notion of “Human Computation“: algorithms that use black-boxes that are implemented by actual humans.  The fascination is probably mostly due to the “inverse”  interface between humans and machines: what we are used to is humans using black-boxes that are implemented by computers; now the roles are switched.   Add to this the unexpected effectiveness of this practice and one can certainly start musing about practical possibilities, various philosophical-leaning directions, as well as actual research questions.

Amazon’s Mechanical Turk brings this possibility into the hands of many, and in particular I have been hearing more and more researchers who are attempting to use it for running experiments on human subjects.  Many questions regrading the validity, ethics, and administration of such experiments suggest themselves, and these questions are starting to be addressed.

I have been following with interest Panos Ipeirotis’s stream of blog posts about the Mechanical Turk , the last of which suggests that the lack of a sufficient reputation mechanism is leading to the failure of the Mechanical Turk labor market.  In other words, “spammers” abound and are hurting the proper functioning of the market.  Building a sufficiently good reputation mechanism would seem to require the proper blend of dealing with the humans and the computers, a blend that may be especially interesting due to the “inverse” roles of humans and machines in this setting.

Advertisement