Crowd sourcing has just been given a recent visibility boost with DARPA’s Red Balloon contest that was won by the MIT team. At the same time, Amazon’s well-established (since 2005!) platform for crowd sourcing, the Mechanical Turk, is gaining attention as a platform for conducting behavioral experiments, especially in behavioral economics and game-theory.
Named after the 18th century human-powered chess-playing “machine” called “the Turk”, this platform allows “requesters” to submit “Human Intelligence Tasks” (HITs) to a multitude of human “workers” who solve them for money. A sample of recent typical tasks include tagging pictures (for 3 cents), writing (and posting on the web) a short essay with a link (for 20 cents), or correcting spellings (for 2 cents, in Japanese). This allows brief and cheap employment of hundreds and thousands of people to work on simple Internet-doable low-level knowledge tasks. You may demand various “qualification exams” from these workers, and design such quals of your own. Obviously workers are in it for the money, but apparently not just that.
Recently, the Mechanical Turk is being used to conduct behavioral experiments. Gabriele Paolacci is methodologically repeating experiments of Kahneman and Tversky and reporting on them in his blog. Panos Ipeirotis reports on his blog studies of some aspects of the Mechanical Turk as well as results of various behavioral game-theory experiments on it. I’ve seen others report on such experiments too. Markus Jacobsson from PARC gives general tips for conducting such human experiments using the Mechanical Turk.
Turk-based behavioral experimentation has the immense appeal of being cheap, fast, and easy to administer. There are obviously pitfalls such as getting a good grasp on the population, but so does any experimental setup. Such a platform may be especially appropriate for Internet-related behavioral experiments such as figuring out bidding behavior in online auctions, or how to best frame a commercial situation on a web-page. Could this be a tool for the yet not-quite-existent experimental AGT?