Experiments on Mechanical Turk

December 7, 2009

Crowd sourcing has just been given a recent visibility boost with DARPA’s Red Balloon contest that was won by the MIT team.  At the same time, Amazon’s well-established (since 2005!) platform for crowd sourcing, the Mechanical Turk, is gaining attention as a platform for conducting behavioral experiments, especially in behavioral economics and game-theory.

Named after the 18th century human-powered chess-playing “machine” called “the Turk”, this platform allows “requesters” to submit “Human Intelligence Tasks” (HITs) to a multitude of human “workers” who solve them for money.  A sample of recent typical tasks include tagging pictures (for 3 cents), writing (and posting on the web) a short essay with a link (for 20 cents), or correcting spellings (for 2 cents, in Japanese).  This allows brief and cheap employment of hundreds and thousands of people to work on simple Internet-doable low-level knowledge tasks.  You may demand various “qualification exams” from these workers, and design such quals of your own.  Obviously workers are in it for the money, but apparently not just that.

Recently, the Mechanical Turk is being used to conduct behavioral experiments.  Gabriele Paolacci is methodologically repeating experiments of Kahneman and Tversky and reporting on them in his blog. Panos Ipeirotis reports on his blog studies of some aspects of the Mechanical Turk as well as results of various behavioral game-theory experiments on it.  I’ve seen others report on such experiments too.  Markus Jacobsson from PARC gives general tips for conducting such human experiments using the Mechanical Turk.

Turk-based behavioral experimentation has the immense appeal of being cheap, fast, and easy to administer.  There are obviously pitfalls such as getting a good grasp on the population, but so does any experimental setup.   Such a platform may be especially appropriate for Internet-related behavioral experiments such as figuring out bidding behavior in online auctions, or how to best frame a commercial situation on a web-page.  Could this be a tool for the yet not-quite-existent experimental AGT?


Google Ad Exchange

September 18, 2009

Google just announced its ad exchange:

The DoubleClick Ad Exchange is a real-time marketplace to buy and sell display advertising space. By establishing an open marketplace where prices are set in a real-time auction, the Ad Exchange enables display ads and ad space to be allocated much more efficiently and easily across the web. It’s just like a stock exchange, which enables stocks to be traded in an open way.

Google's Ad-exchange

There are some existing competitors, but Google’s entry may be game-changing.  Lots of research problems beg themselves.


New Paper: A computational view of Market Efficiency

September 1, 2009

An intriguing new paper, A computational view of Market Efficiency by Jasmina Hasanhodzic, Andrew W. Lo, and Emanuele Viola has just been uploaded to the arXiv.

Abstract

We propose to study market efficiency from a computational viewpoint. Borrowing from theoretical computer science, we define a market to be efficient with respect to resources S (e.g., time, memory) if no strategy using resources S can make a profit. As a first step, we consider memory-m strategies whose action at time t depends only on the m previous observations at times t-m,...,t-1. We introduce and study a simple model of market evolution, where strategies impact the market by their decision to buy or sell. We show that the effect of optimal strategies using memory m can lead to “market conditions” that were not present initially, such as (1) market bubbles and (2) the possibility for a strategy using memory m' > m to make a bigger profit than was initially possible. We suggest ours as a framework to rationalize the technological arms race of quantitative trading firms.


Preference elicitation or manipulation?

July 1, 2009

A “textbook system” based on social choice theory would have a centralized mechanism interacting with multiple software agents, each of them representing  a user.  The centralized mechanism would be designed to optimize some global goal (such as revenue or social welfare) and each software agent would elicit the preferences of its user and then optimize according to user preferences.

Among other irritating findings, behavioral economics also casts doubts on this pretty picture, questioning the very notion that users have preferences; that is preferences that are independent of the elicitation method.  In the world of computation, we have a common example of this “framing” difficulty: the default.  Users rarely change it, but we can’t say that they actually prefer the default to the other alternative since if we change the default then they stick with the new one.  Judicious choice of defaults can obviously be used for the purposes of the centralized mechanism (default browser = Internet explorer); but what should we do if we really just want to make the user happy?  What does this even mean?

The following gripping talk by Dan Ariely demonstrates such issues.


The Advertising Industry

June 11, 2009

The current “killer application” for algorithmic game theory is certainly on-line advertising. The traditional advertising industry is  undergoing great turmoil due to this on-line advertising as well as other technological changes which may not only give viewers more relevant and less annoying ads, but also may start addressing the main problem from the point of view of advertisers, captured by the famous quote  “Half the money I spend on advertising is wasted; the trouble is I don’t know which half”.

This highly entertaining video is from the point of view of traditional advertising (thanks to Eyal Manor for the link).


eBay and antiquities looting

April 22, 2009

From Marginal Revolution, a story on how the eBay has reduced the looting of antiquities…

Before reading question: Can you guess how this works?

Easier question: why should you expect eBay to actually increase the looting of antiquities?


Interest-based advertising in Google

March 22, 2009

Internet advertising is perhaps the number one current application (revenue-wise) for algorithmic game theory, and is paying the salaries of many AGT researchers in Microsoft, Yahoo, Google, and elsewhere.    Beyond the basic “generalized second price auction” used for ad-auctions, there are many other issues and questions involved.  

About a week ago, Google annouced a new beta program for interest-based targeting (a form of behavioral targeting).  The basic idea is very simple: allow advertisers to target ads to segments of people according to their previous Internet behavior and not just the current web-page that they are on.   E.g. if you visit many sports web-sites, the it makes sense to show you sports-related ads even on a general news page rather than, say, detergent ads.  In general, this should result in less annoying (or even actually useful) ads for the viewer, higher effectiveness for the advertiser, and higher revenue for the web-site  ”publisher” (and of course, for my own current employer, Google).

Technically, this is quite easy to do using browser cookies, and is already heavily used by most advertising middlemen on the web.   The catch is of course in the privacy issues.  It seems that the main innovation here was in how Google managed to handle the privacy issues, as explained on it’s public policy blog.  The main technical ingredient here is the ads preferences manager that gives viewers full control of their data.