Following up on a previous post about Amazon.com’s investment in “artificial, artificial intelligence” — which I skeptically titled “Mechanical Turk Is A Mechanical Dud” — Jason Pontin of the NYT recently published a terrific piece on MTurk and the general lay of the land in the “crowdsourcing” arena:
Artificial Intelligence, With Help From the Humans
Excerpt:
Harnessing the collective wisdom of crowds isn’t new. It is employed by many of the “Web 2.0” social networks like Digg and Del.icio.us, which rely on human readers to select the most worthwhile items on the Web to read. But creating marketplaces of mercenary intelligences is genuinely novel.
By way of an update on my own experiment with the Turk:
After the transfer of funds to my Amazon MTurk “Requester” account was FINALLY completed (a transaction that took the e-commerce giant over a week to complete), setting up a series of multiple choice survey questions proved to be quick. I offered $.09 for the completion of each of four questions, and $.10 for the completion of a fifth. Shooting for 1,000 responses to each I discovered that the initial pace of completions proved quite misleading. The longer the questions, or HITs (Human Intelligence Tasks), were up, the slower the responses arrived.
After a week or so, however, I did have 1,000 responses to each of the four $.09 questions. Strangely, the fifth (and most remunerative) question, STILL has not reached 1,000 responses 2 weeks later.
The other salient observations I can offer are these:
1) Despite MTurk’s ability to allow individual respondent accounts access to each HIT only once (thus presumably preventing duplicate entries), responses submitted for one of my questions make it clear that there are a fair number of individuals gaming the system by using multiple accounts.
2) The geographic breakdown of respondents to my survey ( self-reported) was as follows:
60% – US
23% – India
2.5% – UK
2.9% – Canada
11.6% – divided among 19 other countries (the Philippines highest among them at 1.4%)
I’m not, by the way, discouraged from using the Mechanical Turk again, just a bit wiser about its best application. If you can use Amazon.com’s Turk API to design a HIT that calls your own survey form, it does offer — compared to most of the available methods — an amazingly cheap way to gather impressions from a large, if unscientific, sample population.