Welcome to MTurk Crowd

Work on Amazon Mechanical Turk, learn from the best, and have fun doing it. Join the crowd today!

Sign Up

Press/Studies How do Crowdworker Communities and Microtask Markets Influence Each Other?

Discussion in 'AMT News' started by electrolyte, May 15, 2018.

  1. electrolyte

    electrolyte
    Expand Collapse
    The Ghost of MTurk Past

    • Contributor
    Joined:
    Jan 10, 2016
    Messages:
    19,184
    Likes:
    45,926
    How do Crowdworker Communities and Microtask Markets Influence Each Other? A Data-Driven Study on Amazon Mechanical Turk
    Jie Yang, Carlo van der Valk, Tobias Hoßfeld, Judith Redi, and Alessandro Bozzon

    "Crowdworker online communities – operating in fora like mTurkForum and TurkerNation – are an important actor in microwork markets. Albeit central to market dynamics, how the behavior of crowdworker communities and the dynamics of online marketplaces influence each other is yet to be understood. To provide quantitative evidence of such influence, we performed an analysis on 6-years worth of mTurk market activities and community discussions in six fora. We investigated the nature of the relationships that exist between activities in fora, tasks published in mTurk, requesters for such tasks, and task completion speed. We validate – and expand upon – results from previous work by showing that (i) there are differences between market demand and community activities that are specific to fora and task types; (ii) the temporal progression of HIT availability in the market is predictive of the upcoming amount of crowdworker discussions, with significant differences across fora and discussion categories; (iii) activities in fora can have a significant positive impact on the completion speed of tasks available in the market.

    In this work, we adopt a data-driven approach, and collect, enrich, and analyze a dataset containing 6-years worth of discussions produced by crowdworker communities in six online fora – mTurkCrowd, mTurkForum, TurkerNation, mTurkGrind, Reddit HWTF, and Turkopticon. We contribute a taxonomy to categorize discussions according to topic and function (e.g. Comment, Experience, Social), and trained a machine learning classifier to automatically categorize messages, thus enabling large-scale analysis of workers’ discussions in fora."

    http://doc.rero.ch/record/309215/files/yan_hdc.pdf
     
    Collapse Signature Expand Signature
  2. Achilles2357

    Achilles2357
    Expand Collapse
    Active Member

    Joined:
    May 24, 2017
    Messages:
    926
    Likes:
    1,398
    This looks really interesting. I think I am going to read this one very carefully sometime other than tonight. But their third main conclusion, basically that forum activity influences task completion rate, is probably fairly obvious to many people here.
     
    Collapse Signature Expand Signature
  3. Achilles2357

    Achilles2357
    Expand Collapse
    Active Member

    Joined:
    May 24, 2017
    Messages:
    926
    Likes:
    1,398
    I haven't even looked at this since the other day, but one thing I recall is that they claim that something like 59% of mturkers are active in forums. One issue is that it is not clear what this is supposed to mean. Presumably around 59% of workers doing their study said they were active in some mturk community. But what do they think this represents? (Apart from the fact that 59% of people doing their study presumably are somewhat active in some forum...) Is it 59% of the 500,000 or whatever workers that Amazon claims? Is it 59% of workers who do at least one hit a week? At least one hit a day? Who spend at least an hour a day turking? Who turk as a major source of income?

    Suppose that their standard is a worker who does at least one hit more days than not. I suspect their figure is highly inflated. They probably put out a reasonable hit that paid a somewhat decent wage, and workers active online (even if only passively) collected a good chunk, beating out the more clueless casual turkers.

    On the other hand, I suspect that if you ask who earns the average dollar on mturk, the percentage at least somewhat active in worker forums is going to be much higher than 59%.

    I recall reading about another group that was trying to gain data on mturk workers. They put out a "fair" hit that paid reasonably and they realized that mostly experienced workers did it. So they then put out a "junk" hit to get responses from other workers (but they paid a bonus to bring the wage up to the fair rate). I think this was testbirds.de (an interesting source of info).

    I think it is very difficult to understand mturk without trying to do it for real. And then you realize how strange it is.
     
    Collapse Signature Expand Signature
    IPsoft AWS Account likes this.