Welcome to MTurk Crowd

Work on Amazon Mechanical Turk, learn from the best, and have fun doing it. Join the crowd today!

Sign Up

'Lifetime Approval Rate' Misunderstood by Requesters

Discussion in 'Requester Support' started by clickhappier, Jan 15, 2016.

  1. clickhappier

    clickhappier
    Expand Collapse
    ┬──┬ ノ( ゜-゜ノ)

    • Subforum Curator
    • Crowd Pleaser
    Joined:
    Jan 12, 2016
    Messages:
    727
    Likes:
    1,601
    There have been multiple incidents where inexperienced requesters misunderstood the 'Lifetime Approval Rate' stats displayed to them in the Requester GUI.

    On Jan 23, 2015, a requester rejected multiple workers with a feedback note saying "One of the requirements was a lifetime approval rate of +90%. You do not meet these requirements. Sorry." In his apology email, he said "I'm very sorry this happened. MTURK showed my an approval rate of 0%. It was not my intention to reject proper responses." This requester was able to reverse the rejections after workers informed him of his mistake.

    On Feb 19, 2015, another requester rejected multiple workers with a feedback note saying just "lifetime approval rate". Apparently this requester will be unable to reverse these rejections without assistance from MTurk support, because she had promptly irrevocably disposed of the batch (which ought to not even be allowed by MTurk's programming, btw).

    Both requesters did have qualification requirements set on their HITs for their desired approval rates (90% and 95%). Here's some comments I wrote about these incidents:

    "MTURK showed my an approval rate of 0%."

    Heh. I'm guessing he was looking at http://docs.aws.amazon.com/AWSMechTurk/latest/RequesterUI/ViewingWorkerDetails.html , where it says the amount/percent of HITs a worker has had approved *by that one requester*. This was most people's first HIT for this requester, and of course it hadn't been approved yet when he looked at that, so I think it would've said "0% (0/0)" or something like that for nearly every worker... ​

    and

    We saw an incident like this with another inexperienced requester a couple weeks ago. ... as I speculated then: I think they're not comprehending that their use of the "HIT approval rate (%) is not less than x" requirement on the HIT made it impossible for anyone below their specified approval % to accept the HIT (with the exception of workers with less than 100 HITs), and then they go look at the only stats that are visible from the requester side for workers' completed HITs, which is the 'lifetime' percent you've had approved for *that* requester... which was 0%, of course. Then they bizarrely decide this means you and all the other workers need to be rejected because you have a 0% approval rate. ​

    To prevent this from happening, Amazon should clarify the labeling on the 'Manage Workers' page. And I think it would also be beneficial if MTurk would display workers' real overall stats (which are often 99% with thousands of completed HITs) to requesters somewhere in the requester GUI, in addition to the workers' stats from just that one requester.

    What these pages look like, from the documentation link above:

    [​IMG]

    [​IMG]


    In March 2015, I came across another example of this in the MTurk requester support forum, posted March 11, 2015 in the midst of some other confusion... at least this time a more knowledgeable requester was able to set them straight before any damage was done.

    [​IMG]


    And another requester made this same mistake on Dec 30, 2015.
     
  2. clickhappier

    clickhappier
    Expand Collapse
    ┬──┬ ノ( ゜-゜ノ)

    • Subforum Curator
    • Crowd Pleaser
    Joined:
    Jan 12, 2016
    Messages:
    727
    Likes:
    1,601
  3. clickhappier

    clickhappier
    Expand Collapse
    ┬──┬ ノ( ゜-゜ノ)

    • Subforum Curator
    • Crowd Pleaser
    Joined:
    Jan 12, 2016
    Messages:
    727
    Likes:
    1,601
    #3 clickhappier, Jan 26, 2016
    Last edited: Jan 26, 2016
    Tribune likes this.