'Lifetime Approval Rate' Misunderstood by Requesters

clickhappier

┬──┬ ノ( ゜-゜ノ)
Subforum Curator
Crowd Pleaser
Joined
Jan 12, 2016
Messages
728
Reaction score
1,634
Points
593
Location
USA
There have been multiple incidents where inexperienced requesters misunderstood the 'Lifetime Approval Rate' stats displayed to them in the Requester GUI.

On Jan 23, 2015, a requester rejected multiple workers with a feedback note saying "One of the requirements was a lifetime approval rate of +90%. You do not meet these requirements. Sorry." In his apology email, he said "I'm very sorry this happened. MTURK showed my an approval rate of 0%. It was not my intention to reject proper responses." This requester was able to reverse the rejections after workers informed him of his mistake.

On Feb 19, 2015, another requester rejected multiple workers with a feedback note saying just "lifetime approval rate". Apparently this requester will be unable to reverse these rejections without assistance from MTurk support, because she had promptly irrevocably disposed of the batch (which ought to not even be allowed by MTurk's programming, btw).

Both requesters did have qualification requirements set on their HITs for their desired approval rates (90% and 95%). Here's some comments I wrote about these incidents:

"MTURK showed my an approval rate of 0%."

Heh. I'm guessing he was looking at http://docs.aws.amazon.com/AWSMechTurk/latest/RequesterUI/ViewingWorkerDetails.html , where it says the amount/percent of HITs a worker has had approved *by that one requester*. This was most people's first HIT for this requester, and of course it hadn't been approved yet when he looked at that, so I think it would've said "0% (0/0)" or something like that for nearly every worker...​

and

We saw an incident like this with another inexperienced requester a couple weeks ago. ... as I speculated then: I think they're not comprehending that their use of the "HIT approval rate (%) is not less than x" requirement on the HIT made it impossible for anyone below their specified approval % to accept the HIT (with the exception of workers with less than 100 HITs), and then they go look at the only stats that are visible from the requester side for workers' completed HITs, which is the 'lifetime' percent you've had approved for *that* requester... which was 0%, of course. Then they bizarrely decide this means you and all the other workers need to be rejected because you have a 0% approval rate.​

To prevent this from happening, Amazon should clarify the labeling on the 'Manage Workers' page. And I think it would also be beneficial if MTurk would display workers' real overall stats (which are often 99% with thousands of completed HITs) to requesters somewhere in the requester GUI, in addition to the workers' stats from just that one requester.

What these pages look like, from the documentation link above:






In March 2015, I came across another example of this in the MTurk requester support forum, posted March 11, 2015 in the midst of some other confusion... at least this time a more knowledgeable requester was able to set them straight before any damage was done.




And another requester made this same mistake on Dec 30, 2015.