Press/Studies Another article about using mturk (Jun 2015)

jan

Moderator
Moderator
Joined
Jan 12, 2016
Messages
25,800
Reaction score
49,513
Points
1,463
Gender
Female
Caution! MTurk Workers Ahead —Fines Doubled
research by P. D. Harms University of Alabama Justin A. DeSimone Univ

wow, these guys don't like us at all :(

For one, they think because survey respondents come from a dedicated pool of mturkers, rather than representative of the entire mturk pool, we don't represent the population well - especially in organizational research. Worse, imo (and almost revealing) is the pointed and condescending way they evaluate us as a group.
For example here:
"........
many Turkers do sincerely consider what they do to be a job like any other.
Websites like mturkgrind.com, wearedynamo.org, and turkopticon have been
used to advocate for MTurk worker rights. In particular, it has been suggested
that the rates of pay for projects be benchmarked to the U.S. federal
minimum wage. Researchers paying less than this rate may be contacted by
self-appointed “labor reps” pointing them to these sites and threatening to
contact university institutional review boards."

Sadly, they draw their conclusions - in part - from this report:
"We would even go so far as to say that investigators may want to consider paying for bad data as simply a cost of doing research online. Based on personal experience, it is not
worth saving 30 cents to have someone call your house and threaten to drive
from Colorado to Nebraska so they can stab you because you did not pay
them."

I found it here.
https://www.researchgate.net/profile/Peter_Harms/publications

It was pretty discouraging (they even had a whole paragraph on studies that found poor data -respondants reporting different genders on different studies, etc. ).
 
Last edited:
  • Like
Reactions: coffeeowl

coffeeowl

hoo?
Contributor
Mentor
Master Pleaser
Crowd Pleaser
HIT Poster
Joined
Jan 12, 2016
Messages
28,032
Reaction score
63,932
Points
1,738
Location
Location EqualTo US
Gender
Female
Caution! MTurk Workers Ahead —Fines Doubled
research by P. D. Harms University of Alabama Justin A. DeSimone Univ

wow, these guys don't like us at all :(

For one, they think because survey respondents come from a dedicated pool of mturkers, rather than representative of the entire mturk pool, we don't represent the population well - especially in organizational research. Worse, imo (and almost revealing) is the pointed and condescending way they evaluate us as a group.
For example here:
"........
many Turkers do sincerely consider what they do to be a job like any other.
Websites like mturkgrind.com, wearedynamo.org, and turkopticon have been
used to advocate for MTurk worker rights. In particular, it has been suggested
that the rates of pay for projects be benchmarked to the U.S. federal
minimum wage. Researchers paying less than this rate may be contacted by
self-appointed “labor reps” pointing them to these sites and threatening to
contact university institutional review boards."

Sadly, they draw their conclusions from this report:
"We would even go so far as to say that investigators may want to consider paying for bad data as simply a cost of doing research online. Based on personal experience, it is not
worth saving 30 cents to have someone call your house and threaten to drive
from Colorado to Nebraska so they can stab you because you did not pay
them."

I found it here.
https://www.researchgate.net/profile/Peter_Harms/publications

It was pretty discouraging (they even had a whole paragraph on studies that found poor data -respondants reporting different genders on different studies, etc. ).
Someone got pitchforked. Ugh!
 
  • Like
Reactions: jan

Lepi

Some Kind of Sneaky Witch Thief
Contributor
Joined
Jan 12, 2016
Messages
12,892
Reaction score
8,050
Points
1,213
Gender
Female
Well. It's not exactly breaking news that people lie on surveys. While I would like to think that most of us are honest, it wasn't all that rare for people to show up on the forums and go "LOL I lie on surveys it's cool!".

There's this gulf between workers and requesters that make it frustrating for both sides. Requesters shouldn't feel like they have to pay for "bad" data. If someone took your survey and lied about their qualifications, you shouldn't have to pay for that. If someone ignored your instructions or your attention checks (reasonable, well written instructions and attention checks mind you), you shouldn't have to pay them. On the flip side, workers shouldn't have to eat a rejection because you screwed up your survey in some way.

So from the requester side, you have a set amount of money that you have to use to get x amount of results. If you have a set of screener questions, people bitch about you wasting their time with screenrs. If you don't screen people out, you get assholes who lie about their age/gender/work history in order to take the survey. If you pay a poor wage, no one wants to work from you (and apparently people threaten to stab you). If you pay a good wage, you naturally attract more people who want to scam you.

On the worker side, you can have your work rejected for whatever reason the requester sees fit and you have no recourse whatsoever. If you screw up their own survey and forget to ask a question, they can reject you. If the survey doesn't generate a code, they can reject you. If they think you did the survey too fast, they can reject you. If they hit the wrong button, they can reject you. And every rejection has an impact on your ability to work. They can also block your account which may or may not be a problem, depending on which rep you talk to from Amazon. So workers are largely powerless- ultimately all we can do is do the work to the best of our abilities and hope that the requester takes it. And even if you do everything right, you can still get bitch slapped if the requester is in a mood.
 
  • Like
Reactions: jan and essjay

Twirdman

Member
Joined
Jan 18, 2016
Messages
298
Reaction score
603
Points
343
Age
35
Gender
Male
Some of the problems in the article are either non-existent or are easily gotten around. Like they point out how non-US turkers represent a large amount of professional turkers but a large chunk of those can simply be qualled out using the premade location qual from mturk; while it is true some people in foreign countries have US based mturk accounts I'm guessing that number is small compared to the total number of turkers and foreign turkers. These people simply got their account before mturk banned international workers. They mention self selection problems because people leave the survey this can be solved by paying a reasonable wage for the amount of time involved. Obviously nobody is going to work for an hour for like 3 dollars this is true whether you are using an online sample or a physical sample in the real world. The problem is researchers think they can get away with paying turkers significantly less than they would ever consider paying a normal sample. I mean hell look at the numbers quoted 20 minutes for 1.00 or 40 minutes for 1.50 if someone offered this to you in person you would also self screen. They also mention that forums exist where people share attention checks and screeners but of multiple forums I've been on related to mturk this was actually specifically disallowed the best I've seen is it is OK to share the fact that there exist a screener, but as long as you don't design your title or hit to be obvious what demographic you are looking for knowing that a screener exist is not going to allow people to circumvent it.

Some of the other problems are more difficult to fix but can also be handled with a bit of cooperation. Like the issue of people lying about certain things can be worked by having certain researchers issue a qual based on those and then allowing other researchers to use those quals. Obviously this will only work for certain unchanging things like gender or date of birth but could stop some of the problems they were having. There is also the use of paid screeners. While people are upset, arguably unrightly so, about unpaid screeners very few people are upset about paid screeners.

Also I found it incredibly annoying that they did not give any data on how the data obtained from mturk differed from data obtained from other sources. If you are doing a paper saying why mturk data is bad or how it needs robust data validation methods you should show how it compares to traditional methods of data acquisition. The next paper seems like a far more interesting and useful paper.
 
  • Like
Reactions: essjay and jan

Lepi

Some Kind of Sneaky Witch Thief
Contributor
Joined
Jan 12, 2016
Messages
12,892
Reaction score
8,050
Points
1,213
Gender
Female
They also mention that forums exist where people share attention checks and screeners but of multiple forums I've been on related to mturk this was actually specifically disallowed the best I've seen is it is OK to share the fact that there exist a screener, but as long as you don't design your title or hit to be obvious what demographic you are looking for knowing that a screener exist is not going to allow people to circumvent it.
I've seen that mentioned in more than one article, but I have no idea which forums they're referring to, because...yeah. MTG, MTC, MTF, and TN all have rules specifically disallowing the sharing of attention checks. Honestly, I think the widespread sharing of attention checks/answers is just one of those boogeymen than requesters pass around at this point.
 

Twirdman

Member
Joined
Jan 18, 2016
Messages
298
Reaction score
603
Points
343
Age
35
Gender
Male
I've seen that mentioned in more than one article, but I have no idea which forums they're referring to, because...yeah. MTG, MTC, MTF, and TN all have rules specifically disallowing the sharing of attention checks. Honestly, I think the widespread sharing of attention checks/answers is just one of those boogeymen than requesters pass around at this point.
Yep. It is also disallowed on the mturk member facebook group. I have literally never seen a place that allows it. Very rarely you will see a new member do it because they don't know better and then mods quickly edit the post and the person is quickly told to never do that again and he doesn't. I'm inclined to agree it is just a boogeyman passed from requester to requester.