11/14 - Thoroughbred Thursday

[aveline]

I'm not disagreeing that Amazon can't do better (they can, but they also are trying to), but that in particular is way beyond qualifications & a large part of what's going on cannot be solved just by qualifications alone. It's ultimately Amazon's responsibility, but they are doing what needs to be done and that unfortunately means culling a significant amount of accounts that have been stolen, created from identity theft, sold, farmed out, or just straight up belong to people who have been taking advantage of the fact they can scream bloody murder at an IRB and/or threaten someone else's job to prop up their poor work quality and ethic.

It definitely sucks that workers don't really have a voice in the overarching discussion and that some people are going to get nastygrams or even unfairly suspended but I wouldn't shed a single tear for anyone in that particular Tweet not being able to take a survey/HIT I want to do.

LITTLE BIT QUALITY

One of the tweets mentioned bots getting around a screener, and I do think a qual test that doesn't allow retakes or a screener HIT could absolutely lessen the impact of that.

I also said I agree with those requesters' complaints and that Amazon should do something about it. My whole thing is I don't like it being handled over Twitter. It just seems unprofessional to me and it somehow makes me feel even more powerless than I already did. But I think Twitter is a dumpster fire anyway, so maybe it's just me. ?‍

 

[TotalBabe]

HIT Time: ~1 minute

Title: Answer 5 questions about your political orientation (~1 mins) | Accept Requester: Positly [A20SRJ4UG9W0H0] Contact TV: [Hrly=$15.58] [Pay=Good] [Approval=~24 hrs] [Comm=Excellent] [Rej=1] [Blk=0] TO: [Pay=2.73] [Fast=3.71] [Comm=2.80] [Fair=3.66] [Reviews=53] [ToS=1] TO2: No Reviews Reward: 0.25 Duration: 6:00 Available: 6 Description: Answer 5 questions about your political orientation. Please take this seriously, as your answers here will qualify you for future HITs. Qualifications: 34ee50c9eacb4e076cedf82013242ddbbbff8013 GreaterThanOrEqualTo 70; c3db8fbdeecc9805712bbc718f158d9642e7c11b NotIn 0; 80412233201bcb15666b2ac446860a99b1c3bf42 DoesNotExist ; 799ea5dfa364efa477b02b12fce61a3b19f7f6eb DoesNotExist ; 824f5e2b31b4bb23f90e8dc795bc58a0ea4fc6fb DoesNotExist ; 84869885c8ca572b0ebf572fb5fe48f288f072fe NotIn 0; a4accd9e2bf3b27873aaf0e9e8c5b76176a683eb NotIn 0; Total approved HITs GreaterThanOrEqualTo 500; cd4ae775c2c8fc799fef2528d9e339726b91a0ff NotIn 0; HIT approval rate (%) GreaterThanOrEqualTo 96; Location EqualTo US; ab5c70a2a750f5b115e550db4b754b92dfd32213 DoesNotExist

 

[aveline]

Out of curiosity what is this tweet we're talking about.....?

One was posted here:

I've said this elsewhere, and just to be clear I don't have any sort of contacts at Amazon or anything it's just my personal observations and I'm not taking sides/judging/whatever, but I think these emails are a lot more high level than just any X/Y batch. MTurk has everyone's entire work history from day one, it's not like they cannot look into things with some fairly simple queries and pull accounts with X # blocks / % above avg submission on Z batch / etc to start identifying accounts that might be sources of data quality that might be undesirable to requesters. People have a really hard time disassociating their work effort with their work product and we all know on MTurk we don't really get paid for effort.

From what little we get to see they've divulged reaching out for datasets specifically looking for low quality work & the VPN'd up accounts. This one was posted this week:

So even being cheeky and thinking putting "VERY NICE" into the comment box on a survey is funny (something I've seen more than a few workers mention doing) might come back to bite you 6 months later. Could be anything similar from work history. There is a requester resource that specifically looks for some stupid things & associates a quality scored based on it & no one will know if/when the dataset is shared w/ MTurk and/or how they process it (if at all).

Maybe some emails are errors, maybe not, but it's a good opportunity to remember that a lot of things that are just a byproduct of being a human worker can be really damning when tossed into an algorithm that sorts us all based on input we don't get a say in.

 

[DCI]

I'm not disagreeing that Amazon can't do better (they can, but they also are trying to), but that in particular is way beyond qualifications & a large part of what's going on cannot be solved just by qualifications alone. It's ultimately Amazon's responsibility, but they are doing what needs to be done and that unfortunately means culling a significant amount of accounts that have been stolen, created from identity theft, sold, farmed out, or just straight up belong to people who have been taking advantage of the fact they can scream bloody murder at an IRB and/or threaten someone else's job to prop up their poor work quality and ethic.

It definitely sucks that workers don't really have a voice in the overarching discussion and that some people are going to get nastygrams or even unfairly suspended but I wouldn't shed a single tear for anyone in that particular Tweet not being able to take a survey/HIT I want to do.

LITTLE BIT QUALITY

That'd be fine if they were actually competent, but they're not and they never have been. Mturk has always been understaffed and the staff that they do have is generally clueless about what goes on on the site. There is no team of people writing complex algorithms to analyze long term work quality. Almost everything is done with min effort slapdash nonsense.

About half the people I know that are active on mturk right now, including me, got some form of bot warning today. These are all long time workers that are fairly careful about the work they submit and obviously not bots. Even if their work was being subject to some sort of exhaustive analysis, which they're not, I'm very confident that they would score very well.

 

[mboone73]

Greetings from Amazon Mechanical Turk,

You've received a bonus from Jonathan Rolison for work related to 3MGHRFQY2LVVJSDW6HKWDNJX3ZMY0T.
The value of your bonus is: $2.00 USD

The Requester included this note:
This is your bonus for taking part in our study. I am sorry that this has taken so long

PE: $2.00

 

[afallstarxfall]

You've received a bonus from esclab for work related to 3HY86PZXP05IN7JE7JX64Q310I51EN.
The value of your bonus is: $1.00 USD
You've received a bonus from Emile for work related to 3EKTG13IZV6M8UN45NDEKV0QKXQMLK.
The value of your bonus is: $2.50 USD

The Requester included this note:
Thank you again for participating in our study! You were informed during the task that you would receive an amount between $0 and $5. Half of everyone who participated were randomly selected to receive $0 and half were selected to receive $2.50. In order to be fair, everyone who participated will be given $2.50 as a bonus. Thank you again for your time and attention!

 

[Blakkat]

I'm returning this one in 5 mins, I already did it before

Title: Experiment: Rating virtual character gesticulation (Approx 30 min) | Accept Requester: Patrik Jonell [A362A6894M8EYS] Contact TV: [Hrly=$8.66] [Pay=Fair] [Approval=~24 hrs] [Comm=Unrated] [Rej=0] [Blk=0] TO: [Pay=1.00] [Fast=0.00] [Comm=1.00] [Fair=1.00] [Reviews=3] [ToS=0] TO2: No Reviews Reward: 3.00 Duration: 1:00:00 Available: 0 Description: Do not do this study twice. We are interested in how people perceive a virtual character (a software artifact that looks and acts like a human) when it speaks phrases. Your task is to compare two videos at a time of a virtual character and judge the two videos based on 4 critera. Note! There are control questions. Failing too many of them will result in disqualification and no payment. Qualifications: Not available from queue exports.

 

[thismegan]

One was posted here:

I'm going over two requesters' tweets from that post...

My student and I used @amazonmturk to collect data on @Snapchat use, and 200+ bots figured out the screening criteria and completed the survey. So glad we included open-ended questions to catch them. From now on, I will always include open-ended questions.

Here is a sample response from a bot to the question, "can you say a bit more about how you use Snapchat to follow the news or current events?"

Bot: "Users can stay up-to-date on breaking news, sports, entertainment, and more. How do you add a person or channel to your Snapchat? ...you take a picture of the snapcode with your phone and then open your Snapchat...."

lol, it's like someone who doesn't speak English just typed snapchat into a search engine and copied the text from the first link into the response box. I wonder why it's so hard to screen out foreign workers who can't possible respond to questions given in English.

 

[coffeeowl]

Let me back in Data Science. It's been a month.

 

[balsaer]

I'm so afraid I'm going to lose my account I'm having an anxiety attack. I don't use scripts beyond turkmaster and the mts extension. Wtf did I do wrong to get this scary ass email?

I'm right there with you. Anxiety attack, and only using PandaCrazy and MTS.

I use separate browsers though. I wonder if it could be that the cookies (that we discard to avoid the too big error) raise flags. But other people are only using a single browser and single profile.

From discussions I've had, there's no commonality of the tasks (batches/requesters).

There's also, you know, no rejections or blocks.

What stinks the most is there's just no one to appeal to. It's (for me and likely many of us) our livelihoods on the line, and there's just not zip we have for recourse.

I emailed the mturk-worker support email and got a nice form response that my question would be forwarded to the investigation team for review. Anyone else getting this email should respond as well; bots don't tend to get irate, but by golly we're Americans, and we do. (No, not you Canadians, ya'll too polite for that sort of thing! )

 

[Blakkat]

Title: Breast Cancer Survivor Experiences | Accept Requester: Kate Sweeny [A2Z9EJ9KFEE1CW] Contact TV: [Hrly=$13.54] [Pay=Good] [Approval=~24 hrs] [Comm=Unrated] [Rej=0] [Blk=0] TO: [Pay=3.26] [Fast=4.73] [Comm=4.00] [Fair=5.00] [Reviews=34] [ToS=1] TO2: No Reviews Reward: 2.00 Duration: 12:00:00 Available: 0 Description: You will reflect on and write about your experience with breast cancer and complete measures of well-being. Qualifications: Not available from queue exports.

 

[MinkyMona]

I'm right there with you. Anxiety attack, and only using PandaCrazy and MTS.

I use separate browsers though. I wonder if it could be that the cookies (that we discard to avoid the too big error) raise flags. But other people are only using a single browser and single profile.

From discussions I've had, there's no commonality of the tasks (batches/requesters).

There's also, you know, no rejections or blocks.

What stinks the most is there's just no one to appeal to. It's (for me and likely many of us) our livelihoods on the line, and there's just not zip we have for recourse.

I emailed the mturk-worker support email and got a nice form response that my question would be forwarded to the investigation team for review. Anyone else getting this email should respond as well; bots don't tend to get irate, but by golly we're Americans, and we do. (No, not you Canadians, ya'll too polite for that sort of thing! )

I sent them an email but didn't receive a response. Honestly I don't even feel like turking anymore after this week. I'm still missing money, got this stupid email and getting an error that I don't have permission when trying to transfer from AP to my bank. I feel like taking that toaster into the shower.

 

[JenniLeigh]

Greetings from Amazon Mechanical Turk,

You've received a bonus from Jonathan Rolison for work related to 3MGHRFQY2LVVJSDW6HKWDNJX3ZMY0T.
The value of your bonus is: $2.00 USD

The Requester included this note:
This is your bonus for taking part in our study. I am sorry that this has taken so long

PE: $2.00

Wow.

You did a survey?

 

[TotalBabe]

HIT Time: ~3:50

Title: Guessing temperatures(~ 5 minutes) | Accept Requester: Moore Lab [A33V7NSHY0CNZ2] Contact TV: [Hrly=$25.41] [Pay=Generous] [Approval=~24 hrs] [Comm=Unrated] [Rej=0] [Blk=0] TO: [Pay=3.47] [Fast=3.74] [Comm=2.45] [Fair=3.85] [Reviews=114] [ToS=0] TO2: No Reviews Reward: 0.65 Duration: 2:00:00 Available: 1 Description: In this survey, you will guess yearly high temperatures of cities in the United States. Qualifications: Exc: [220528027-180682] DoesNotExist ; Total approved HITs GreaterThanOrEqualTo 100; HIT approval rate (%) GreaterThanOrEqualTo 99; Location In US

 

[Markb]

Caught some crowd-testing but they seem to be broken. Working for anyone?

 

[Blakkat]

Title: Study on Decision Making (read documents related to a lawsuit)(~ 30 minutes) | Accept Requester: David Hagmann [A2OZJB9LYMCMUC] Contact TV: [Hrly=$15.99] [Pay=Good] [Approval=~24 hrs] [Comm=Unrated] [Rej=0] [Blk=0] TO: [Pay=4.18] [Fast=4.81] [Comm=4.00] [Fair=4.76] [Reviews=68] [ToS=0] TO2: No Reviews Reward: 1.00 Duration: 3:00:00 Available: 0 Description: Approximately 30 minutes. Up to $7.50 in additional bonuses. Qualifications: Not available from queue exports.

 

[ChrisTurk]

One of the tweets mentioned bots getting around a screener, and I do think a qual test that doesn't allow retakes or a screener HIT could absolutely lessen the impact of that.

I also said I agree with those requesters' complaints and that Amazon should do something about it. My whole thing is I don't like it being handled over Twitter. It just seems unprofessional to me and it somehow makes me feel even more powerless than I already did. But I think Twitter is a dumpster fire anyway, so maybe it's just me. ?‍

Hah, def agree Twitter isn't the ideal place, but it's where a lot of academic discussion happens (irony abounds) & so that's just MTurk taking advantage of an opportunity.

But this is where I think workers not being in that discussion really sucks, because the requester is seeing the effect (bad data quality) and just assuming the cause (bots) and it's a really bad look for anyone who knows better, but the problem is still there.

Screeners are pretty ineffective against people trained in avoiding them & qualtrics/monkey has security problems in that venue anyway. Quals present huge sampling problems to many academic requesters (nvm the technical problem of managing that + attrition) but, again, there are definitely paths to take to improve data quality as a requester that some just straight up skip out on then complain when they get what they put in (garbage in, garbage out )

Def appreciate the point on Twitter.. I've been following academic circles more closely lately and I think as often as we remove things on forums thinking "whoa wouldn't want a requester to see that" some of them would be real flummoxed if half a dozen Turkers showed up on campus having read their SM posts

That'd be fine if they were actually competent, but they're not and they never have been. Mturk has always been understaffed and the staff that they do have is generally clueless about what goes on on the site. There is no team of people writing complex algorithms to analyze long term work quality. Almost everything is done with min effort slapdash nonsense.

About half the people I know that are active on mturk right now, including me, got some form of bot warning today. These are all long time workers that are fairly careful about the work they submit and obviously not bots. Even if their work was being subject to some sort of exhaustive analysis, which they're not, I'm very confident that they would score very well.

I've been on the cracking side of too many jokes at MTurk's expense to argue with that haha. But I've spoken to a lot of workers I know submit high quality data who also got the emails.. but most understand why they'd stick out at least? It doesn't have to be like a countdown to Armageddon thing.

100% agree if you take the message as impending doom it's a poop sandwhich for sure. But I don't think it's unreasonable to think someone did like "grab me all workers who submit HITs 50% faster than avg" => take a peek => suspend scammers => email the rest a head's up that, even if your work quality is good, you stick out versus the average so you match our queries for "bots" keep an eye on yourself. That's pretty much the minimum effort MO you're alluding to honestly. Which is why it's good to know you might get fed through an algorithm/query on any given day.

YMMV, that's been my take on it speaking to some folks who I'd say the exact same thing about when talking it through. Obviously all anyone here can do is speculate out of their backsides as to the mystery behind the curtain lol but like someone else said it's totally possible to get that email and suffer no ill effects.

 

[coffeeowl]

Hah, def agree Twitter isn't the ideal place, but it's where a lot of academic discussion happens (irony abounds) & so that's just MTurk taking advantage of an opportunity.

But this is where I think workers not being in that discussion really sucks, because the requester is seeing the effect (bad data quality) and just assuming the cause (bots) and it's a really bad look for anyone who knows better, but the problem is still there.

Screeners are pretty ineffective against people trained in avoiding them & qualtrics/monkey has security problems in that venue anyway. Quals present huge sampling problems to many academic requesters (nvm the technical problem of managing that + attrition) but, again, there are definitely paths to take to improve data quality as a requester that some just straight up skip out on then complain when they get what they put in (garbage in, garbage out )

Def appreciate the point on Twitter.. I've been following academic circles more closely lately and I think as often as we remove things on forums thinking "whoa wouldn't want a requester to see that" some of them would be real flummoxed if half a dozen Turkers showed up on campus having read their SM posts

I've been on the cracking side of too many jokes at MTurk's expense to argue with that haha. But I've spoken to a lot of workers I know submit high quality data who also got the emails.. but most understand why they'd stick out at least? It doesn't have to be like a countdown to Armageddon thing.

100% agree if you take the message as impending doom it's a poop sandwhich for sure. But I don't think it's unreasonable to think someone did like "grab me all workers who submit HITs 50% faster than avg" => take a peek => suspend scammers => email the rest a head's up that, even if your work quality is good, you stick out versus the average so you match our queries for "bots" keep an eye on yourself. That's pretty much the minimum effort MO you're alluding to honestly. Which is why it's good to know you might get fed through an algorithm/query on any given day.

YMMV, that's been my take on it speaking to some folks who I'd say the exact same thing about when talking it through. Obviously all anyone here can do is speculate out of their backsides as to the mystery behind the curtain lol but like someone else said it's totally possible to get that email and suffer no ill effects.

tl;dr

 

[Yatagarasu]

Let me back in Data Science. It's been a month.

They should have "confused" as an option on these.

 

[MinkyMona]

Well that's the joy of having an anxiety disorder everything is scary and doom and gloom when it catches you off guard. That's me today and I have no one to talk to about it. I feel like Amazon has been picking on me all week. I've seen so much shit happen with this company over the years and they never fail to cause unwarranted stress by not being transparent. I think I'm really tired of it.