Requester Question to MTurkers on Alternatives to HIT Pay Reduction

[Annabel]

Hi! My name is Annabel, and I'm a requester on MTurk (http://turkopticon.ucsd.edu/A3HB3KPJ2RZ2K8). I'm on here to ask you guys how I can make my HITs more efficient so I can keep running them. Right now, my funding is becoming a bit limited, but our project still needs a lot more information.

I've decreased the amount of pay for my HITs in the past before, but I'd like to avoid decreasing the amount of pay again, especially since I'd like to keep a good relationship with the people who've been working on my HITs. Each HIT currently pays $0.50. I was advised to lower this amount to $0.20, but I am trying to avoid this.

Right now, my HITs revolve around asking people to compare two sentences with each other, and to select the sentence that is the more important one. (Title: Comparing two statements with each other) The goal of these HITs is to collect enough information to create an app or a website that can automatically read the Terms & Conditions and give people only the important parts.

I do have an idea to address this (change the task so that people compare groups of sentences with each other so we get more data, and slightly increase the base pay), but was wondering whether there were other ways out there.

 

[Jaded]

First, I think it's awesome that you don't want to lower the pay again. I know last time I did your hits (it's been a while since I've seen them,) it was one question per page which caused a bit of loading time. If there were more questions per page (and fewer pages) it might make them quicker so that a pay reduction wouldn't really impact how much one was earning from the hits.

Hopefully, that makes sense.

 

[LurkyTurkey]

Welcome! I recall working on these HITs before and some of the sentences were equally ambiguous, making it hard to give a useful answer. Clauses would help address that, but it might also be useful to specify what is relevant.

If you want to PM me an example or link to refresh my memory, I can give more specific feedback. I do recall the T&C weren't the typical ones I encounter as a shopper. It seems like they included info about subscriptions and maybe even litigation.

 

[chuchotant]

Hi A @Annabel . I just said to my mom this morning, I miss Annabel. Remember when it was Annabel weekend. funny that you should sign up! Welcome. And thank you so much for thinking of the workers and reaching out to us!

I remember specifically thinking that it would be easier if all of the 10 binary comparisons were on one page (like @Jaded suggested). Another thing, some comparisons seemed to be made between really inconsequential information (ie "These are the terms and conditions") and really important legal stuff (ie statements about arbitration etc). I'm not sure if you have enough data already, but to group the inconsequential statements with each other so that I am making a choice between in consequential statements and group the heavy stuff together so I am making a choice between heavy stuff.
I felt like I wasnt being really that helpful to you if I chose an obviously more important statement when the only other option was a statement just telling me these were the terms and conditions.
In other words, here was the choice I was making on many of the choices before:

These are the TOC vs Important Legal Stuff

Whereas if you could group them:
These are TOC vs Customer service number is 800-
Important Legal Stuff vs Important Legal Stuff

I hope this makes sense.

 

[LurkyTurkey]

Hi A @Annabel . I just said to my mom this morning, I miss Annabel. Remember when it was Annabel weekend. funny that you should sign up! Welcome. And thank you so much for thinking of the workers and reaching out to us!

I remember specifically thinking that it would be easier if all of the 10 binary comparisons were on one page (like @Jaded suggested). Another thing, some comparisons seemed to be made between really inconsequential information (ie "These are the terms and conditions") and really important legal stuff (ie statements about arbitration etc). I'm not sure if you have enough data already, but to group the inconsequential statements with each other so that I am making a choice between in consequential statements and group the heavy stuff together so I am making a choice between heavy stuff.
I felt like I wasnt being really that helpful to you if I chose an obviously more important statement when the only other option was a statement just telling me these were the terms and conditions.
In other words, here was the choice I was making on many of the choices before:

These are the TOC vs Important Legal Stuff

Whereas if you could group them:
These are TOC vs Customer service number is 800-
Important Legal Stuff vs Important Legal Stuff

I hope this makes sense.

Yes. This seems like it would yield more substantial, useful answers and perhaps avoid a pay reduction by generating better results.

 

[Annabel]

I should also mention, I'll be outputting another batch this weekend (at the same price) to hopefully reach out to all of the MTurkers who usually work on my HITs to ask for their input too. Feel free to just repeat your suggestions there in the box if you decide to do it!

First, I think it's awesome that you don't want to lower the pay again. I know last time I did your hits (it's been a while since I've seen them,) it was one question per page which caused a bit of loading time. If there were more questions per page (and fewer pages) it might make them quicker so that a pay reduction wouldn't really impact how much one was earning from the hits.

Hopefully, that makes sense.

Oh! Good point. I'll list creating just one page full of comparisons as an option.

Welcome! I recall working on these HITs before and some of the sentences were equally ambiguous, making it hard to give a useful answer. Clauses would help address that, but it might also be useful to specify what is relevant.

If you want to PM me an example or link to refresh my memory, I can give more specific feedback. I do recall the T&C weren't the typical ones I encounter as a shopper. It seems like they included info about subscriptions and maybe even litigation.

Thank you!

The link to the website is right here: http://crowdverify.herokuapp.com/
Generating a list of similarly important sentences might be a bit tricky, especially since the project is about figuring out what consumers care about!

 

[Annabel]

Forgot to respond to the other comments! Also, I'll be outputting another batch this weekend (at the same price) to hopefully reach out to all of the MTurkers who usually work on my HITs to ask for their input too. Feel free to just repeat your suggestions there in the box if you decide to do it!

Hi A @Annabel . I just said to my mom this morning, I miss Annabel. Remember when it was Annabel weekend. funny that you should sign up! Welcome. And thank you so much for thinking of the workers and reaching out to us!

I remember specifically thinking that it would be easier if all of the 10 binary comparisons were on one page (like @Jaded suggested). Another thing, some comparisons seemed to be made between really inconsequential information (ie "These are the terms and conditions") and really important legal stuff (ie statements about arbitration etc). I'm not sure if you have enough data already, but to group the inconsequential statements with each other so that I am making a choice between in consequential statements and group the heavy stuff together so I am making a choice between heavy stuff.
I felt like I wasnt being really that helpful to you if I chose an obviously more important statement when the only other option was a statement just telling me these were the terms and conditions.
In other words, here was the choice I was making on many of the choices before:

These are the TOC vs Important Legal Stuff

Whereas if you could group them:
These are TOC vs Customer service number is 800-
Important Legal Stuff vs Important Legal Stuff

I hope this makes sense.

What a coincidence! One of your admins ( @Jaded) reached out to me about this site earlier today, and of course I had to sign up!

And thank you for your kind words! I'll see if I can put them all on one page. As for the inconsequential items, as mentioned in response to an earlier comment, that is a bit tricky since we're trying to figure out what people find important with this round of studies. For the obvious cases like 1-800 number, that shouldn't be a problem, but for other stuff, that might not be possible

I could go more into detail about why, but I'm not quite sure if anyone wants to read a really long post on it!

 

[Jaded]

I could go more into detail about why, but I'm not quite sure if anyone wants to read a really long post on it!

You'd be surprised!

 

[Annabel]

You'd be surprised!

Hm. Let me think of a way to shorten the explanation then -

In order to create HITs where the sentences in the comparison are either both unimportant or important, we would need to figure out a way to know how important the sentences are in the first place.

Solution #1: The super long, slow way is for me to read all of the sentences, and then choose which ones to line up in a HIT. (Probably not possible since people could argue that what I think is important isn't actually that important to other people.)

Solution #2: The less slow way is to have something like a competitive ranking system, where sentences that aren't selected as important over other sentences are assigned a lower score. (We would need to code the website differently which might take a bit of time! We are considering using this though)

Solution #3: Create a computer program that can automatically detect what's important or not, and then create HITs based off of how important those sentences are.

There are two main ways of doing this:
1) Sentence matching: This means that we look at how important a sentence is judged to be in one Terms & Conditions policy, and search for all sentences that are the same or close to the same in other Terms & Conditions policies. One potential problem here is that a sentence that is judged as important in one Terms & Conditions may not actually be important in another one. We are trying to address this issue right now actually. Not very easy.
2) Machine-learning algorithms: This means that we use existing algorithms to figure out how important a sentence is. This is more accurate than sentence matching, but needs a lot of data to work (which we are trying to collect).

 

[chuchotant]

Hm. Let me think of a way to shorten the explanation then -

In order to create HITs where the sentences in the comparison are either both unimportant or important, we would need to figure out a way to know how important the sentences are in the first place.

Solution #1: The super long, slow way is for me to read all of the sentences, and then choose which ones to line up in a HIT. (Probably not possible since people could argue that what I think is important isn't actually that important to other people.)

Solution #2: The less slow way is to have something like a competitive ranking system, where sentences that aren't selected as important over other sentences are assigned a lower score. (We would need to code the website differently which might take a bit of time! We are considering using this though)

Solution #3: Create a computer program that can automatically detect what's important or not, and then create HITs based off of how important those sentences are.

There are two main ways of doing this:
1) Sentence matching: This means that we look at how important a sentence is judged to be in one Terms & Conditions policy, and search for all sentences that are the same or close to the same in other Terms & Conditions policies. One potential problem here is that a sentence that is judged as important in one Terms & Conditions may not actually be important in another one. We are trying to address this issue right now actually. Not very easy.
2) Machine-learning algorithms: This means that we use existing algorithms to figure out how important a sentence is. This is more accurate than sentence matching, but needs a lot of data to work (which we are trying to collect).

Hi A @Annabel
I just did your new set of hits and noticed a few changes which are a bit different than the last batch!
Question:
On the solution 2 way: For comparisons that are not deemed equally important (where a choice is actually made) could a question be generated to ask what score on a scale of 1-5 the respondent would give each sentence. Then all the same numbers could be grouped together for comparisons. You would get a variety of what some people view as a 5 vs others, but then the 5's could be paired off against each other (tournament style, lol) etc. Is that what you were thinking? We as workers could add some more value to the comparison making that way.

On the solution 3 type 1 way, that was sort of a question I emailed you (I emailed you because I didnt know that question 2 was mandatory in your hit). I noticed the names of the companies were replaced with "e-commerce" site, but it does make a difference when I consider the type of company (retailer vs content publisher) that the terms apply to.

Hopefully you get some more really good suggestions!

 

[Itsmee]

Server would only let me advance to the 2nd set of comparisons before the crash.... Refreshed and refreshed...We will help you figure something out.

 

[Annabel]

Hi A @Annabel
I just did your new set of hits and noticed a few changes which are a bit different than the last batch!
Question:
On the solution 2 way: For comparisons that are not deemed equally important (where a choice is actually made) could a question be generated to ask what score on a scale of 1-5 the respondent would give each sentence. Then all the same numbers could be grouped together for comparisons. You would get a variety of what some people view as a 5 vs others, but then the 5's could be paired off against each other (tournament style, lol) etc. Is that what you were thinking? We as workers could add some more value to the comparison making that way.

On the solution 3 type 1 way, that was sort of a question I emailed you (I emailed you because I didnt know that question 2 was mandatory in your hit). I noticed the names of the companies were replaced with "e-commerce" site, but it does make a difference when I consider the type of company (retailer vs content publisher) that the terms apply to.

Hopefully you get some more really good suggestions!

Hm, that could be a way to do it! I will list this as an option! Thank you!

Server would only let me advance to the 2nd set of comparisons before the crash.... Refreshed and refreshed...We will help you figure something out.

Thank you! I will try and figure out what the problem is once the batch expires.

 

[chuchotant]

Server would only let me advance to the 2nd set of comparisons before the crash.... Refreshed and refreshed...We will help you figure something out.

Instead of refreshing when the server crashed, I just exxed out of the screeen and hit the link (in the mturk window) again. It put me right back on the page and at the number of completed comparisons. Hope that helps!

 

[Itsmee]

Instead of refreshing when the server crashed, I just exxed out of the screeen and hit the link (in the mturk window) again. It put me right back on the page and at the number of completed comparisons. Hope that helps!

I tried that and it didn't work for me. I was using Firefox...

 

[Annabel]

I tried that and it didn't work for me. I was using Firefox...

It's probably either one of two things (none of which are your fault):
1) Too many people on at the same time, so I would need to get more servers
2) I coded the website wrong when trying to get it to compare statements from different websites!

I think it's probably the second one, but will make sure once the HIT has been taken by everyone!

 

[Itsmee]

It's probably either one of two things (none of which are your fault):
1) Too many people on at the same time, so I would need to get more servers
2) I coded the website wrong when trying to get it to compare statements from different websites!

I think it's probably the second one, but will make sure once the HIT has been taken by everyone!

Thank you A @Annabel

 

[chuchotant]

Hi A @Annabel ,
Not sure if you know, but when I enter the link to do a new set of 40, a message pops up saying You're done. I know I'm not done, and I go ahead and do my 40, but wanted to alert you that it comes up. thanks!

 

[Annabel]

Hi A @Annabel ,
Not sure if you know, but when I enter the link to do a new set of 40, a message pops up saying You're done. I know I'm not done, and I go ahead and do my 40, but wanted to alert you that it comes up. thanks!

Thanks for letting me know! Yeah, it's a message that's carried over from the last time you completed the HIT! Someone did reach out to me with a solution for that, but I've been a bit busy lately and haven't been able to use it yet..