Rep Present Hummingbird Watch

HummingbirdWatch

New Member
Requester
Joined
Aug 5, 2016
Messages
39
Reaction score
28
Points
18
Hello MTurk Crowd community,

Our company is interested in trialing with MTurk. We are hoping to have a trial HIT released later this week.

Here's how it works: We would upload images collected by remote cameras, and the workers would look at the images and click yes/no if the image contained a specific item (similar to validating search result images). Speed and accuracy are paramount. These images would have to be processed immediately after being uploaded, and accurately.

We understand that if we want our images to be processed quickly and accurately, we need to maintain a reputation of honesty and fairness with the worker community.

Any advice on making our time with MTurk successful would be appreciated!

Thanks,

HW
 
Last edited:

MerryLou

unpalatable
Contributor
Crowd Pleaser
Sole Survivor
Joined
Jan 12, 2016
Messages
11,256
Reaction score
20,500
Points
1,688
Gender
Female
:wave: Making an account here and seeking out info from 'the other side' is a great way to start!

Setting up a qualification test or a small batch of HITs as a kind of preliminary round for workers to earn a qualification from you is a good way to make sure you'll get consistently accurate results from workers.

Looking forward to trying out your HITs!
 
  • Like
Reactions: HummingbirdWatch

HummingbirdWatch

New Member
Requester
Joined
Aug 5, 2016
Messages
39
Reaction score
28
Points
18
:wave: Making an account here and seeking out info from 'the other side' is a great way to start!

Setting up a qualification test or a small batch of HITs as a kind of preliminary round for workers to earn a qualification from you is a good way to make sure you'll get consistently accurate results from workers.

Looking forward to trying out your HITs!
Thanks for your reply MerryLou. We are currently putting together a few preliminary HITs to replicate those we would find in our actual project. Is there a means to 'soft' reject worker responses, where their reputations wouldn't be affected and we could provide feedback?

Realistically, we'll likely have to tweak our HIT delivery (for example, clearer instructions..), and we don't want good quality workers to pay the price for our 'growing pains'.
 

TSolo315

SnapNCrackle
Administrator
Joined
Jan 12, 2016
Messages
4,761
Reaction score
16,179
Points
2,363
Gender
Male
Thanks for your reply MerryLou. We are currently putting together a few preliminary HITs to replicate those we would find in our actual project. Is there a means to 'soft' reject worker responses, where their reputations wouldn't be affected and we could provide feedback?

Realistically, we'll likely have to tweak our HIT delivery (for example, clearer instructions..), and we don't want good quality workers to pay the price for our 'growing pains'.
No, unfortunately there is no way to reject a HIT without affecting approval rate. It would be a nice feature to have -- a way for requesters to send a HIT back to a worker so they could fix any mistakes.
 

MerryLou

unpalatable
Contributor
Crowd Pleaser
Sole Survivor
Joined
Jan 12, 2016
Messages
11,256
Reaction score
20,500
Points
1,688
Gender
Female
As far as I know, you can also leave a feedback messages on an approved HIT.

If it's feasible, you could post some kind of an example here and them I'm sure you could at least get feedback on the instructions before testing on mturk.
 

HummingbirdWatch

New Member
Requester
Joined
Aug 5, 2016
Messages
39
Reaction score
28
Points
18
No, unfortunately there is no way to reject a HIT without affecting approval rate. It would be a nice feature to have -- a way for requesters to send a HIT back to a worker so they could fix any mistakes.
Thanks for your reply TSolo315. Forgive me if the answer is in another post, but is there perhaps a way to accept the submission that is incorrect, but change the reward to be 0 or 0.01 and provide feedback for the worker? This may sound dangerously close to accepting work without pay, but I'm trying to think of a work around to hurting reputations that don't need to be hurt.
 

HummingbirdWatch

New Member
Requester
Joined
Aug 5, 2016
Messages
39
Reaction score
28
Points
18
As far as I know, you can also leave a feedback messages on an approved HIT.

If it's feasible, you could post some kind of an example here and them I'm sure you could at least get feedback on the instructions before testing on mturk.

Thanks MerryLou for your response. We are hoping to have our first test HIT ready to go by Thursday or Friday, I'll post the HIT instructions on the forum earlier so we can see if there's room for improvement.
 
  • Like
Reactions: clickhappier

HummingbirdWatch

New Member
Requester
Joined
Aug 5, 2016
Messages
39
Reaction score
28
Points
18
You may also want to try the Requester Sandbox to post some practice HITs before you post live HITs.
Thanks electrolyte for your reply. We have looked into the Requester Sandbox, and will definitely be testing our HITs that way. We predict that our staff will be prone to bias and miss things that mturk workers will see pretty quick.
 

TSolo315

SnapNCrackle
Administrator
Joined
Jan 12, 2016
Messages
4,761
Reaction score
16,179
Points
2,363
Gender
Male
Thanks for your reply TSolo315. Forgive me if the answer is in another post, but is there perhaps a way to accept the submission that is incorrect, but change the reward to be 0 or 0.01 and provide feedback for the worker? This may sound dangerously close to accepting work without pay, but I'm trying to think of a work around to hurting reputations that don't need to be hurt.
Well, you can do it backwards. You can set up your trial hits with .01 as the pay and then send the full amount as a bonus to people who answer correctly. You will have to make this clear to workers when you set up the HIT.

Feedback can be given along with bonuses (people receive emails when they get a bonus.) You can send penny bonuses to people who don't answer correctly with feedback as well.
 
Last edited:

clickhappier

┬──┬ ノ( ゜-゜ノ)
Subforum Curator
Crowd Pleaser
Joined
Jan 12, 2016
Messages
728
Reaction score
1,629
Points
593
Location
USA
Thanks electrolyte for your reply. We have looked into the Requester Sandbox, and will definitely be testing our HITs that way. We predict that our staff will be prone to bias and miss things that mturk workers will see pretty quick.
To clarify, your staff would be the ones to do the testing of your sandbox HITs, as workers would receive no real earnings in the sandbox.
 

HummingbirdWatch

New Member
Requester
Joined
Aug 5, 2016
Messages
39
Reaction score
28
Points
18
To clarify, your staff would be the ones to do the testing of your sandbox HITs, as workers would receive no real earnings in the sandbox.
Apologies, I wasn't very clear with my response. What I meant to say was that we would like to do the initial testing in the Requester Sandbox with our own staff. Once we move on from the sandbox into the trial week, we predict more issues to come to light once our staff are no longer the ones doing the HITs. When this becomes the case, it would be beneficial if there were a mechanism for feedback that wouldn't hurt the worker's reputation (or ours). Thanks for the clarification.
 
  • Like
Reactions: clickhappier

HummingbirdWatch

New Member
Requester
Joined
Aug 5, 2016
Messages
39
Reaction score
28
Points
18
Well, you can do it backwards. You can set up your trial hits with .01 as the pay and then send the full amount as a bonus to people who answer correctly. You will have to make this clear to workers when you set up the HIT.

Feedback can be given along with bonuses (people receive emails when they get a bonus.) You can send penny bonuses to people who don't answer correctly with feedback as well.
This method sounds like it could work. Thanks for your input. I've read that dealing with bonuses can be a fairly labor intensive approach for the requester, but it may be the only way to provide feedback during our trial week (without rejecting submissions).
 

Metallica

Never free, never me
Mentor
Contributor
Joined
Jan 17, 2016
Messages
5,207
Reaction score
13,955
Points
1,113
Age
29
Gender
Male
This method sounds like it could work. Thanks for your input. I've read that dealing with bonuses can be a fairly labor intensive approach for the requester, but it may be the only way to provide feedback during our trial week (without rejecting submissions).
I'm not too sure if he's been around lately but you could try to give GlobalWorkForge @GlobalWorkForge a shout. He's pretty experienced with the requester side and has a hand in, or runs multiple different requester accounts.
 
  • Like
Reactions: HummingbirdWatch

Randomacts

Banned
Banned
Joined
Jul 25, 2016
Messages
1,877
Reaction score
1,515
Points
363
Age
23
Gender
Male
This method sounds like it could work. Thanks for your input. I've read that dealing with bonuses can be a fairly labor intensive approach for the requester, but it may be the only way to provide feedback during our trial week (without rejecting submissions).
It can be automated, I'm just not sure how exactly.

You should try and contact the Sticky team https://turkopticon.ucsd.edu/reports?id=A1I5WZRIXY5GS9 they have the whole thing automated although I doubt you could fully automate your process like they do.
 
Last edited:
  • Like
Reactions: HummingbirdWatch

HummingbirdWatch

New Member
Requester
Joined
Aug 5, 2016
Messages
39
Reaction score
28
Points
18
It can be automated, I'm just not sure how exactly.

You should try and contact the Sticky team https://turkopticon.ucsd.edu/reports?id=A1I5WZRIXY5GS9 they have the whole thing automated although I doubt you could fully automate your process like they do.
Thanks for the link, I'll have a look at it and see if we can apply it to our process. Automating our workflow will be a priority once we establish something that works in the meantime.
 

Zacharias

Experiencing Flim Flam Withdrawal
Contributor
Joined
Feb 10, 2016
Messages
252
Reaction score
668
Points
418
Age
35
Location
Ohio
Gender
Male
I'm looking forward to working with your hits. A requester that is putting in the work to get things right out of the gate and give feedback to develop their worker pool is a breath of fresh air. Good luck with this project.
 
  • Like
Reactions: Randomacts

HummingbirdWatch

New Member
Requester
Joined
Aug 5, 2016
Messages
39
Reaction score
28
Points
18
Hello everyone!

We are working hard to have our HITs ready for the week long trial, and I'm happy to have a schedule update for you. The daily HIT schedule will go as follows (Pacific Time):
  1. 6:00 am
  2. 10:00 am
  3. 2:00 pm
  4. 4:30 pm
  5. 5:00 pm
  6. 5:30 pm
  7. 9:00 pm
The first day of the 7 day trial is this Friday (2016-08-12).

Each HIT will consist of 3 assignments, and should take less than a minute to complete.

More information to come! Thanks,

HW
 
  • Like
Reactions: Squatty