Welcome to MTurk Crowd

Work on Amazon Mechanical Turk, learn from the best, and have fun doing it. Join the crowd today!

Sign Up

Rep Present Hummingbird Watch

Discussion in 'Requesters' started by HummingbirdWatch, Aug 8, 2016.

  1. HummingbirdWatch

    HummingbirdWatch
    Expand Collapse
    New Member

    • Requester
    Joined:
    Aug 5, 2016
    Messages:
    39
    Likes:
    28
    Hello MTurk Crowd community,

    Our company is interested in trialing with MTurk. We are hoping to have a trial HIT released later this week.

    Here's how it works: We would upload images collected by remote cameras, and the workers would look at the images and click yes/no if the image contained a specific item (similar to validating search result images). Speed and accuracy are paramount. These images would have to be processed immediately after being uploaded, and accurately.

    We understand that if we want our images to be processed quickly and accurately, we need to maintain a reputation of honesty and fairness with the worker community.

    Any advice on making our time with MTurk successful would be appreciated!

    Thanks,

    HW
     
    #1 HummingbirdWatch, Aug 8, 2016
    Last edited: Aug 8, 2016
  2. MerryLou

    MerryLou
    Expand Collapse
    unpalatable

    • Contributor
    • Crowd Pleaser
    • Sole Survivor
    Joined:
    Jan 12, 2016
    Messages:
    11,251
    Likes:
    20,480
    :wave: Making an account here and seeking out info from 'the other side' is a great way to start!

    Setting up a qualification test or a small batch of HITs as a kind of preliminary round for workers to earn a qualification from you is a good way to make sure you'll get consistently accurate results from workers.

    Looking forward to trying out your HITs!
     
    Collapse Signature Expand Signature
    HummingbirdWatch likes this.
  3. HummingbirdWatch

    HummingbirdWatch
    Expand Collapse
    New Member

    • Requester
    Joined:
    Aug 5, 2016
    Messages:
    39
    Likes:
    28
    Thanks for your reply MerryLou. We are currently putting together a few preliminary HITs to replicate those we would find in our actual project. Is there a means to 'soft' reject worker responses, where their reputations wouldn't be affected and we could provide feedback?

    Realistically, we'll likely have to tweak our HIT delivery (for example, clearer instructions..), and we don't want good quality workers to pay the price for our 'growing pains'.
     
  4. TSolo315

    TSolo315
    Expand Collapse
    SnapNCrackle

    • Moderator
    Joined:
    Jan 12, 2016
    Messages:
    2,648
    Likes:
    8,917
    No, unfortunately there is no way to reject a HIT without affecting approval rate. It would be a nice feature to have -- a way for requesters to send a HIT back to a worker so they could fix any mistakes.
     
    Collapse Signature Expand Signature
  5. MerryLou

    MerryLou
    Expand Collapse
    unpalatable

    • Contributor
    • Crowd Pleaser
    • Sole Survivor
    Joined:
    Jan 12, 2016
    Messages:
    11,251
    Likes:
    20,480
    As far as I know, you can also leave a feedback messages on an approved HIT.

    If it's feasible, you could post some kind of an example here and them I'm sure you could at least get feedback on the instructions before testing on mturk.
     
    Collapse Signature Expand Signature
  6. electrolyte

    electrolyte
    Expand Collapse
    The Ghost of MTurk Past

    • Contributor
    Joined:
    Jan 10, 2016
    Messages:
    19,184
    Likes:
    45,924
    You may also want to try the Requester Sandbox to post some practice HITs before you post live HITs.
     
    Collapse Signature Expand Signature
    HummingbirdWatch and MerryLou like this.
  7. HummingbirdWatch

    HummingbirdWatch
    Expand Collapse
    New Member

    • Requester
    Joined:
    Aug 5, 2016
    Messages:
    39
    Likes:
    28
    Thanks for your reply TSolo315. Forgive me if the answer is in another post, but is there perhaps a way to accept the submission that is incorrect, but change the reward to be 0 or 0.01 and provide feedback for the worker? This may sound dangerously close to accepting work without pay, but I'm trying to think of a work around to hurting reputations that don't need to be hurt.
     
  8. HummingbirdWatch

    HummingbirdWatch
    Expand Collapse
    New Member

    • Requester
    Joined:
    Aug 5, 2016
    Messages:
    39
    Likes:
    28

    Thanks MerryLou for your response. We are hoping to have our first test HIT ready to go by Thursday or Friday, I'll post the HIT instructions on the forum earlier so we can see if there's room for improvement.
     
    clickhappier likes this.
  9. HummingbirdWatch

    HummingbirdWatch
    Expand Collapse
    New Member

    • Requester
    Joined:
    Aug 5, 2016
    Messages:
    39
    Likes:
    28
    Thanks electrolyte for your reply. We have looked into the Requester Sandbox, and will definitely be testing our HITs that way. We predict that our staff will be prone to bias and miss things that mturk workers will see pretty quick.
     
  10. TSolo315

    TSolo315
    Expand Collapse
    SnapNCrackle

    • Moderator
    Joined:
    Jan 12, 2016
    Messages:
    2,648
    Likes:
    8,917
    Well, you can do it backwards. You can set up your trial hits with .01 as the pay and then send the full amount as a bonus to people who answer correctly. You will have to make this clear to workers when you set up the HIT.

    Feedback can be given along with bonuses (people receive emails when they get a bonus.) You can send penny bonuses to people who don't answer correctly with feedback as well.
     
    Collapse Signature Expand Signature
    #10 TSolo315, Aug 9, 2016
    Last edited: Aug 9, 2016
  11. clickhappier

    clickhappier
    Expand Collapse
    ┬──┬ ノ( ゜-゜ノ)

    • Subforum Curator
    • Crowd Pleaser
    Joined:
    Jan 12, 2016
    Messages:
    727
    Likes:
    1,607
    To clarify, your staff would be the ones to do the testing of your sandbox HITs, as workers would receive no real earnings in the sandbox.
     
  12. HummingbirdWatch

    HummingbirdWatch
    Expand Collapse
    New Member

    • Requester
    Joined:
    Aug 5, 2016
    Messages:
    39
    Likes:
    28
    Apologies, I wasn't very clear with my response. What I meant to say was that we would like to do the initial testing in the Requester Sandbox with our own staff. Once we move on from the sandbox into the trial week, we predict more issues to come to light once our staff are no longer the ones doing the HITs. When this becomes the case, it would be beneficial if there were a mechanism for feedback that wouldn't hurt the worker's reputation (or ours). Thanks for the clarification.
     
    clickhappier likes this.
  13. HummingbirdWatch

    HummingbirdWatch
    Expand Collapse
    New Member

    • Requester
    Joined:
    Aug 5, 2016
    Messages:
    39
    Likes:
    28
    This method sounds like it could work. Thanks for your input. I've read that dealing with bonuses can be a fairly labor intensive approach for the requester, but it may be the only way to provide feedback during our trial week (without rejecting submissions).
     
  14. Metallica

    Metallica
    Expand Collapse
    No remorse, no repent

    • Contributor
    • Mentor
    Joined:
    Jan 17, 2016
    Messages:
    3,964
    Likes:
    9,122
    I'm not too sure if he's been around lately but you could try to give @GlobalWorkForge@GlobalWorkForge a shout. He's pretty experienced with the requester side and has a hand in, or runs multiple different requester accounts.
     
    HummingbirdWatch likes this.
  15. Randomacts

    Randomacts
    Expand Collapse
    Banned
    Banned

    Joined:
    Jul 25, 2016
    Messages:
    1,877
    Likes:
    1,515
    It can be automated, I'm just not sure how exactly.

    You should try and contact the Sticky team https://turkopticon.ucsd.edu/reports?id=A1I5WZRIXY5GS9 they have the whole thing automated although I doubt you could fully automate your process like they do.
     
    #15 Randomacts, Aug 10, 2016
    Last edited: Aug 10, 2016
    HummingbirdWatch likes this.
  16. HummingbirdWatch

    HummingbirdWatch
    Expand Collapse
    New Member

    • Requester
    Joined:
    Aug 5, 2016
    Messages:
    39
    Likes:
    28
    I'll drop @GlobalWorkForge@GlobalWorkForge a line and see if he can offer a new requester some advice! Thanks Janthir.
     
  17. HummingbirdWatch

    HummingbirdWatch
    Expand Collapse
    New Member

    • Requester
    Joined:
    Aug 5, 2016
    Messages:
    39
    Likes:
    28
    Thanks for the link, I'll have a look at it and see if we can apply it to our process. Automating our workflow will be a priority once we establish something that works in the meantime.
     
  18. Metallica

    Metallica
    Expand Collapse
    No remorse, no repent

    • Contributor
    • Mentor
    Joined:
    Jan 17, 2016
    Messages:
    3,964
    Likes:
    9,122
  19. Zacharias

    Zacharias
    Expand Collapse
    Experiencing Flim Flam Withdrawal

    • Contributor
    Joined:
    Feb 10, 2016
    Messages:
    245
    Likes:
    651
    I'm looking forward to working with your hits. A requester that is putting in the work to get things right out of the gate and give feedback to develop their worker pool is a breath of fresh air. Good luck with this project.
     
    Randomacts likes this.
  20. HummingbirdWatch

    HummingbirdWatch
    Expand Collapse
    New Member

    • Requester
    Joined:
    Aug 5, 2016
    Messages:
    39
    Likes:
    28
    Hello everyone!

    We are working hard to have our HITs ready for the week long trial, and I'm happy to have a schedule update for you. The daily HIT schedule will go as follows (Pacific Time):
    1. 6:00 am
    2. 10:00 am
    3. 2:00 pm
    4. 4:30 pm
    5. 5:00 pm
    6. 5:30 pm
    7. 9:00 pm
    The first day of the 7 day trial is this Friday (2016-08-12).

    Each HIT will consist of 3 assignments, and should take less than a minute to complete.

    More information to come! Thanks,

    HW
     
    Squatty likes this.