Welcome to MTurk Crowd

Work on Amazon Mechanical Turk, learn from the best, and have fun doing it. Join the crowd today!

Sign Up

Test for new Image Review task

Discussion in 'Requesters' started by Cliff DesPeaux, Aug 12, 2018.

  1. Cliff DesPeaux

    Cliff DesPeaux
    Expand Collapse
    New Member

    • Requester
    Joined:
    Jul 28, 2017
    Messages:
    71
    Likes:
    303
    Hi MTurk Crowd,

    I have a new upcoming Image Review task that I am recruiting Turkers for. A few test HITs have launched already with very minimal qual requirements, but I am having trouble getting accuracy vs the task guidelines.

    In an attempt to increase accuracy and help Turkers better understand our needs, I have created a test to help explain the guidelines and to use as a way to find qualified Turkers.

    If you are interested in working these future HITs, please complete the Image Test here.

    I welcome feedback, either in the comment thread here, or via this page.

    Thank you, all!
     
    ducky likes this.
  2. nana

    nana
    Expand Collapse
    Member

    • Contributor
    Joined:
    Jun 13, 2016
    Messages:
    389
    Likes:
    1,412
    Hi just did your test but there is one that I have a question about that is being scored wrong for me. It is Image 50, I marked it as both focused on an individual and nudity. There is only one person in the image but saying that it should not be marked as this answer. Not sure if I am not understanding that correctly or if it is not being scored properly.
     
  3. Cliff DesPeaux

    Cliff DesPeaux
    Expand Collapse
    New Member

    • Requester
    Joined:
    Jul 28, 2017
    Messages:
    71
    Likes:
    303
    Thank you for highlighting that one -- I mistakenly didn't mark it as both. It is now fixed! And, don't worry about re-taking the test.
     
  4. DareAngel3

    DareAngel3
    Expand Collapse
    Well-Known Member

    • Contributor
    Joined:
    Jan 12, 2016
    Messages:
    1,866
    Likes:
    4,706
    If I understand the instructions correctly, as long as you flagged it for something your answer will be counted towards the qualification. Therefore the numerical score isn't accurate/the deciding factor. Is this correct, @Cliff DesPeaux@Cliff DesPeaux?
     
  5. DareAngel3

    DareAngel3
    Expand Collapse
    Well-Known Member

    • Contributor
    Joined:
    Jan 12, 2016
    Messages:
    1,866
    Likes:
    4,706
    I would also argue that 15 has a logo, in addition to it being blurry.
     
  6. DareAngel3

    DareAngel3
    Expand Collapse
    Well-Known Member

    • Contributor
    Joined:
    Jan 12, 2016
    Messages:
    1,866
    Likes:
    4,706
    I was able to revise my answers and get near perfect. :DCI:
     
  7. Leatherass

    Leatherass
    Expand Collapse
    Member

    Joined:
    Jun 21, 2018
    Messages:
    173
    Likes:
    251
    Hi Cliff,thanks for the post.

    If possible any info on the upcoming task would be great to have.

    Such as, pay per task, how many images per task, batches or single HIT's, other qualifications, your expectations, or anything of that nature that may be of interest to us, the workers.

    Thanks again for posting!
     
    Collapse Signature Expand Signature
    Concise, jklmnop, DareAngel3 and 2 others like this.
  8. Concise

    Concise
    Expand Collapse
    Buddhist Bacchante

    Joined:
    Apr 28, 2016
    Messages:
    2,290
    Likes:
    3,721

    Thank you for this post. I chose not to take this qual test, because it had many questions, and was unpaid. In my experience, when requesters ask for MTurk worker's time for free to review anything, and have not shared what they pay, how they have structured their work business design... It's reasonable to pause. I'll wait for updates on TO.
     
    Collapse Signature Expand Signature
    Leatherass likes this.
  9. jmr

    jmr
    Expand Collapse
    Member

    Joined:
    Aug 7, 2018
    Messages:
    142
    Likes:
    168
    Hi! I did one of these a few days ago and to be perfectly honest, I was just grossed out by the bug photos, I had a bad bug experience and just couldn't risk seeing those photos. Otherwise I didn't have any issues with the HIT I completed
     
  10. nana

    nana
    Expand Collapse
    Member

    • Contributor
    Joined:
    Jun 13, 2016
    Messages:
    389
    Likes:
    1,412
    I did this last night, can you tell us when the hits will be up or how to find out if we qualified?
     
  11. Cliff DesPeaux

    Cliff DesPeaux
    Expand Collapse
    New Member

    • Requester
    Joined:
    Jul 28, 2017
    Messages:
    71
    Likes:
    303
    Sorry about that! Yes there are some bugs in there from time to time.

    I am hoping to launch some test HITs with those who make the qual group later today. I will reach out directly to the Turkers when they launch.
     
  12. wascal wabbit

    wascal wabbit
    Expand Collapse
    Drunken Spelunker

    • Contributor
    Joined:
    Dec 22, 2017
    Messages:
    393
    Likes:
    867
    so what is the cutoff score?
     
  13. Cliff DesPeaux

    Cliff DesPeaux
    Expand Collapse
    New Member

    • Requester
    Joined:
    Jul 28, 2017
    Messages:
    71
    Likes:
    303
    TBD! But don't worry too much about it -- this is not a perfect system and I will be looking through the results manually, not relying on the score exclusively.
     
    wascal wabbit likes this.
  14. wascal wabbit

    wascal wabbit
    Expand Collapse
    Drunken Spelunker

    • Contributor
    Joined:
    Dec 22, 2017
    Messages:
    393
    Likes:
    867
    Cool, thanks. I could go back through and get a perfect score, but I got a 74 naturally.