Test for new Image Review task

[Cliff DesPeaux]

Hi MTurk Crowd,

I have a new upcoming Image Review task that I am recruiting Turkers for. A few test HITs have launched already with very minimal qual requirements, but I am having trouble getting accuracy vs the task guidelines.

In an attempt to increase accuracy and help Turkers better understand our needs, I have created a test to help explain the guidelines and to use as a way to find qualified Turkers.

If you are interested in working these future HITs, please complete the Image Test here.

I welcome feedback, either in the comment thread here, or via this page.

Thank you, all!

 

[nana]

Hi just did your test but there is one that I have a question about that is being scored wrong for me. It is Image 50, I marked it as both focused on an individual and nudity. There is only one person in the image but saying that it should not be marked as this answer. Not sure if I am not understanding that correctly or if it is not being scored properly.

 

[Cliff DesPeaux]

Hi just did your test but there is one that I have a question about that is being scored wrong for me. It is Image 50, I marked it as both focused on an individual and nudity. There is only one person in the image but saying that it should not be marked as this answer. Not sure if I am not understanding that correctly or if it is not being scored properly.

Thank you for highlighting that one -- I mistakenly didn't mark it as both. It is now fixed! And, don't worry about re-taking the test.

 

[DareAngel3]

Hi just did your test but there is one that I have a question about that is being scored wrong for me. It is Image 50, I marked it as both focused on an individual and nudity. There is only one person in the image but saying that it should not be marked as this answer. Not sure if I am not understanding that correctly or if it is not being scored properly.

If I understand the instructions correctly, as long as you flagged it for something your answer will be counted towards the qualification. Therefore the numerical score isn't accurate/the deciding factor. Is this correct, C @Cliff DesPeaux?

 

[DareAngel3]

Thank you for highlighting that one -- I mistakenly didn't mark it as both. It is now fixed! And, don't worry about re-taking the test.

I would also argue that 15 has a logo, in addition to it being blurry.

 

[DareAngel3]

I was able to revise my answers and get near perfect.

 

[Leatherass]

Hi Cliff,thanks for the post.

If possible any info on the upcoming task would be great to have.

Such as, pay per task, how many images per task, batches or single HIT's, other qualifications, your expectations, or anything of that nature that may be of interest to us, the workers.

Thanks again for posting!

 

[Concise]

Hi Cliff,thanks for the post.

If possible any info on the upcoming task would be great to have.

Such as, pay per task, how many images per task, batches or single HIT's, other qualifications, your expectations, or anything of that nature that may be of interest to us, the workers.

Thanks again for posting!

Thank you for this post. I chose not to take this qual test, because it had many questions, and was unpaid. In my experience, when requesters ask for MTurk worker's time for free to review anything, and have not shared what they pay, how they have structured their work business design... It's reasonable to pause. I'll wait for updates on TO.

 

[jmr]

Hi! I did one of these a few days ago and to be perfectly honest, I was just grossed out by the bug photos, I had a bad bug experience and just couldn't risk seeing those photos. Otherwise I didn't have any issues with the HIT I completed

 

[nana]

I did this last night, can you tell us when the hits will be up or how to find out if we qualified?

 

[Cliff DesPeaux]

Hi! I did one of these a few days ago and to be perfectly honest, I was just grossed out by the bug photos, I had a bad bug experience and just couldn't risk seeing those photos. Otherwise I didn't have any issues with the HIT I completed

Sorry about that! Yes there are some bugs in there from time to time.

I did this last night, can you tell us when the hits will be up or how to find out if we qualified?

I am hoping to launch some test HITs with those who make the qual group later today. I will reach out directly to the Turkers when they launch.

 

[wascal wabbit]

so what is the cutoff score?

 

[Cliff DesPeaux]

so what is the cutoff score?

TBD! But don't worry too much about it -- this is not a perfect system and I will be looking through the results manually, not relying on the score exclusively.

 

[wascal wabbit]

Cool, thanks. I could go back through and get a perfect score, but I got a 74 naturally.