Test for new Image Review task

Cliff DesPeaux

New Member
Requester
Joined
Jul 28, 2017
Messages
71
Reaction score
303
Points
303
Gender
Male
Hi MTurk Crowd,

I have a new upcoming Image Review task that I am recruiting Turkers for. A few test HITs have launched already with very minimal qual requirements, but I am having trouble getting accuracy vs the task guidelines.

In an attempt to increase accuracy and help Turkers better understand our needs, I have created a test to help explain the guidelines and to use as a way to find qualified Turkers.

If you are interested in working these future HITs, please complete the Image Test here.

I welcome feedback, either in the comment thread here, or via this page.

Thank you, all!
 
  • Like
Reactions: ducky

nana

Active Member
Contributor
Joined
Jun 13, 2016
Messages
652
Reaction score
2,054
Points
568
Gender
Female
Hi just did your test but there is one that I have a question about that is being scored wrong for me. It is Image 50, I marked it as both focused on an individual and nudity. There is only one person in the image but saying that it should not be marked as this answer. Not sure if I am not understanding that correctly or if it is not being scored properly.
 

Cliff DesPeaux

New Member
Requester
Joined
Jul 28, 2017
Messages
71
Reaction score
303
Points
303
Gender
Male
Hi just did your test but there is one that I have a question about that is being scored wrong for me. It is Image 50, I marked it as both focused on an individual and nudity. There is only one person in the image but saying that it should not be marked as this answer. Not sure if I am not understanding that correctly or if it is not being scored properly.
Thank you for highlighting that one -- I mistakenly didn't mark it as both. It is now fixed! And, don't worry about re-taking the test.
 

DareAngel3

Well-Known Member
Contributor
Mentor
Crowd Pleaser
Joined
Jan 12, 2016
Messages
5,553
Reaction score
11,231
Points
2,713
Age
36
Location
Central IL
Gender
Female
Hi just did your test but there is one that I have a question about that is being scored wrong for me. It is Image 50, I marked it as both focused on an individual and nudity. There is only one person in the image but saying that it should not be marked as this answer. Not sure if I am not understanding that correctly or if it is not being scored properly.
If I understand the instructions correctly, as long as you flagged it for something your answer will be counted towards the qualification. Therefore the numerical score isn't accurate/the deciding factor. Is this correct, C @Cliff DesPeaux?
 

DareAngel3

Well-Known Member
Contributor
Mentor
Crowd Pleaser
Joined
Jan 12, 2016
Messages
5,553
Reaction score
11,231
Points
2,713
Age
36
Location
Central IL
Gender
Female
Thank you for highlighting that one -- I mistakenly didn't mark it as both. It is now fixed! And, don't worry about re-taking the test.
I would also argue that 15 has a logo, in addition to it being blurry.
 

DareAngel3

Well-Known Member
Contributor
Mentor
Crowd Pleaser
Joined
Jan 12, 2016
Messages
5,553
Reaction score
11,231
Points
2,713
Age
36
Location
Central IL
Gender
Female
I was able to revise my answers and get near perfect. :DCI:
 

Leatherass

Member
Joined
Jun 21, 2018
Messages
326
Reaction score
491
Points
263
Gender
Male
Hi Cliff,thanks for the post.

If possible any info on the upcoming task would be great to have.

Such as, pay per task, how many images per task, batches or single HIT's, other qualifications, your expectations, or anything of that nature that may be of interest to us, the workers.

Thanks again for posting!
 

Concise

Buddhist Bacchante
Joined
Apr 28, 2016
Messages
4,092
Reaction score
6,398
Points
813
Hi Cliff,thanks for the post.

If possible any info on the upcoming task would be great to have.

Such as, pay per task, how many images per task, batches or single HIT's, other qualifications, your expectations, or anything of that nature that may be of interest to us, the workers.

Thanks again for posting!

Thank you for this post. I chose not to take this qual test, because it had many questions, and was unpaid. In my experience, when requesters ask for MTurk worker's time for free to review anything, and have not shared what they pay, how they have structured their work business design... It's reasonable to pause. I'll wait for updates on TO.
 
  • Like
Reactions: Leatherass

jmr

Member
Joined
Aug 7, 2018
Messages
154
Reaction score
195
Points
243
Location
nyc
Gender
Female
Hi! I did one of these a few days ago and to be perfectly honest, I was just grossed out by the bug photos, I had a bad bug experience and just couldn't risk seeing those photos. Otherwise I didn't have any issues with the HIT I completed
 

nana

Active Member
Contributor
Joined
Jun 13, 2016
Messages
652
Reaction score
2,054
Points
568
Gender
Female
I did this last night, can you tell us when the hits will be up or how to find out if we qualified?
 

Cliff DesPeaux

New Member
Requester
Joined
Jul 28, 2017
Messages
71
Reaction score
303
Points
303
Gender
Male
Hi! I did one of these a few days ago and to be perfectly honest, I was just grossed out by the bug photos, I had a bad bug experience and just couldn't risk seeing those photos. Otherwise I didn't have any issues with the HIT I completed
Sorry about that! Yes there are some bugs in there from time to time.

I did this last night, can you tell us when the hits will be up or how to find out if we qualified?
I am hoping to launch some test HITs with those who make the qual group later today. I will reach out directly to the Turkers when they launch.
 

wascal wabbit

Drunken Spelunker
Contributor
Joined
Dec 22, 2017
Messages
393
Reaction score
869
Points
368
Location
Earf
Gender
Female
Cool, thanks. I could go back through and get a perfect score, but I got a 74 naturally.