Rep Present SelfConcept $60 10 Hour HIT Discussion

weisian

New Member
Requester
Joined
Jul 22, 2016
Messages
71
Reaction score
205
Points
33
Age
33
I wanted to start a thread where people could ask questions about this HIT (or complain, or talk about how awesome this HIT is, whatever). It also goes by "Behavioral Study of Cognitive Function". I'm a researcher involved in the project.

Below I've put some comments I've made before in regard to previous comments in the Daily HIT threads as a reference. This will hopefully make it easier for new people thinking about this HIT to find information, especially as I roll out new HITs in the future.

Check-In

We have implemented a check-in system where you must complete a separate HIT (linked from the $60 HIT) to proceed. This separate HIT takes no time, and pays nothing. Its only point is to make sure we can track you so that we can communicate with you if needed and pay you at the end should you fail to complete the entire battery.

Communication

This HIT has had some technical issues in the original roll out. While I have dealt with most of them (thanks in part to communication from this site and the turkers working), but others may present themselves. I will always update the Ad at the beginning of the HIT with updates so please read that if you are participating.

Time Estimate

This HIT does actually take 10 hours. This seems to be the main dissapointment among turkers, where people think I am only saying it takes 10 hours, but you can actually do it in 6 hours (or something). This is not the case. It will take around 10 hours.

How do we know that? The 10 hour estimate was originally based on our pilot work, but we have changed the list of tasks a bit so things could change. When last I looked, the people who had run through the tasks so far (20 or so people having gotten to each task), if I average the amount of time taken on each task and sum them I get 5.6 hours. This doesn't include surveys (there are ~20 surveys). I don't have info on how long each survey takes, but if they each take an average of 10 minutes, we'd still only be at 9 hours, and I doubt people spend 10 minutes on average across all surveys (some surveys have 1 question, some have 40).

The time estimate I gave above does not include instruction time for each task. I'd imagine that each task (not surveys) should take at least 2 minutes of instruction time. If that's true, and there are 35 tasks I believe, that's around another hour and ten minutes. So let's guess 7 hours plus survey time.

It is difficult to perfectly determine how long such a long HIT takes, but we believe we've been slightly conservative with our 10 hour estimate. There will be a bit of wiggle room, but most tasks take a set amount of time regardless of how you take them.

Payment and Submission (and bonuses)

I saw a post about finishing the 63 tasks and the HIT suddenly moving to a blank screen and removed from your queue. We didn't handle the submission process the best, so I understand that this can be a bit confusing. Regardless, what's going on is that the HIT is submitted, and most of the time is automatically verified and credited. Once in a while our code screws up (there are some timing issues coordinating with Mturk's site) and I have to go in afterwards to manually approve. Either way, don't worry! Your work is almost certainly submitted.

And I'll just mention this here - the HIT mentions bonuses. The bonuses will take a little bit for us to get to, but we will get to them! I know immediate payment is preferable, but this part takes a little more work. Bonuses normally range from $5-$15, depending on performance. I'll code up the "bonus calculator" and get them out as soon as possible - hopefully no later than a week after the end date for this HIT (next Thursday), but hopefully much earlier!


Will we have more HITs?

Yes (as of 7/26). We will have 4 more rounds of HITs.


Finally, I'd like to say how appreciate I've been of this community (and the broader community like people who post on turkopticon). It is refreshing to see people care about their work and care about their community such that they keep each other informed.
 
Last edited:

coffeeowl

hoo?
Contributor
Mentor
Master Pleaser
Crowd Pleaser
HIT Poster
Joined
Jan 12, 2016
Messages
28,032
Reaction score
63,932
Points
1,738
Location
Location EqualTo US
Gender
Female
Thanks for starting this thread and responding to everyone's questions! I have not been able to catch your HIT yet but hope I can soon.
 
  • Like
Reactions: T. Leela

Hotlamp

Thats Not Me
Joined
May 13, 2016
Messages
2,640
Reaction score
3,637
Points
513
Age
32
Gender
Male
I wanted to start a thread where people could ask questions about this HIT (or complain, or talk about how awesome this HIT is, whatever). It also goes by "Behavioral Study of Cognitive Function". I'm a researcher involved in the project.

Below I've put some comments I've made before in regard to previous comments in the Daily HIT threads as a reference. This will hopefully make it easier for new people thinking about this HIT to find information, especially as I roll out new HITs in the future.

Communication

This HIT has had some technical issues in the original roll out. While I have dealt with most of them (thanks in part to communication from this site and the turkers working), but others may present themselves. I will always update the Ad at the beginning of the HIT with updates so please read that if you are participating.

Time Estimate

This HIT does actually take 10 hours. This seems to be the main dissapointment among turkers, where people think I am only saying it takes 10 hours, but you can actually do it in 6 hours (or something). This is not the case. It will take around 10 hours.

How do we know that? The 10 hour estimate was originally based on our pilot work, but we have changed the list of tasks a bit so things could change. When last I looked, the people who had run through the tasks so far (20 or so people having gotten to each task), if I average the amount of time taken on each task and sum them I get 5.6 hours. This doesn't include surveys (there are ~20 surveys). I don't have info on how long each survey takes, but if they each take an average of 10 minutes, we'd still only be at 9 hours, and I doubt people spend 10 minutes on average across all surveys (some surveys have 1 question, some have 40).

The time estimate I gave above does not include instruction time for each task. I'd imagine that each task (not surveys) should take at least 2 minutes of instruction time. If that's true, and there are 35 tasks I believe, that's around another hour and ten minutes. So let's guess 7 hours plus survey time.

It is difficult to perfectly determine how long such a long HIT takes, but we believe we've been slightly conservative with our 10 hour estimate. There will be a bit of wiggle room, but most tasks take a set amount of time regardless of how you take them.

Payment and Submission (and bonuses)

I saw a post about finishing the 63 tasks and the HIT suddenly moving to a blank screen and removed from your queue. We didn't handle the submission process the best, so I understand that this can be a bit confusing. Regardless, what's going on is that the HIT is submitted, and most of the time is automatically verified and credited. Once in a while our code screws up (there are some timing issues coordinating with Mturk's site) and I have to go in afterwards to manually approve. Either way, don't worry! Your work is almost certainly submitted.

And I'll just mention this here - the HIT mentions bonuses. The bonuses will take a little bit for us to get to, but we will get to them! I know immediate payment is preferable, but this part takes a little more work. Bonuses normally range from $5-$15, depending on performance. I'll code up the "bonus calculator" and get them out as soon as possible - hopefully no later than a week after the end date for this HIT (next Thursday), but hopefully much earlier!


Will we have more HITs?

Yes (as of 7/26). We will have 4 more rounds of HITs.


Finally, I'd like to say how appreciate I've been of this community (and the broader community like people who post on turkopticon). It is refreshing to see people care about their work and care about their community such that they keep each other informed.
Thanks for posting here. Look forward to the hit.
 

ellie

not to disappear
Joined
Jul 22, 2016
Messages
257
Reaction score
736
Points
293
Age
32
Location
Oregon
Gender
Female
Thanks for being so informative, really looking forward to the hit!
 

Briamari

New Member
Joined
Jul 27, 2016
Messages
2
Reaction score
0
Points
1
Age
44
Gender
Female
Thanks for posting this! I just completed this hit today after picking it up last week. I did have the blank screen at submit and it still hasn't shown up in my completed. Very interesting hit by the way!
 

weisian

New Member
Requester
Joined
Jul 22, 2016
Messages
71
Reaction score
205
Points
33
Age
33
Thanks for posting this! I just completed this hit today after picking it up last week. I did have the blank screen at submit and it still hasn't shown up in my completed. Very interesting hit by the way!
It should say "0 out of 63" HITs left, and then have a "Finish" button rather than "Next Experiment". After that it just takes you to a blank screen I guess and submits. Like I said, we haven't gotten this part down the best. Others have mentioned that it may show up in your completed HIT for the day you accepted the HIT. I'm not really familiar with mturk from the worker side, so maybe someone else can chime in.
 

Briamari

New Member
Joined
Jul 27, 2016
Messages
2
Reaction score
0
Points
1
Age
44
Gender
Female
It should say "0 out of 63" HITs left, and then have a "Finish" button rather than "Next Experiment". After that it just takes you to a blank screen I guess and submits. Like I said, we haven't gotten this part down the best. Others have mentioned that it may show up in your completed HIT for the day you accepted the HIT. I'm not really familiar with mturk from the worker side, so maybe someone else can chime in.
It should say "0 out of 63" HITs left, and then have a "Finish" button rather than "Next Experiment". After that it just takes you to a blank screen I guess and submits. Like I said, we haven't gotten this part down the best. Others have mentioned that it may show up in your completed HIT for the day you accepted the HIT. I'm not really familiar with mturk from the worker side, so maybe someone else can chime in.
You are correct I found it, thank you for your help! I received my bonuses this morning!
 

NovaLord

Stargazer
Contributor
Joined
Jun 14, 2016
Messages
527
Reaction score
1,240
Points
418
Age
31
Gender
Male
Is there a certain time when these are posted so I can make sure I grab one?
 

weisian

New Member
Requester
Joined
Jul 22, 2016
Messages
71
Reaction score
205
Points
33
Age
33
Is there a certain time when these are posted so I can make sure I grab one?
We will be sending these out at different times. Though turkers here have ways to get first dibs on these HITs we currently don't want to subsample the mechanical turk population to the mturkcrowd population. That implicitly happens (due to this community's greater attentiveness and pandaing (that's a term I'm just learning, not sure if I'm using it correctly), already. Automated preview and accept is actually quite annoying (it subverts the normal process), but is a reality of mechanical turk now so what can we do? On the positive side, though we may over-represent mturkcrowd and other more-engaged turkers, we probably get better data from you guys as well.

That was a long way to say I'm not currently planning on telling you when I submit new HITs. Sorry :)
 
  • Like
Reactions: jan

weisian

New Member
Requester
Joined
Jul 22, 2016
Messages
71
Reaction score
205
Points
33
Age
33
The first round of the HIT should be expired now for all workers. Let me outline here next steps if anyone is confused.

I will assign qualifications to everyone who needs a dummy HIT and then create these HITs. If I have your email I will contact you to alert you of this fact, but if not look for a SelfConcept HIT called "Dummy HIT". It will have a $0 base pay and I will pay out your payment through the bonus system. I plan to get these out tonight.

If you failed to complete the HIT for any reason besides technical issues you will be paid according to the reduced payment outlined in the instructions. If you have had issues and had contacted me about them then you will be paid a prorated payment.

We had originally hoped we could contact workers who failed to complete the HIT, but have found that that is impossible. This makes it difficult to follow up, but we would like to pay everyone who did work for us. If you have emailed me I can contact you, so no problem there. If you haven't emailed me either (1) just look out for the "Dummy HIT" or (2) email me so that I can contact you in the future. Every incomplete worker will have the qualifications assigned that are necessary to access the Dummy HIT.

Edit: For future HITS we have implemented a check-in system (just a $0 HIT) so that we can find and contact you after the HIT regardless of whether you finish it or not. With that emailing just to open communication with us is unnecessary. But do email us about any problems you may have!

Please post any questions below.
 
Last edited:
  • Like
Reactions: kryss

weisian

New Member
Requester
Joined
Jul 22, 2016
Messages
71
Reaction score
205
Points
33
Age
33
I have just sent out the next round of assignments. Please note, if you completed this HIT before you cannot complete it again.
 
Last edited:

NovaLord

Stargazer
Contributor
Joined
Jun 14, 2016
Messages
527
Reaction score
1,240
Points
418
Age
31
Gender
Male
I have just sent out the next round of assignments. Please note, if you completed this HIT before you cannot complete it again.
So if I did a check in HIT this morning, but wasn't quick enough to grab the main HIT you sent out an hour ago, do I have to do a second check in before the next round happens?
 

weisian

New Member
Requester
Joined
Jul 22, 2016
Messages
71
Reaction score
205
Points
33
Age
33
So if I did a check in HIT this morning, but wasn't quick enough to grab the main HIT you sent out an hour ago, do I have to do a second check in before the next round happens?
No, if you ever get the main HIT you will be good to go. For others - please do not complete this "check-in" HIT unless directed to from another HIT! Don't just accept it randomly...as it says in the Ad.
 

corsies

Member
Joined
May 30, 2016
Messages
284
Reaction score
350
Points
313
Age
58
Gender
Female
I am completing the tasks today and it says my task # is stuck at 33 even though I am completing them the task # is not decreasing. is there a way to fix this? I am worried that it will not submit at the end. How can I get this fixed before continuing on in the tasks? Do I have to clear my cookies? not sure what to do at this point? thanks
 

weisian

New Member
Requester
Joined
Jul 22, 2016
Messages
71
Reaction score
205
Points
33
Age
33
I am completing the tasks today and it says my task # is stuck at 33 even though I am completing them the task # is not decreasing. is there a way to fix this? I am worried that it will not submit at the end. How can I get this fixed before continuing on in the tasks? Do I have to clear my cookies? not sure what to do at this point? thanks
Hi Corsies,

I responded to you by email, but I'll respond here in case other people have this question. Every time you complete a survey it should automatically take you to the next survey/task. Every time you complete a task it should take you to a screen that has two buttons: "Next Experiment" and "Redo Experiment". If you hit "Redo experiment" (which you should only do if you think your attention or performance was compromised for some external reason - like taking an urgent phone call or something), the task will be placed back into the queue and come back at some point (randomly).

If you press "Next Experiment" you assert that you have finished the experiment and you move on to the next task. Your data for that last task should be saved now and the next time you see your experiment counter it should decrease by one. If "Next Experiment" isn't working (which a minority of people have reported for a minority of tasks, seemingly randomly), contact us. I'm not sure if there is anything we can do about it, but we can try. You should not be completing multiple tasks within being credited for them in your count.
 

corsies

Member
Joined
May 30, 2016
Messages
284
Reaction score
350
Points
313
Age
58
Gender
Female
Hi Corsies,

I responded to you by email, but I'll respond here in case other people have this question. Every time you complete a survey it should automatically take you to the next survey/task. Every time you complete a task it should take you to a screen that has two buttons: "Next Experiment" and "Redo Experiment". If you hit "Redo experiment" (which you should only do if you think your attention or performance was compromised for some external reason - like taking an urgent phone call or something), the task will be placed back into the queue and come back at some point (randomly).

If you press "Next Experiment" you assert that you have finished the experiment and you move on to the next task. Your data for that last task should be saved now and the next time you see your experiment counter it should decrease by one. If "Next Experiment" isn't working (which a minority of people have reported for a minority of tasks, seemingly randomly), contact us. I'm not sure if there is anything we can do about it, but we can try. You should not be completing multiple tasks within being credited for them in your count.
thanks for your response. I know we are working together on this issue. I also wanted you to know that sometimes when I complete a task the counter drops not once but three times so yesterday when I did a task it dropped from 32 to 29 which I thought was odd. I am not sure if a survey combined with a task is counted as 2 tasks? I know we will get this resolved thanks for your help. Corrie
 

coffeeowl

hoo?
Contributor
Mentor
Master Pleaser
Crowd Pleaser
HIT Poster
Joined
Jan 12, 2016
Messages
28,032
Reaction score
63,932
Points
1,738
Location
Location EqualTo US
Gender
Female
No problems with mine. Thank you for the fast approval! :thumbup:
 

weisian

New Member
Requester
Joined
Jul 22, 2016
Messages
71
Reaction score
205
Points
33
Age
33
thanks for your response. I know we are working together on this issue. I also wanted you to know that sometimes when I complete a task the counter drops not once but three times so yesterday when I did a task it dropped from 32 to 29 which I thought was odd. I am not sure if a survey combined with a task is counted as 2 tasks? I know we will get this resolved thanks for your help. Corrie
The 63 tasks include "tasks" and "surveys". When you complete a survey, you don't get shown the "you have X out of 63 tasks left" screen - just an oversight on our part. So if you complete two surveys and then one task, the next time you see the "you have X ouf of 63 tasks left" it should be 3 less than the last time you saw it.
 
  • Like
Reactions: Azazael

T. Leela

Oh Lord...
Contributor
Joined
May 12, 2016
Messages
11,050
Reaction score
14,046
Points
1,388
Location
Colorado
Gender
Female
Sorry to bother you abt this but somehow I ended up returning the survey along with some other things, when i was sure I was being very careful not to do so. I'd tried to check in once, but it gave me some error message so I'm not sure what went through. I hadn't really worked on much of it yet, but wondered if there is anything I can do. thanks
 

weisian

New Member
Requester
Joined
Jul 22, 2016
Messages
71
Reaction score
205
Points
33
Age
33
Sorry to bother you abt this but somehow I ended up returning the survey along with some other things, when i was sure I was being very careful not to do so. I'd tried to check in once, but it gave me some error message so I'm not sure what went through. I hadn't really worked on much of it yet, but wondered if there is anything I can do. thanks
Hey, can you email me (or message me here) your worker ID and your issue and I'll work on getting it sorted out.