Rep Present Surgery Videos Summarization

LOS

New Member
Requester
Joined
Mar 12, 2016
Messages
72
Reaction score
90
Points
18
Hello folks, by now some of you might have seen these HITs on the MTurk page.
https://www.mturk.com/mturk/preview?groupId=3A5N6ETZEU1R273R6CW4BZ5MOYQJFL

We are researchers from the Computer Science department at the Johns Hopkins University.
We wanted to join in here and make sure I was available for people to send me comments, suggestions, etc.
It is my first time posting HITs on MTurk. Also, this is a pretty neat project that we are working on. We would like to see how we can improve this.
We have already received some feedback from folks who have tried the HIT and are actively trying to incorporate them. We realize that this isn't possible without the Turker community and would like to try our best in our capacity to have a smooth and nice experience for all.
 

electrolyte

The Ghost of MTurk Past
Contributor
Joined
Jan 10, 2016
Messages
19,184
Reaction score
45,965
Points
1,313
Welcome!

Title: Qualification - Summarize Surgery RALH (WARNING: HIT contains anatomy/organ images and blood. Worker discretion is advised.) | PANDA
Requester: LOS [A2MG9E64Z06RY7] (TO)
TO Ratings:

☢☢☢☢☢ 5.00 Communicativity
☢☢☢☢☢ 1.00 Generosity
☢☢☢☢☢ 5.00 Fairness
☢☢☢☢☢ 5.00 Promptness
Number of Reviews: 2
(Submit a new TO rating for this requester)
Description: This HIT should be completed with success to grant you the qualification needed to complete other HITs hosted by us. Please review the information and take a short test. The survey contains graphical surgical images with blood at times.
Time: 45 minutes
Hits Available: 1
Reward: $0.50
Qualifications: Total approved HITs is not less than 100, BlockRobHystSummarizer has not been granted, RobHystSummarizer has not been granted, HIT approval rate (%) is not less than 95, Adult Content Qualification is 1

Title: Summarize Surgical Videos, Duration: 30 minutes (WARNING: HIT contains organ images and blood. Worker discretion is advised.) | PANDA
Requester: LOS [A2MG9E64Z06RY7] (TO)
TO Ratings:

☢☢☢☢☢ 5.00 Communicativity
☢☢☢☢☢ 1.00 Generosity
☢☢☢☢☢ 5.00 Fairness
☢☢☢☢☢ 5.00 Promptness
Number of Reviews: 2
(Submit a new TO rating for this requester)

Description: The survey consists of observing portions of videos of surgical procedures (with graphical surgical images), and answering multiple choice questions that provide a description of the video. Each survey contains a set of 10 videos with seven multiple choice questisons per video.
Time: 1 hour 30 minutes
Hits Available: 4
Reward: $2.00
Qualifications: RobHystSummarizer has been granted, Total approved HITs is not less than 100, BlockRobHystSummarizer has not been granted, HIT approval rate (%) is not less than 95, Adult Content Qualification is 1
 

LOS

New Member
Requester
Joined
Mar 12, 2016
Messages
72
Reaction score
90
Points
18
We have published updated HITs taking into account different recommendations from Turkers who participated in the initial few HITs.
We are new at this and from academia, so please do bear with us, since we are constrained in terms of support for this. We will try our best to address any issues that could/might affect your HIT submission.

https://www.mturk.com/mturk/preview?groupId=3Z7ICGQCYAUFMNCPH9TY0Z19E1ATOO
(Please use the preview link since there are repeated HITs and you would end up returning a HIT if you have already performed it previously)

UPDATE: per HIT payment has been increased to $4.5

We have also increased the total time allotted since we realized Turkers would not be solely working on this HIT or they could get disrupted for whatever other reasons.
 
  • Like
Reactions: clickhappier

clickhappier

┬──┬ ノ( ゜-゜ノ)
Subforum Curator
Crowd Pleaser
Joined
Jan 12, 2016
Messages
728
Reaction score
1,629
Points
593
Location
USA
LOS @LOS would it be possible for you to describe more of what the HITs are like, to help people get a better idea of how interested they'd be in pursuing the apparently-takes-20min-for-$0.50 qual HIT beforehand? :) Examples of what the questions are and such.
 
  • Like
Reactions: LOS

LOS

New Member
Requester
Joined
Mar 12, 2016
Messages
72
Reaction score
90
Points
18
LOS @LOS would it be possible for you to describe more of what the HITs are like, to help people get a better idea of how interested they'd be in pursuing the apparently-takes-20min-for-$0.50 qual HIT beforehand? :) Examples of what the questions are and such.
Are you suggesting including some text about this on the HIT page, or posting something here?
 

electrolyte

The Ghost of MTurk Past
Contributor
Joined
Jan 10, 2016
Messages
19,184
Reaction score
45,965
Points
1,313
Export for the new $4.50 version:

Title: Summarize Surgical Videos, Duration: 45 minutes (WARNING: HIT contains organ images and blood. Worker discretion is advised.) | PANDA
Requester: LOS [A2MG9E64Z06RY7] (TO)
TO Ratings:

☢☢☢☢☢ 5.00 Communicativity
☢☢☢☢☢ 1.00 Generosity
☢☢☢☢☢ 5.00 Fairness
☢☢☢☢☢ 5.00 Promptness
Number of Reviews: 2
(Submit a new TO rating for this requester)

Description: The survey consists of observing portions of videos of surgical procedures (with graphical surgical images), and answering multiple choice questions that provide a description of the video. Each survey contains a set of 10 videos with seven multiple choice questisons per video.
Time: 2 hours 30 minutes
Hits Available: 4
Reward: $4.50
Qualifications: RobHystSummarizer has been granted, Total approved HITs is not less than 100, BlockRobHystSummarizer has not been granted, HIT approval rate (%) is not less than 95, Adult Content Qualification is 1
 
  • Like
Reactions: clickhappier

LOS

New Member
Requester
Joined
Mar 12, 2016
Messages
72
Reaction score
90
Points
18
LOS @LOS would it be possible for you to describe more of what the HITs are like, to help people get a better idea of how interested they'd be in pursuing the apparently-takes-20min-for-$0.50 qual HIT beforehand? :) Examples of what the questions are and such.
I have included the following on the front page of the qualification HIT.

About the HITs:

You will be asked to look at a 90 seconds long video that is part of a longer length surgery. Based on the video, you will be asked 6 questions related to:
  1. whether you saw a particular object appear in the video - Yes/No,
  2. what was the activity being performed e.g. cutting tissue, burning tissue or stitching,
  3. what were the robotic tools used for,
  4. how was the skill of the surgeon performing the surgery - similar to C-SATS
Additionally, I have mentioned the fact that we pay a bonus of $0.5 to workers who pass the test in addition to granting them the qualification.
 
Last edited:

LOS

New Member
Requester
Joined
Mar 12, 2016
Messages
72
Reaction score
90
Points
18
We have uploaded shorter versions of the previous HITs on MTurk. These contain a total of 6 videos.
Many workers had requested that the total number of videos in a HIT be reduced, and it made quite sense to do so.
Requester page: https://www.mturk.com/mturk/searchbar?selectedSearchType=hitgroups&requesterId=A2MG9E64Z06RY7

If you have already completed ALL the previous HITs then you don't have anything new and we have assigned qualifications to disallow previous completions.
Otherwise please take up whichever of the longer or shorter versions and help our research.

As always we are fortunate to have such an opportunity to get data that can be used towards meaningful research improving the lives of patients, surgeons, other healthcare providers and our community. And, we are open to receive feedback to improve our HITs and make it convenient for the workers while abiding by our research goals.
Please do review us on TO or other forums so we can obtain the required amount of data sooner :)
 
  • Like
Reactions: clickhappier

Cursed Scroll

Mogg Fanatic
Contributor
Joined
Feb 2, 2016
Messages
178
Reaction score
491
Points
338
Age
40
Gender
Male
Just an FYI for everyone here:

I did the $4.50 today. It took me 55 mins going very slow. I got coffee twice, and emaiiled the requestor during that time. The more videos I watched the faster I got at going thru the questions. I think I might be able to get down to 35 mins with no distractions. I emailed the requestor 2 times, once after I returned a hit after 1 video, and once after I did the whole 10 videos. Each time they got back to me within 5 mins.

The first time I emailed about having the submit button also being the the same as " go to next video" button. They changed that within 10 mins of my email. The second time was about what to do if the instruments where idle but action was happening behind tissue that you could not see. Again, they replied very quick.
 

LOS

New Member
Requester
Joined
Mar 12, 2016
Messages
72
Reaction score
90
Points
18
Just an FYI for everyone here:

I did the $4.50 today. It took me 55 mins going very slow. I got coffee twice, and emaiiled the requestor during that time. The more videos I watched the faster I got at going thru the questions. I think I might be able to get down to 35 mins with no distractions. I emailed the requestor 2 times, once after I returned a hit after 1 video, and once after I did the whole 10 videos. Each time they got back to me within 5 mins.

The first time I emailed about having the submit button also being the the same as " go to next video" button. They changed that within 10 mins of my email. The second time was about what to do if the instruments where idle but action was happening behind tissue that you could not see. Again, they replied very quick.
Thanks Cursed Scroll @Cursed Scroll for this input. We think that initially workers will need time to get accustomed to the HIT. This also has a learning curve like everything else. However, we won't know that unless workers perform these HITs and we are seeing similar trends with other workers as well.
 
  • Like
Reactions: Cursed Scroll

LOS

New Member
Requester
Joined
Mar 12, 2016
Messages
72
Reaction score
90
Points
18
We have launched a new batch of HITs (15 of them) with 35 more to follow later tonight.
We hope you enjoy doing those and help us with our research.
Please let us know if there are any troubles, glitches, etc. We will try our best to resolve them as quickly as possible.

Requester page:
https://www.mturk.com/mturk/searchbar?selectedSearchType=hitgroups&requesterId=A2MG9E64Z06RY7

Regarding additional qualifications:
We will be monitoring the progress of the HITs and accordingly open new slots for qualification tests. We will try and make sure that we post once they are available here.
 

Roscoe E Goldchain

Hood Haiku Specialist
Contributor
HIT Poster
Joined
Jan 25, 2016
Messages
11,714
Reaction score
28,767
Points
1,438
Age
33
Location
VA
Gender
Male
We have launched a new batch of HITs (15 of them) with 35 more to follow later tonight.
We hope you enjoy doing those and help us with our research.
Please let us know if there are any troubles, glitches, etc. We will try our best to resolve them as quickly as possible.

Requester page:
https://www.mturk.com/mturk/searchbar?selectedSearchType=hitgroups&requesterId=A2MG9E64Z06RY7

Regarding additional qualifications:
We will be monitoring the progress of the HITs and accordingly open new slots for qualification tests. We will try and make sure that we post once they are available here.
Hi, this newest batch seems to have some problems

On my second vid, it never loaded. Returned after trying multiple times, Tried again & this time the third vid will not load
 
  • Like
Reactions: Shadowsbane1988

Shadowsbane1988

Literally Pam
Contributor
Joined
Jan 12, 2016
Messages
605
Reaction score
1,077
Points
418
Age
30
Gender
Female
Hi, this newest batch seems to have some problems

On my second vid, it never loaded. Returned after trying multiple times, Tried again & this time the third vid will not load
I'm having the same issue as Roscoe - the second video just won't load :(
 
  • Like
Reactions: LOS

LOS

New Member
Requester
Joined
Mar 12, 2016
Messages
72
Reaction score
90
Points
18
Hi, this newest batch seems to have some problems

On my second vid, it never loaded. Returned after trying multiple times, Tried again & this time the third vid will not load
I have resolved this. Please continue. Apologies for this issue.
 
  • Like
Reactions: Shadowsbane1988

LOS

New Member
Requester
Joined
Mar 12, 2016
Messages
72
Reaction score
90
Points
18
The new batch for qualification score 5+ have been published as well - 10 of them for now.
 
  • Like
Reactions: Shadowsbane1988

Cursed Scroll

Mogg Fanatic
Contributor
Joined
Feb 2, 2016
Messages
178
Reaction score
491
Points
338
Age
40
Gender
Male
After doing 3 of the $4.50 hits and 2 of the $2.25 hits on sunday, I tried to do one today. After 1 video, It said I failed an attention check. I really do not think I did, and another worker on this site said they had the same problem. I'll be emailing them again. I actually like these hits, hope it can get worked out.