Welcome to MTurk Crowd

Work on Amazon Mechanical Turk, learn from the best, and have fun doing it. Join the crowd today!

Sign Up

Rep Present C-SATS - Surgical Video Review

Discussion in 'Requesters' started by C-SATS Inc, Sep 15, 2016.

  1. C-SATS Inc

    C-SATS Inc
    Expand Collapse
    New Member

    • Requester
    Joined:
    Sep 15, 2016
    Messages:
    5
    Likes:
    31
    Hi everybody, we're C-SATS, and we assess surgical skill through video review. Please read this post and our FAQ to learn more about our HITs. If you don't find an answer to your question, please feel free to ask it below.

    A little background about us - we started as a research project to try and measure how good a doctor is at surgery, and quickly learned that the easiest way to do that is simply by watching video. Our research shows that non-surgeons can accurately assess technical skills of a surgeon [1], [2]. That's where you all come in.

    If you've ever done a HIT for us before, you'll know our tasks usually consist of watching a ~10 minute video and then answering 6-8 questions about the skill of the surgeon. Sometimes the videos are much shorter, and the pay for that HIT will reflect the difference. These videos are captured from inside the human body, so they can be a bit graphic if you're easily bothered by such things. But what you're watching for isn't regarding the anatomy - it's about how fluidly and confidently the surgeon uses the surgical tools.

    The first time you do a new review type for us, you might also be asked to watch a brief training video. You'll only have to do that once for that survey type.

    Happy to answer any questions about our HITs and we welcome feedback & suggestions for how to do things better.

    Useful Links:


    C-SATS
     
    #1 C-SATS Inc, Sep 15, 2016
    Last edited: Sep 15, 2016
    RicanGuy86, corvid, ducky and 19 others like this.
  2. Jaded

    Jaded
    Expand Collapse
    The real themildone

    • Administrator
    Joined:
    Jan 10, 2016
    Messages:
    40,842
    Likes:
    104,508
    Welcome to mTurkCrowd!
     
    Collapse Signature Expand Signature
  3. SleepComa

    SleepComa
    Expand Collapse
    Drifting through the stars...

    Joined:
    Sep 7, 2016
    Messages:
    3,769
    Likes:
    7,979
    You guys are loved by the Mturk community :)
     
    Tigerpants and C-SATS Inc like this.
  4. Jenningka

    Jenningka
    Expand Collapse
    Active Member

    • Contributor
    Joined:
    Jun 28, 2016
    Messages:
    637
    Likes:
    1,098
    Welcome! Love your hits, looking forward to more.
     
    C-SATS Inc likes this.
  5. Roscoe E Goldchain

    Roscoe E Goldchain
    Expand Collapse
    Hood Haiku Specialist

    • Contributor
    • HIT Poster
    Joined:
    Jan 25, 2016
    Messages:
    11,534
    Likes:
    28,303
    hi!
     
    Collapse Signature Expand Signature
    C-SATS Inc likes this.
  6. C-SATS Inc

    C-SATS Inc
    Expand Collapse
    New Member

    • Requester
    Joined:
    Sep 15, 2016
    Messages:
    5
    Likes:
    31
    Thanks very much. What would you say the primary reason is for being liked? The HITs are interesting? And what could be done better?
     
  7. humbleturker

    humbleturker
    Expand Collapse
    Well-Known Member

    • Contributor
    • Crowd Pleaser
    • HIT Poster
    Joined:
    Jul 8, 2016
    Messages:
    10,029
    Likes:
    21,990
    You're fair, you don't insult us with the pay, and your HITs are unique. Better than rummaging around in the same surveys each day.
     
    Collapse Signature Expand Signature
  8. Serial Mom

    Serial Mom
    Expand Collapse
    I’m smiling…that alone should scare you.

    • Contributor
    Joined:
    Aug 18, 2016
    Messages:
    4,226
    Likes:
    9,444
    .
     
    Collapse Signature Expand Signature
    #8 Serial Mom, Sep 15, 2016
    Last edited: Sep 15, 2016
  9. Tigerpants

    Tigerpants
    Expand Collapse
    Murderer of Threads

    • Contributor
    Joined:
    Apr 1, 2016
    Messages:
    9,378
    Likes:
    7,978
    I can't speak for everyone, but since I started Mturk about a year ago you have been my favorite requester. I find the content very interesting and loved when you added the procedures and steps.

    As far as improvement goes I can't think of much. The directions are clear and the little notes on the scales help a lot.

    I especially loved the critical view of safety on the gallbladder HITs last week or so, those where so interesting!
     
  10. coffeeowl

    coffeeowl
    Expand Collapse
    taking notes

    • Contributor
    • Crowd Pleaser
    • HIT Poster
    • Master Pleaser
    • Mentor
    Joined:
    Jan 12, 2016
    Messages:
    23,834
    Likes:
    52,577
    Love your HITs!! Thank you very much for reaching out and for updating your FAQ's. Much appreciated!
     
    Collapse Signature Expand Signature
    Serial Mom, C-SATS Inc and Tigerpants like this.
  11. A6_Foul_Out

    A6_Foul_Out
    Expand Collapse
    Collector of $2.00 Baseball UNU Bonuses

    • HIT Poster
    • Contributor
    Joined:
    Aug 17, 2016
    Messages:
    11,175
    Likes:
    20,791
    Fascinating.

    Truly.
     
    Collapse Signature Expand Signature
    C-SATS Inc and Tigerpants like this.
  12. Yatagarasu

    Yatagarasu
    Expand Collapse
    all up in the videos, all on the records, dancin'

    • Contributor
    • Crowd Pleaser
    • HIT Poster
    Joined:
    Jan 13, 2016
    Messages:
    9,415
    Likes:
    11,626
    Oh nice, one of my favorite requesters. Glad to see you here.
     
    Collapse Signature Expand Signature
  13. Squatty

    Squatty
    Expand Collapse
    .

    • Contributor
    • Crowd Pleaser
    • HIT Poster
    Joined:
    Mar 17, 2016
    Messages:
    21,286
    Likes:
    44,987
    I hate to be the negative Nelly.

    I do really enjoy your hits. They are interesting and the Gallbladders last week were amazing. I loved the training video, so I knew what I was looking for.

    But, the pay really isn't that good. You are paying about 11 cents a minute, which comes to $6.60/hour. This is below minimum wage.
    If it is at all possible, could you try to increase the pay to 14 cents a minute?

    Thank you again!
     
    Collapse Signature Expand Signature
  14. humbleturker

    humbleturker
    Expand Collapse
    Well-Known Member

    • Contributor
    • Crowd Pleaser
    • HIT Poster
    Joined:
    Jul 8, 2016
    Messages:
    10,029
    Likes:
    21,990
    I can coattail off of this because I'm just curious about the altered pay rates. Some videos are, as Squatty mentioned, about 11c/min. That being said, you had amazing batches of gallbladders as well as the 50 cent batch that I believe was posted yesterday or the day before that the pay rate was far better. I have two questions: in cases like this, how do you dictate the increased pay in these situations and also, like Squatty mentioned, is it at all possible to increase the pay a little on videos that are more in tune with the wage relationship that she mentioned?
     
    Collapse Signature Expand Signature
    C-SATS Inc, NotTurkfactor and Squatty like this.
  15. C-SATS Inc

    C-SATS Inc
    Expand Collapse
    New Member

    • Requester
    Joined:
    Sep 15, 2016
    Messages:
    5
    Likes:
    31
    Thanks for the honest feedback. Pay rate is something we are working to get a better handle on.

    To respond to both of you at once: obviously, everyone works at different speeds. We try and compute a fair rate that's going to a) allow us to stay in business and b) attract quality crowd workers like yourselves, not scammers. Sometimes, depending on the customer and the kind of video, amount of training, number of questions, etc, your perceived pay rate may jump around. That could be because we over/under estimated how long the HIT would take. That said, your perceived pay rate seems plausible. And yes, we need to ensure we pay a fair pay rate. We may have overestimated the amount of work for the gallbladders HITs and underestimated elsewhere. One of our growing edges needs to be getting to a consistent pay rate / hr.

    In an ideal world, I think we move away from a flat pay rate for all workers and start paying repeating workers who do a good job and pay attention during HITs a higher rate. Honestly, MTurk doesn't make that super easy for us, but as we grow we're looking harder into it.
     
  16. C-SATS Inc

    C-SATS Inc
    Expand Collapse
    New Member

    • Requester
    Joined:
    Sep 15, 2016
    Messages:
    5
    Likes:
    31
    A question for all of you: what kind of training do you wish we provided? Would it be useful to know more about the anatomy or procedure you're looking at? Are there any parts of our HITs that you think should be better explained?

    Interestingly, our research has suggested that teaching workers about anatomy doesn't improve the accuracy of the ratings they give. That might be because the ratings are focused on technical skill, not judgement or decision making.
     
    #16 C-SATS Inc, Sep 15, 2016
    Last edited: Sep 15, 2016
    Tigerpants likes this.
  17. jessers

    jessers
    Expand Collapse
    beer wench

    • Contributor
    Joined:
    Feb 16, 2016
    Messages:
    94
    Likes:
    161
    Can you explain what you expect in terms of written responses? At one time I read that workers should comment in every box, but more recently it was being passed around to "only comment when necessary", meaning some HITs could have no writing whatsoever if the only feedback would be something already covered in the questions (e.g. "works slow but confident").
     
    #17 jessers, Sep 15, 2016
    Last edited: Sep 15, 2016
  18. B1llyBlanco

    B1llyBlanco
    Expand Collapse
    New Member

    Joined:
    Jan 12, 2016
    Messages:
    34
    Likes:
    35
    1.You guys should give some type of guideline as to what constitutes a pass and a fail. For example do they need to be at least all 3's to be passable because since this is a surgeon we are grading, is just average(3 out of 5) good enough? Surgery, at least how I see it is very important and you need to be great at it to be passed so I have been grading on the harsher side. Are these people just students and this is the first in many tests they will take to be qualified to do these procedures or is this the final test and we are saying that if we pass them that we think they are ready to do that actual surgery? I think this would make a difference in the way people rate the pass/fail part at least.

    2. I think that just saying which of the two surgeons are better for the training isn't enough. I think you should show two video for each procedure, one that shows a proficient surgeon and shows the things that they are doing well and one that points out all the things that would be wrong or should be marked off. You should make people watch them once and then embed links to the videos so people can re-watch them whenever they need to.

    3. What you were saying about making HITs for people who do a good job. You could have us enter our worker ID's on each HIT and take the time to look for workers who do a lot of your HITs and are marking the answers the correct way and then make a Qualification on Mturk for them. It may be time consuming at first, but youd have that worker pool on hand from then on and weed out the people you dont want. Or you could use the Mturk ID's to just track bad workers and give them Qualifications that if they have them, it denies them access to the HITs.

    4. As someone mentioned earlier, it's unclear sometimes if you want us to write in the text boxes. Sometimes it's labeled as optional and other times it isnt. I would like clarification on that.
     
    Tigerpants and C-SATS Inc like this.
  19. C-SATS Inc

    C-SATS Inc
    Expand Collapse
    New Member

    • Requester
    Joined:
    Sep 15, 2016
    Messages:
    5
    Likes:
    31
    Sure thing. Recently, you've probably noticed the "OPTIONAL" tag above most of the comment boxes. Obviously, those are not required, though they are appreciated.

    In general, the comments are intended to be used for if/when you have something specific to point out. Maybe you saw the surgeon drop a needle or burn something - that's not something that would get captured in the 1-5 scale. A useful comment like that will get passed directly along to the surgeon themselves for their review! We very much value thoughtful comments like that, but don't expect you to write them all the time. I think eventually we'd implement a bonus system for reviews that leave useful comments. But to be clear, saying "great efficiency overall" when you are giving them a 5 isn't very informative, and you're welcome to not bother saying anything in that case.
     
    #19 C-SATS Inc, Sep 15, 2016
    Last edited: Sep 15, 2016
    jessers likes this.
  20. laby

    laby
    Expand Collapse
    New Member

    Joined:
    Jan 12, 2016
    Messages:
    78
    Likes:
    132
    I agree with @B1llyBlanco@B1llyBlanco about some extra clarification about when to pass/fail. There is of course the obvious ones but then there are lot that are more where they got the job done but they look like they could use more practice to get more comfortable with the tools or to get faster, etc. I did a few of the ones where they were moving the ring over the tube, some of them weren't bad but not the best either, I passed them but left comments about where they could use more practice. Was that ok? Should I have just failed them instead?
     
    Tigerpants likes this.