Welcome to MTurk Crowd

Work on Amazon Mechanical Turk, learn from the best, and have fun doing it. Join the crowd today!

Sign Up

Rep Present New requester: seeking feedback on a series of HITs

Discussion in 'Requesters' started by pilazza, Apr 28, 2018.

  1. pilazza

    pilazza
    Expand Collapse
    New Member

    • Requester
    Joined:
    Apr 27, 2018
    Messages:
    3
    Likes:
    2
    Hi everyone,

    I'm a newbie requester looking to get some feedback on a series of HITs for identifying and labeling forum/blog/Q&A sites. Early prototypes of the HITs (on the worker sandbox) are up at:

    HIT #1: Identify forum sites:
    https://workersandbox.mturk.com/projects/3GT259VZ8HNMZVGLGI8ZNNUORCVUY6/tasks
    This is a simple single question with instructions. I've got keyboard shortcuts for the options for easy completion. Would it be helpful to have the site itself in an iframe so it displays inline? Or is it better as a link that opens in a new tab/window?

    HIT #2: Find forum root:
    https://workersandbox.mturk.com/projects/3YBH5XUK848XZQQBBMMX49NFP58UX5/tasks
    This is a text box entry, again after looking at and manipulating a website. Same question about iframe vs. link as before. Also, is it clear enough what I mean by forum "root"? Would some examples help?

    HIT #3-5: Label index/thread/post pages:
    https://workersandbox.mturk.com/projects/3EHDCT959WNPFF90D07VRH2NT5YV02/tasks
    https://workersandbox.mturk.com/projects/3YE2HAAZLWCM4LQQC4AI1U3QX6ZUZX/tasks
    https://workersandbox.mturk.com/projects/38NH2CI7SPV5V1LGB7LFGOQU9VUVUZ/tasks
    These involve a custom Javascript overlay that walks you through labeling several (~20) fields on a webpage. I'm most concerned about the granularity of these HITs - I know that it's best to break HITs up into individual tasks that can be completed quickly, but in these cases, the later stages are dependent upon what's selected in early stages, and I can use software to reduce the amount of work the Turker has to do or improve the error-checking on their input. (For example, if no categories are selected, it should hide the request to label category fields, and the set of legal click targets for the field is modified by the items selected in earlier stages.) Are these too daunting & long? Should I break them up so that each "stage" is a separate HIT, moving some of the software logic that's currently in the Javascript to the server? So that each individual labeling task is a separate HIT? Or is it more convenient to be able to click through a number of fields at once all on the same document?

    I'm also interested in feedback on wording, instruction clarity, UI, and pricing, if you have any opinions on those. I'll likely change the color scheme to match the default templates and add some example screenshots in the future.

    Thanks for any input!
     
    Bonitagirl33 and TSolo315 like this.
  2. Achilles2357

    Achilles2357
    Expand Collapse
    Active Member

    Joined:
    May 24, 2017
    Messages:
    966
    Likes:
    1,457
    I think you need a separate signup to get into the sandbox. I've thought about it but I haven't done it yet, so I'll just respond on the fly....
    1) If the hit is only a few cents, I'd say iframe, otherwise new tab. For workers, every keystroke costs money. Even if you provide shortcuts for easy completion, it is quite plausible that some worker will come up with better shortcuts and release this via a script. But some workers will pick this up and some won't. The track record for requesters grasping worker workflow is not that great. Hits done in the iframe carry the risk of accidental early submission, especially if you are trying to do them efficiently. But they are MUCH better if you have some workflow established.
    2) As for instruction clarity, I'd ask yourself how you would respond if a worker complains about an "unjust" rejection. Anyone liable to respond to your posting here will be a fairly good mturker (some much better than me...) But depending on your pay, some, or many, of your hits will be done by less skilled turkers. I'd ask yourself whether you can clearly justify rejections by citing the instructions you have given.
    3-5) As I said, I don't have the sandbox login right now, but my own sense, and my sense of what others think, is that if there are repeated actions, it is best to have these in multiple hits, so that workers can optimize their own workflow. Multiple hits carry overhead, but if a worker commits to doing many of your hits and trusts you, and has established a good workflow for your hits, then having to do many hits is a good thing, and increases a worker's overall rating (both in hits done and acceptances).

    All of that aside, I'd say the most important issues from the worker side are: 1) pay, and 2) trustworthiness (is there a danger of mass rejection?)
     
    Collapse Signature Expand Signature
  3. pilazza

    pilazza
    Expand Collapse
    New Member

    • Requester
    Joined:
    Apr 27, 2018
    Messages:
    3
    Likes:
    2
    Hi, a very belated thanks for your feedback.

    I've redone the series of HITs fairly significantly, based on Turker feedback and some additional requirements on our backend. It now looks something like this:

    https://s3.us-east-2.amazonaws.com/pilazza-turk-in/bitcoin.stackexchange.com/select_threads.html
    https://s3.us-east-2.amazonaws.com/pilazza-turk-in/bitcointalk.org/label_threads.html

    Unfortunately I'm finding that Turkers are almost always getting confused about what they're supposed to do. Either they submit the HIT without selecting anything, or they don't submit anything and put it back. Ideally the Turker would select items like what are shown in the examples:

    https://pilazza-turk-resource.s3.us-east-2.amazonaws.com/example_html/select_threads_6.html
    https://s3.us-east-2.amazonaws.com/pilazza-turk-resource/example_html/label_threads_0.html

    But instead they're selecting nothing. I ran an experiment that asked a bunch of Turkers about the terminology they'd use for the highlighted items, and found that their answers were all over the map. Perhaps the problem is that there's no common vocabulary for labeling the parts of a forum site, and so when I ask them to select a certain header/topic/thread/comment/etc. they don't know what I mean?

    Anyone have any ideas for what might be going wrong, how to clarify the instructions so that they make more sense, or other strategies I could use that would improve accuracy?