Welcome to MTurk Crowd

Work on Amazon Mechanical Turk, learn from the best, and have fun doing it. Join the crowd today!

Sign Up

How to Publish a Survey on Mechanical Turk

Discussion in 'Requester Support' started by TSolo315, Feb 28, 2016.

  1. TSolo315

    TSolo315
    Expand Collapse
    SnapNCrackle

    • Moderator
    Joined:
    Jan 12, 2016
    Messages:
    3,102
    Likes:
    10,522
    How to Publish a Survey on Mechanical Turk

    Should I create my survey directly on Mechanical Turk or should I use online survey software?

    When publishing a survey on Mechanical Turk there are two primary options you can choose from. You can write a survey directly in a HIT using the tools provided on the mechanical turk website, or write a survey using online survey software, like Qualtrics, and then create a link to the survey in your HIT (Qualtrics will be used as the example online survey software in this tutorial as it is the most commonly utilized by requesters.)

    [​IMG]

    Using online survey software is the more popular of the two options, as the survey creation interface on Mechanical Turk is rather limited; If you have any need for features such as timed pages or skip/branch logic, using the Mechanical Turk interface is simply not an option. This software also offers clean results pages that a requester may prefer to those on Mechanical Turk. If the survey is relatively straightforward however, consisting of simple multiple choice, short answer, or writing prompt questions, the Mechanical Turk survey creation interface is an available option. When creating a survey using the Mechanical Turk interface, you will require an understanding of basic HTML.

    Publishing a survey created using online survey software:

    Publishing a survey that was created using online survey software is simple. First, of course, you will need to have written a survey. If you are unfamiliar with software like Qualtrics, it is designed to be relatively user friendly -- anyone can open a trial account and start writing a survey. Tutorials for beginners can be found here: http://www.qualtrics.com/university/researchsuite/misc-pages/misc/learn-qualtrics-in-5-steps/

    When the survey is ready and has been launched, you will be provided with a survey link (a link respondents can click that will lead them to your survey.) The following video will outline the steps required to use this link to create a HIT on Mechanical Turk. You must first have created and be logged into a Mechanical Turk requester account here: https://requester.mturk.com/.



    A few points of clarification:
    • Amazon charges a fee of 40% (up from 20%) if you post more than nine assignments per HIT. To avoid this extra fee, make sure to only post a maximum of nine assignments per HIT. This can be very annoying if you need a large number of responses, as you will need to set up a large number of HITs. Here is one method to make it a little easier: https://docs.google.com/presentatio...oZidKzKXqFVJaLH4/edit#slide=id.ga3bec7adb_1_6
    • It was recommended in the video that the Masters qualification not be used. Using the Masters qualification changes the payout to Amazon from 20% to 25%, and there has been no evidence provided that those with the Masters qualification deliver higher quality results. It is no longer required to hit the "advanced" button on the HIT set up page to remove masters and to set custom qualification requirements, as these options are now shown on the main page.
    • After altering the HIT properties and clicking the Design Layout button, you will be brought to the design layout tab. Everything within this tab can be edited, including the hit instructions, and most importantly, the survey link. This is where you will input the survey link you received upon launching your survey (replacing the example link.)
    • A survey code refers to the confirmation code you will provide to respondents at the end of your survey. Respondents will input this code into the relevant field to confirm that they finished the survey prior to submitting the HIT.
      If you wish to provide each respondent with a separate survey code, you can do so by following the method described in the Qualtrics FAQ, “Qualtrics automatically generates an ID which is associated with every survey response. While this ID is not consistent across respondents or surveys, it can function as a code number to display to respondents. To have a code number for respondents, you can use the “Response IDs.” These are a random combination of letters and numbers automatically generated by Qualtrics in connection with each distinct response. Keep in mind that there is no response ID until the response has been submitted; thus it is suggested to pipe in the response ID in an End of Survey Message. Our piped text feature will capture each respondent’s unique ID, which are stored with each response. In setting up your End of Survey Message, use the icon on the left {a}, “Insert Piped Text.” You’ll want to pipe text from an “embedded data” field, “ResponseID.” This way each respondent can view his/her response ID, and this information will also automatically show up in the results.”
    Publishing a survey created using the Mechanical Turk Interface:

    Creating and publishing a survey using Mechanical Turk is very similar to publishing one created using survey software. First, from the Mechanical Turk requester dashboard (https://requester.mturk.com/) click on the Create tab, and then click New Project. You will then choose the Survey option (as opposed to the Survey Link option) from the Start a New Project Menu and click Create Project.

    [​IMG]

    You will find yourself on the HIT Properties page – fill out the information following the same procedure outlined in the How to Launch a Survey on Mechanical Turk video. When you are finished, click the Design Layout button. You will be moved to the Design Layout Tab; it is here that you will create your survey. The page is pre-filled with example questions to help you visualize what each question type will look like in your HIT. Click on the Source button to begin editing your survey in HTML format. You can click on the Source button again at any time to visualize your changes.

    [​IMG]

    After you are done creating your survey, click the Preview button at the bottom right corner of the screen. You will be redirected to a final preview of what the survey will look like in the HIT. If you are satisfied, click Finish.

    To publish your survey, move to the Create tab once more and click on the Publish Batch button, next to the title of the survey you just created. Give your HIT one final preview, double check your HIT properties (and that you have enough money available in your account,) and then click the Publish HITs button.

    [​IMG]

    Final notes: If relevant, make sure your server can handle the load from workers. Many times surveys will fail because of the unanticipated server load from workers all doing the HIT at the same time.

    It is possible to set up and test your HITs using the Requester Sandbox: https://requester.mturk.com/developer/sandbox

    Additional resources:

    http://www.mturkcrowd.com/threads/avoiding-common-mistakes-when-publishing-a-hit.300/

    http://www.mturkcrowd.com/threads/video-guides-tutorials-for-mturk-requesters.65/

    https://requester.mturk.com/help/getting_started.html

    http://www.emotionsandmedia.com/wp-content/uploads/2014/06/MTurkPrimer.pdf
     
    Collapse Signature Expand Signature
    Jaded, Tribune, aveline and 5 others like this.
  2. TQueen

    TQueen
    Expand Collapse
    Southern Sweetness

    • Contributor
    • Crowd Pleaser
    Joined:
    Jan 12, 2016
    Messages:
    15,881
    Likes:
    41,247
    Thanks for putting this up @TSolo315@TSolo315 , great info and very helpful :thumbup:
     
    Collapse Signature Expand Signature
  3. TSolo315

    TSolo315
    Expand Collapse
    SnapNCrackle

    • Moderator
    Joined:
    Jan 12, 2016
    Messages:
    3,102
    Likes:
    10,522
    Collapse Signature Expand Signature
    Jaded and Tribune like this.