How to Publish a Survey on Mechanical Turk

TSolo315

SnapNCrackle
Administrator
Joined
Jan 12, 2016
Messages
5,000
Reaction score
16,975
Points
2,538
Gender
Male
How to Publish a Survey on Mechanical Turk

Should I create my survey directly on Mechanical Turk or should I use online survey software?

When publishing a survey on Mechanical Turk there are two primary options you can choose from. You can write a survey directly in a HIT using the tools provided on the mechanical turk website, or write a survey using online survey software, like Qualtrics, and then create a link to the survey in your HIT (Qualtrics will be used as the example online survey software in this tutorial as it is the most commonly utilized by requesters.)



Using online survey software is the more popular of the two options, as the survey creation interface on Mechanical Turk is rather limited; If you have any need for features such as timed pages or skip/branch logic, using the Mechanical Turk interface is simply not an option. This software also offers clean results pages that a requester may prefer to those on Mechanical Turk. If the survey is relatively straightforward however, consisting of simple multiple choice, short answer, or writing prompt questions, the Mechanical Turk survey creation interface is an available option. When creating a survey using the Mechanical Turk interface, you will require an understanding of basic HTML.

Publishing a survey created using online survey software:

Publishing a survey that was created using online survey software is simple. First, of course, you will need to have written a survey. If you are unfamiliar with software like Qualtrics, it is designed to be relatively user friendly -- anyone can open a trial account and start writing a survey. Tutorials for beginners can be found here: http://www.qualtrics.com/university/researchsuite/misc-pages/misc/learn-qualtrics-in-5-steps/

When the survey is ready and has been launched, you will be provided with a survey link (a link respondents can click that will lead them to your survey.) The following video will outline the steps required to use this link to create a HIT on Mechanical Turk. You must first have created and be logged into a Mechanical Turk requester account here: https://requester.mturk.com/.


A few points of clarification:
  • Amazon charges a fee of 40% (up from 20%) if you post more than nine assignments per HIT. To avoid this extra fee, make sure to only post a maximum of nine assignments per HIT. This can be very annoying if you need a large number of responses, as you will need to set up a large number of HITs. Here is one method to make it a little easier: https://docs.google.com/presentatio...oZidKzKXqFVJaLH4/edit#slide=id.ga3bec7adb_1_6
  • It was recommended in the video that the Masters qualification not be used. Using the Masters qualification changes the payout to Amazon from 20% to 25%, and there has been no evidence provided that those with the Masters qualification deliver higher quality results. It is no longer required to hit the "advanced" button on the HIT set up page to remove masters and to set custom qualification requirements, as these options are now shown on the main page.
  • After altering the HIT properties and clicking the Design Layout button, you will be brought to the design layout tab. Everything within this tab can be edited, including the hit instructions, and most importantly, the survey link. This is where you will input the survey link you received upon launching your survey (replacing the example link.)
  • A survey code refers to the confirmation code you will provide to respondents at the end of your survey. Respondents will input this code into the relevant field to confirm that they finished the survey prior to submitting the HIT.
    Log in or register now. to view Spoiler content!
Publishing a survey created using the Mechanical Turk Interface:

Creating and publishing a survey using Mechanical Turk is very similar to publishing one created using survey software. First, from the Mechanical Turk requester dashboard (https://requester.mturk.com/) click on the Create tab, and then click New Project. You will then choose the Survey option (as opposed to the Survey Link option) from the Start a New Project Menu and click Create Project.



You will find yourself on the HIT Properties page – fill out the information following the same procedure outlined in the How to Launch a Survey on Mechanical Turk video. When you are finished, click the Design Layout button. You will be moved to the Design Layout Tab; it is here that you will create your survey. The page is pre-filled with example questions to help you visualize what each question type will look like in your HIT. Click on the Source button to begin editing your survey in HTML format. You can click on the Source button again at any time to visualize your changes.



After you are done creating your survey, click the Preview button at the bottom right corner of the screen. You will be redirected to a final preview of what the survey will look like in the HIT. If you are satisfied, click Finish.

To publish your survey, move to the Create tab once more and click on the Publish Batch button, next to the title of the survey you just created. Give your HIT one final preview, double check your HIT properties (and that you have enough money available in your account,) and then click the Publish HITs button.



Final notes: If relevant, make sure your server can handle the load from workers. Many times surveys will fail because of the unanticipated server load from workers all doing the HIT at the same time.

It is possible to set up and test your HITs using the Requester Sandbox: https://requester.mturk.com/developer/sandbox

Additional resources:

http://www.mturkcrowd.com/threads/avoiding-common-mistakes-when-publishing-a-hit.300/

http://www.mturkcrowd.com/threads/video-guides-tutorials-for-mturk-requesters.65/

https://requester.mturk.com/help/getting_started.html

http://www.emotionsandmedia.com/wp-content/uploads/2014/06/MTurkPrimer.pdf