Avoiding Common Mistakes when Publishing a HIT

TSolo315

SnapNCrackle
Administrator
Joined
Jan 12, 2016
Messages
5,000
Reaction score
16,975
Points
2,538
Gender
Male
Avoiding Common Mistakes when Publishing a HIT

In this tutorial we will be covering a list of common mistakes that requesters make when designing and publishing a HIT. Learning how to avoid these mistakes will increase the quality of your results and will help workers feel more comfortable while working on your HITs.

1. Making the HIT Instructions Confusing

When designing a HIT, it is important to make the HIT instructions (those found directly in the HIT preview) as clear and straightforward as possible. While it may sound obvious, it is not uncommon for HITs to produce poor results due to ambiguous or incomplete instructions. When writing the HIT instructions, you will want to leave as little room for ambiguity as possible. Try to anticipate any problems workers may come across and provide appropriate solutions. Also, before publishing your HIT, make sure to test that everything is working properly.
Log in or register now. to view Spoiler content!

HIT instructions should also be clear regarding what a worker should expect from the HIT, including anything that may prevent a worker from successfully completing the HIT, and consequently forcing the worker to return it. There are a variety of reasons a worker may be unable to complete a HIT, including environmental, technical, and disability related issues. For example, someone with a hearing disability would be unable to complete a hit with audio elements. Mentioning in the hit instructions that audio is a requirement would help this worker avoid accepting the HIT, only to return it halfway through upon learning that audio is a necessity. If you are publishing a survey with a screening section, make sure to explicitly mention this in the instructions, and to place the screening section at the very beginning of the survey. Few things will frustrate a worker more than spending a significant amount of time on a HIT only to be forced to return it.

Here is an example of HIT instructions that are well written:



2. Using a Short HIT Timer

A common mistake that is made by new requesters is setting the HIT timer to the amount of time they believe the hit will take. Because not everyone works at the same pace, this can cause some workers to feel rushed, and in turn, submit lower quality data. In the worst case scenario, the HIT will expire prior to the worker finishing, and they will be unable to submit their work. It is suggested that requesters set the HIT timer to at least 2-3 times the time they expect the hit to take. This will allow workers to work at a comfortable pace and will leave room for any minor interruptions.



When publishing a survey on Mechanical Turk, it is a good practice to provide an estimate in the HIT instructions of the time required to complete the survey. Try to provide as accurate an estimate as possible – ask friends or colleagues to take the survey and record their times. This will also be helpful in catching any potential errors prior to publishing the survey.

3. Violating Amazon’s Terms of Service

Before publishing a HIT, it is important to make sure the HIT does not violate Amazon’s Terms of Service. Common violations include asking workers for personal identifying information, asking workers to register for a website, or asking workers to violate the terms and conditions of another website. The full policy page can be found here: https://www.mturk.com/mturk/help?helpPage=policies.

4. Rejecting HITs That Should Not be Rejected

Rejecting a HIT should be reserved for instances when a worker submits poor or unusable work, or for obvious cases of cheating. Because rejections can affect a worker’s ability to perform certain HITs in the future, workers take them very seriously. Wrongly rejecting HITs is an easy way to find your inbox filled with confused or frustrated emails. While the use of rejections is understandable when poor work is submitted, extra care should be taken to make sure the rejection is well deserved.

To safeguard against poor data, it is common for requesters to include attention checks in their surveys. While such precautions are understandable, it is important that these questions be obvious to anyone completing the survey honestly. Do not use attention check questions that are particularly complicated or that are meant to “trick” respondents. If you are going to reject a HIT due to a missed attention check, you want to be sure the question was missed due to a lack of effort or attention, not from any unrelated confusion.

If you accidentally reject a hit that should not have been rejected, it is possible to reverse the rejection, as long as you have not deleted the HIT results from the requester interface. A guide on how to reverse rejections can be found here: http://docs.aws.amazon.com/AWSMechTurk/latest/RequesterUI/PublishingYourBatchofHITs.html

5. Asking workers Not to Repeat a HIT That You or Your Colleagues Have Posted Previously

It is perfectly reasonable not to want repeat survey responses. Expecting workers to remember taking your survey among all the other surveys on Mechanical Turk, however, may be asking a bit too much. It is much more practical to use one of the many available methods to prevent repeat responses. A list of these methods can be found here: http://wiki.wearedynamo.org/index.php?title=Basics_of_how_to_be_a_good_requester#Avoid_duplicates.2Fretakes_in_fair_ways.

IMPORTANT: Do not block workers to prevent survey retakes! Blocking a worker can put their account at risk, often resulting in that worker receiving an email threatening their account with suspension. Blocks should only be used in cases of obvious scamming/cheating.

6. Paying Amazon Too High of a Fee

Amazon charges a fee of 40% (up from 20%) if you post more than nine assignments per HIT. To avoid this extra fee, make sure to only post a maximum of nine assignments per HIT. This can be very annoying if you need a large number of survey responses, as you will need to set up a large number of HITs. Here is one method to make it a little easier: https://docs.google.com/presentatio...oZidKzKXqFVJaLH4/edit#slide=id.ga3bec7adb_1_6

Using the Masters qualification, which is on by default, will also result in an extra 5% fee. There has been no evidence provided that those with the Masters qualification deliver higher quality results. You are better off setting up your own custom qualifications.


Additional resources:

http://wiki.wearedynamo.org/index.php?title=Basics_of_how_to_be_a_good_requester