New Requester: Seeking Advice on How to Limit a Batch to One Worker

[liam_w]

Hi everyone,

Brand new requester here. We have a large number of stores which we’d like to score on attributes that impact their performance (visibility, ingress-egress, etc.). There is some subjectivity involved in this scoring, so we would like to set up a feedback loop to help the worker align with how we would score a store.

Because of this learning curve we’d like to keep the number of workers limited and allow them to work on a large number of stores. We would start small, with five HITs in a batch, and give the worker feedback on their scoring. How can we set up our request so that one worker completes all of the HITs in a batch rather than the HITs being split out among multiple workers?

Thanks for your help!

 

[esprit-hyperdoux]

You probably want to run a pilot HIT that you've designed in such a way as to allow you to rate worker's performance on it (so like, you can compare their responses to data you already have) and give workers a qualification on the basis of that, which you can use to restrict your subsequent HITs to workers from a specific pool.
I'd just say to make sure that you put a fair reward on your initial, open-qual HIT, especially if there's a lot of complicated instructions to follow, making sure that the workers get a fair hourly rate based on all the time they put in. Also, don't reject work that was completed in good faith! Not only is that not a cool thing to do, but as a new requester you need to work on getting a reputation for fairness, so turkers will be interested working in your HITs. Your pilot will allow you get an idea of where you need to adjust your instructions & layouts to avoid confusing or misleading the workers going forward. The platform doesn't give much of an opportunity for giving feedback *without* rejecting work unfortunately. (you shouldn't be soliciting contact info through your HITs either).
One suggestion that comes to mind though is to assign quals to individual turkers, so exactly one person can work on a given HIT, and giving them feedback in the instructions.
Anyway, good luck!

 

[liam_w]

Thank you for your advice. Based on your insights and some additional research, I’m contemplating the following approach to enable a worker to address multiple HITs at a time. Can you tell me whether any of this is out of line?

First, I thought I might ask whether anyone on this forum is interested in participating. The pilot HIT is in the MT sandbox under the Requester Dena Whitaker. Anyone who checks it out and is interested may respond here with their worker ID.

Second, I would group the five stores for the pilot into a batch of HITs. I understand that if I keep the metadata the same for all of the HITS then they will be grouped together for the worker. I will create a custom qualification for these HITs (say, “Store Classifier 1”) and assign one of the workers who has responded in the forum to this classification. It seems like this would limit all five HITs in the batch to that one worker.

Third, I would create a second, third, fourth and fifth batch of five HITs apiece. Each of these batches would be slightly differentiated by the description (Store Classification 2, 3, 4, 5) for the HITs within them. Then, like the first batch, I would create a custom qualification for the HITS within each (say, “Store Classifier 2, 3, 4, 5”) and assign one of the workers who has responded in the forum to each of these classifications.

As you recommend, I will not reject any work which appears to have been done in good faith. In the sandbox the reward is listed as $2.50, but that’s the “long term” rate. For this first set of batches I’ll set the rate at $3.50 to help offset the time to go through the instructions. For workers who have done a good job and whom I’d like to offer further batches to do, I will give a $.01 bonus. In the message accompanying the bonus I will give feedback on how the worker’s scoring for each store aligned with our scoring.

I hope to develop a core of workers who can devote a significant amount of time to this project. At the end of the project I will also give some bonus to those who make significant contributions.

Thanks!

 

[TSolo315]

Here's the link to the HIT for those interested in looking it over:

https://workersandbox.mturk.com/projects/329HSLC9XFD8SUST84EEK0NT4D4GH7/tasks?ref=w_pl_prvw

 

[charlie]

looks like they're all gone. you get anywhere with them at this point?