(continuation) ... So, yeah. Some HITs never get reviewed because, why would you tell a million people how awesome your favourite batch is? I mean, being nice is good and all but, that's the sort of mentality that lots of people have and I don't disagree with it. Regardless of what my specific thoughts are I understand. No point in telling your competition how to make
your money, other than for karma. Lots of people care more about paying bills that collecting karma points.
Next point, just like reviews on Amazon, not all reviews are created equally, so to speak. Some people are bad at HITs. So, they review people from the perspective of being rejected for what they thought was good work, regardless of how little they followed the instructions. If that's the only review, the requester's score is tanked. But, unfortunately, that doesn't reflect the requester, just the reviewer.
And... TO(turkopticon) is going 'ReadOnly' soon, if not already. So, if that's where you get your reviews, no new ones will be coming.
To touch on the point of no reviews, they're almost always new requesters. As with anything, some new things are good, some are bad, some start off bad and turn good and sometimes the revers happens. Some requesters, when new, are just as new to Mturk as you are. So, they can sometimes have novice problems. That said, some requesters will work through those problems, usually with the help of turkers like you or me. For batches, you'll do a few, message the requester,"Hello. I am completing, and enjoying, your HITs from the batch that posted earlier but, I have a few(questions, suggestions, clarifications, etc). I have been coming across a lot of xxxx that could be marked either xxx or, xxx. What is the best choice in this situation? I only ask to help me to provide you with the best possible data. Thanks in advance for a quick reply.
TurkerAYOU86987987"
If they get back to you with a good answer, probably a good requester. And now they know that you care about completing quality work. If they go to Qualified batches, your in good shape.
If it's not a batch, then helping them by messaging about any errors or broken studies has gotten me a bonus more than once. Some people would review it as "Broken. Do not attempt!" Then they have a bad score, broken HIT and no idea what they are doing. I do some of those too. I like being able to prove that a study isn't broken, just not done correctly. I also like to find the broken ones and help the requesters occasionally.
If a study(HIT) from a requester has no reviews but is being posted by a school(university, college, school "lab, etc.) are pretty safe. You can take that into consideration when deciding to do a non-reviewed HIT. Same with HITs from giant companies.(Amazon, Google[Endor], Snapchat, etc)
Basically, to sum up and answer your question: take all reviews, or lack of reviews with a grain of salt. There are other things that should get equal consideration
for you like pay, timer, type of work and if it is something that
you can be okay with doing for the pay that is offered. Because, no matter what any of the reviews say, or don't say, only you know what you like and no amount of reviews can convince you that a HIT is worth doing if
you don't feel the same about that type of, or that specific HIT. If you don't like transcription, doesn't matter what the reviews are. They'll still suck, to you. Something as small as a "Recall a time when..." writing prompt can change someone's review significantly. It only takes one extra page to turn a review from, "Great HIT!" to ,"Bubble HELL!"
So, use your best judgement and don't let a review sway your decision too much in either direction(barring extreme cases). And with no reviews, still rely on the information that you have(pay, description, timer) to decide for yourself if it looks like it's worth doing. Most of the time you'll be fine and, sometimes when you should be fine, you still get a random rejection or a horrible requester. Just never get too deep(too many pending HITs) with one requester, until you feel safe with that requester.
Good luck.
Sorry so long.
