Rep Present Requester: How to get 36 people online at the same time (multiplayer game)

[amaatouq]

Hi everyone,

I am a noob requester and I would like to get some feedback and help on running an online multiplayer game on MTurk. The game requires 36 people to be online simultaneously (players are randomized across across different conditions).

So far, I tried to notify a panel of turkers and post on forums about the upcoming experiment time. I then posted a batch of 45 HITs with a link to our platform. In this HIT, the base pay was $2 and one can earn up to $8 of bonuses based on performance. The bonus varied across conditions, as some conditions had higher average play time.

The first 36 workers who clicked the link and passed the attention checks played the game. For the surplus workers (9 workers), they received the base pay of $2 and went to the exit survey directly. I also gave the workers the option to leave the lobby after 3 minutes in case the game didn't start (they also received the base pay of $2 should they decide to leave or they get the option to remain in the lobby and continue waiting).

This hasn't been a huge success the last time I tried it. It took 30 minutes for the game to accumulate enough players to start. As many people left after waiting for 3 minutes in the lobby. I also had to submit another batch of 20 HITs to keep pumping people in.

I am wondering if there is a better way to do this. One way I've been thinking about is the following:
Having a batch of 100 HITs with $0.1 base pay. The first 36 players will play and receive the base payment of $2 + the bonuses based on their condition. The remaining 64 players (surplus as the game already started, or if they left the lobby) will receive $0.1 .. This way, I can keep pumping players into the lobby until the game starts (so people will not have to wait long) and at the same time, I keep within the budget of the project.

Does this sound like a fair thing to do? I am open to suggestions and guidance.

 

[electrolyte]

Hi everyone,

I am a noob requester and I would like to get some feedback and help on running an online multiplayer game on MTurk. The game requires 36 people to be online simultaneously (players are randomized across across different conditions).

So far, I tried to notify a panel of turkers and post on forums about the upcoming experiment time. I then posted a batch of 45 HITs with a link to our platform. In this HIT, the base pay was $2 and one can earn up to $8 of bonuses based on performance. The bonus varied across conditions, as some conditions had higher average play time.

The first 36 workers who clicked the link and passed the attention checks played the game. For the surplus workers (9 workers), they received the base pay of $2 and went to the exit survey directly. I also gave the workers the option to leave the lobby after 3 minutes in case the game didn't start (they also received the base pay of $2 should they decide to leave or they get the option to remain in the lobby and continue waiting).

This hasn't been a huge success the last time I tried it. It took 30 minutes for the game to accumulate enough players to start. As many people left after waiting for 3 minutes in the lobby. I also had to submit another batch of 20 HITs to keep pumping people in.

I am wondering if there is a better way to do this. One way I've been thinking about is the following:
Having a batch of 100 HITs with $0.1 base pay. The first 36 players will play and receive the base payment of $2 + the bonuses based on their condition. The remaining 64 players (surplus as the game already started, or if they left the lobby) will receive $0.1 .. This way, I can keep pumping players into the lobby until the game starts (so people will not have to wait long) and at the same time, I keep within the budget of the project.

Does this sound like a fair thing to do? I am open to suggestions and guidance.

How long does the whole thing take? You may have problems getting workers to accept it and stick around in the waiting room if the total estimated time on it is long and workers are only guaranteed $2.

 

[Yungskell]

A @amaatouq I would look at how long the game last and also if its possible lower the amount of people to play the game 36 people is quite a few i feel like it qwould be difficult as you said to maintain enough people. Alot of people I think would be turned away if they waited for it and only got paid .01 being queued up...maybe do say 1.00 for completeing the hit if its under 10-20mins and then pay out like .10 cents for being queued

 

[TSolo315]

Up the base pay and kill the return after three minutes option (maybe set a max wait time.) If the HIT pays well, you shouldn't have much trouble getting people to stay. Make it clear that it pays well in the title/instructions.

Submit the HIT here and on other forums when you post it.

Release 100+ HITs and only allow the first 45ish to participate, asking the others to return the HIT. Make sure this is also clear in the instructions/title and happens very early in the HIT.

If I joined a HIT with little knowledge of what it entails or how well it pays, and a long ambiguous wait time, I would easily take the $2 after a few minutes as well.

 

[sryan66611]

I think ur idea seems feasible but it could be problematic as you might not get enough people as @electrolyte pointed out because of the low pay and people being leary if you are a new requester.

Another option could be set up a .01 task for a qualification hit and have the attention checks or whatever you want in there. After that set up a hit with a qual and let those people who qualified via email to join the hit.

 

[SweetiePuff]

Another consideration and something I've seen frequently (with game type HITs) ... is it implicitly stated in the HIT title: START RIGHT AWAY

You will find many turkers, such as myself, that will grab a HIT and for a variety of reasons and do not start that HIT right away. If your HIT is dependent on turkers playing amongst each other (testing server load, etc) then make it crystal clear that the HIT should be started immediately. This may eliminate the 'pumping in' of add'l players.

Really interested in this HIT ... good luck!

 

[turkenator]

I haven't done one myself in quite a while now, but Yale Institute for Network Science used to do HITs similar to this. Their system seemed to work well. I remember it being something like this:

HIT pays $1.00
Read the rules of the game.
Take comprehension test.
If failed - "Thanks, but you did not pass the test, please return HIT and try again next time"
If passed - go into a waiting room
In the waiting room, wait for the necessary number of participants. There was a timer, like 10 minutes I think.
If the timer runs out before enough people are there - "Thanks, but not enough participants are here, please submit for the base pay."
Extra people who made it into the waiting room, but the game proceeded without them - "Thanks, but you were not needed to participate in the game, please submit for base pay."
People who successfully completed the game - "Submit for the base pay, and any extra earnings from the game will be paid by bonus later."​

 

[amaatouq]

Great everyone! thank you very much for the constructive feedback. We incorporated many of these ideas into our game. We will see how that would work today (Thursday 13th of April)!

http://www.mturkcrowd.com/threads/thur-13-april-2pm-est-guess-the-correlation-and-get-up-to-10.2252/