Rep Present Requester: How to get 36 people online at the same time (multiplayer game)

amaatouq

Member
Requester
Joined
Apr 9, 2017
Messages
132
Reaction score
393
Points
263
Age
31
Hi everyone,

I am a noob requester and I would like to get some feedback and help on running an online multiplayer game on MTurk. The game requires 36 people to be online simultaneously (players are randomized across across different conditions).

So far, I tried to notify a panel of turkers and post on forums about the upcoming experiment time. I then posted a batch of 45 HITs with a link to our platform. In this HIT, the base pay was $2 and one can earn up to $8 of bonuses based on performance. The bonus varied across conditions, as some conditions had higher average play time.

The first 36 workers who clicked the link and passed the attention checks played the game. For the surplus workers (9 workers), they received the base pay of $2 and went to the exit survey directly. I also gave the workers the option to leave the lobby after 3 minutes in case the game didn't start (they also received the base pay of $2 should they decide to leave or they get the option to remain in the lobby and continue waiting).

This hasn't been a huge success the last time I tried it. It took 30 minutes for the game to accumulate enough players to start. As many people left after waiting for 3 minutes in the lobby. I also had to submit another batch of 20 HITs to keep pumping people in.

I am wondering if there is a better way to do this. One way I've been thinking about is the following:
Having a batch of 100 HITs with $0.1 base pay. The first 36 players will play and receive the base payment of $2 + the bonuses based on their condition. The remaining 64 players (surplus as the game already started, or if they left the lobby) will receive $0.1 .. This way, I can keep pumping players into the lobby until the game starts (so people will not have to wait long) and at the same time, I keep within the budget of the project.

Does this sound like a fair thing to do? I am open to suggestions and guidance.
 

electrolyte

The Ghost of MTurk Past
Contributor
Joined
Jan 10, 2016
Messages
19,184
Reaction score
45,970
Points
1,313
Hi everyone,

I am a noob requester and I would like to get some feedback and help on running an online multiplayer game on MTurk. The game requires 36 people to be online simultaneously (players are randomized across across different conditions).

So far, I tried to notify a panel of turkers and post on forums about the upcoming experiment time. I then posted a batch of 45 HITs with a link to our platform. In this HIT, the base pay was $2 and one can earn up to $8 of bonuses based on performance. The bonus varied across conditions, as some conditions had higher average play time.

The first 36 workers who clicked the link and passed the attention checks played the game. For the surplus workers (9 workers), they received the base pay of $2 and went to the exit survey directly. I also gave the workers the option to leave the lobby after 3 minutes in case the game didn't start (they also received the base pay of $2 should they decide to leave or they get the option to remain in the lobby and continue waiting).

This hasn't been a huge success the last time I tried it. It took 30 minutes for the game to accumulate enough players to start. As many people left after waiting for 3 minutes in the lobby. I also had to submit another batch of 20 HITs to keep pumping people in.

I am wondering if there is a better way to do this. One way I've been thinking about is the following:
Having a batch of 100 HITs with $0.1 base pay. The first 36 players will play and receive the base payment of $2 + the bonuses based on their condition. The remaining 64 players (surplus as the game already started, or if they left the lobby) will receive $0.1 .. This way, I can keep pumping players into the lobby until the game starts (so people will not have to wait long) and at the same time, I keep within the budget of the project.

Does this sound like a fair thing to do? I am open to suggestions and guidance.
How long does the whole thing take? You may have problems getting workers to accept it and stick around in the waiting room if the total estimated time on it is long and workers are only guaranteed $2.
 
  • Like
Reactions: amaatouq

Yungskell

Mturk Hustler
Banned
Contributor
Joined
Feb 18, 2017
Messages
1,884
Reaction score
2,223
Points
338
Age
29
Location
Cincinnati, OH
Gender
Male
A @amaatouq I would look at how long the game last and also if its possible lower the amount of people to play the game 36 people is quite a few i feel like it qwould be difficult as you said to maintain enough people. Alot of people I think would be turned away if they waited for it and only got paid .01 being queued up...maybe do say 1.00 for completeing the hit if its under 10-20mins and then pay out like .10 cents for being queued
 
  • Like
Reactions: amaatouq

TSolo315

SnapNCrackle
Administrator
Joined
Jan 12, 2016
Messages
5,000
Reaction score
16,957
Points
2,538
Gender
Male
Up the base pay and kill the return after three minutes option (maybe set a max wait time.) If the HIT pays well, you shouldn't have much trouble getting people to stay. Make it clear that it pays well in the title/instructions.

Submit the HIT here and on other forums when you post it.

Release 100+ HITs and only allow the first 45ish to participate, asking the others to return the HIT. Make sure this is also clear in the instructions/title and happens very early in the HIT.

If I joined a HIT with little knowledge of what it entails or how well it pays, and a long ambiguous wait time, I would easily take the $2 after a few minutes as well.
 
  • Like
Reactions: amaatouq

sryan66611

Well-Known Member
Mentor
Contributor
Joined
Feb 13, 2016
Messages
3,412
Reaction score
5,893
Points
838
Age
33
Gender
Male
I think ur idea seems feasible but it could be problematic as you might not get enough people as electrolyte @electrolyte pointed out because of the low pay and people being leary if you are a new requester.

Another option could be set up a .01 task for a qualification hit and have the attention checks or whatever you want in there. After that set up a hit with a qual and let those people who qualified via email to join the hit.
 

SweetiePuff

No Lifer
Contributor
Crowd Pleaser
HIT Poster
Joined
Oct 10, 2016
Messages
19,283
Reaction score
34,569
Points
1,838
Location
Jersey Girl
Gender
Female
Another consideration and something I've seen frequently (with game type HITs) ... is it implicitly stated in the HIT title: START RIGHT AWAY

You will find many turkers, such as myself, that will grab a HIT and for a variety of reasons and do not start that HIT right away. If your HIT is dependent on turkers playing amongst each other (testing server load, etc) then make it crystal clear that the HIT should be started immediately. This may eliminate the 'pumping in' of add'l players.

Really interested in this HIT ... good luck!
 
  • Like
Reactions: amaatouq

turkenator

━╤デ╦︻(▀̿̿Ĺ̯̿̿▀̿ ̿)
Contributor
Joined
Feb 5, 2016
Messages
187
Reaction score
328
Points
388
Gender
Male
I haven't done one myself in quite a while now, but Yale Institute for Network Science used to do HITs similar to this. Their system seemed to work well. I remember it being something like this:

HIT pays $1.00
Read the rules of the game.
Take comprehension test.
If failed - "Thanks, but you did not pass the test, please return HIT and try again next time"
If passed - go into a waiting room
In the waiting room, wait for the necessary number of participants. There was a timer, like 10 minutes I think.
If the timer runs out before enough people are there - "Thanks, but not enough participants are here, please submit for the base pay."
Extra people who made it into the waiting room, but the game proceeded without them - "Thanks, but you were not needed to participate in the game, please submit for base pay."
People who successfully completed the game - "Submit for the base pay, and any extra earnings from the game will be paid by bonus later."​
 
  • Like
Reactions: amaatouq