Rep Present Question–answer pairs and question answering

Julian Michael

New Member
Requester
Joined
Mar 28, 2017
Messages
18
Reaction score
45
Points
13
Age
30
Gender
Male
The two hits completely contradict each other, but I can't even talk about it without discussing the qualification test on the one task.
Could you please explain the issue? I would like to fix any ambiguities or problems if possible. I also want the qualification test to be reasonable. It is mainly meant to get people to make sure they fully understand the instructions before starting on the task so they don't bring down other people's agreement rates.
 

VGB

dr. peepee poopoo
Joined
Jan 21, 2016
Messages
5,753
Reaction score
15,213
Points
1,038
Age
38
Location
Milwaukee via Boston
Gender
Male
Hi everyone, I'm new to the forums (and a relatively new requester). I have been posting HITs for writing question–answer pairs about English sentences, as well as answering them. Several people working on my task recommended I check out MTurk Crowd and introduce myself.

Since I can't post links yet, the HIT type IDs for my tasks are 36SUH4ZPJUVEFKCRRRIAVB1LGOZ705 (question–answer writing) and 3XRW87W7OXAP1BEXLSFQFDKLIPNTQ0 (question answering).

I'd be happy to receive any feedback on the task design, etc. I will also post about it in the daily work thread and I'm happy to have any more people join onto the task.. :) there is a bit of a backlog on the question answering task right now so there are lots of HITs available.

I will not be able to make huge changes to the design any time soon, but feedback would be really valuable for designing future tasks.

Thanks, and nice to meet you all!
Julian
Welcome to the forum!

I've been doing these for the past couple of weeks and have no issues with the explanations, the instructions are quite thorough and go over everything you'd need to know.

From a task design standpoint, if there was a way to put the instructions into a collapsible box/frame it might help. Due to the sheer amount of them, it makes navigating the task a little harder than necessary once you understand how they are done.

Also a slight pay bump to the answering portion would be nice, .09 or .10. Some of the text requires some mental gymnastics to parse, which makes even simple 3-4 question answers take extra time. These are slightly mitigated by the occasional extremely simple snippet that can be done in seconds, though.

Overall they're quite enjoyable, and you've been friendly through the whole process, to boot.
 

Julian Michael

New Member
Requester
Joined
Mar 28, 2017
Messages
18
Reaction score
45
Points
13
Age
30
Gender
Male
Welcome to the forum!

I've been doing these for the past couple of weeks and have no issues with the explanations, the instructions are quite thorough and go over everything you'd need to know.

From a task design standpoint, if there was a way to put the instructions into a collapsible box/frame it might help. Due to the sheer amount of them, it makes navigating the task a little harder than necessary once you understand how they are done.

Also a slight pay bump to the answering portion would be nice, .09 or .10. Some of the text requires some mental gymnastics to parse, which makes even simple 3-4 question answers take extra time. These are slightly mitigated by the occasional extremely simple snippet that can be done in seconds, though.

Overall they're quite enjoyable, and you've been friendly through the whole process, to boot.
Thanks!

That advice is helpful. I tried to calibrate the payment for answering so that the two tasks would go at about equal speed. However, the answering task has slowed down since I added the qualification so I was thinking of increasing it already... ok, I will do that in the next batch (once all the ones currently uploaded are completed).

Re: the instructions, I was a bit worried that making them collapsed by default would lead to more people not bothering to read them, or missing them. But I have a compromise solution: I'll just put them at the bottom.
 

Jharkan

Internets EXPERT
Contributor
Joined
Jul 9, 2016
Messages
11,873
Reaction score
20,418
Points
1,288
Location
Earth
Gender
Male
Could you please explain the issue? I would like to fix any ambiguities or problems if possible. I also want the qualification test to be reasonable. It is mainly meant to get people to make sure they fully understand the instructions before starting on the task so they don't bring down other people's agreement rates.
well, I guess I'm just stupid,
if other people are able to do the hits

even though the test results I got suggest that you must include redundant sentences,
and then the task instructions themselves say not to include anything redundant.
so when I receive conflicting information like that,
and then see people claim to have no problems,
obviously there's a disconnect,
and I'm probably the common problem.
 

humbleturker

Well-Known Member
Contributor
Crowd Pleaser
HIT Poster
Joined
Jul 8, 2016
Messages
10,554
Reaction score
23,278
Points
1,738
Age
33
Gender
Male
I'm just gonna say this is a properly difficult qual test. I'm not complaining, and I did a few of those HITs before they became qualled. It'll just make it more worth it when I eventually pass. :p
 

WalkingEmphasis

Banned
Banned
Contributor
Joined
Apr 11, 2016
Messages
4,424
Reaction score
8,154
Points
888
well, I guess I'm just stupid,
if other people are able to do the hits

even though the test results I got suggest that you must include redundant sentences,
and then the task instructions themselves say not to include anything redundant.
so when I receive conflicting information like that,
and then see people claim to have no problems,
obviously there's a disconnect,
and I'm probably the common problem.
I think you probably misunderstood something. I can't think of where in the task instructions you might have thought it said not to include redundant answers. The good thing is, you can retake the qualification if you still want to.
 
  • Like
Reactions: Vanyanka and Pleco

Jharkan

Internets EXPERT
Contributor
Joined
Jul 9, 2016
Messages
11,873
Reaction score
20,418
Points
1,288
Location
Earth
Gender
Male
I think you probably misunderstood something. I can't think of where in the task instructions you might have thought it said not to include redundant answers. The good thing is, you can retake the qualification if you still want to.
6th requirement

doesn't matter, I redid the test, but can't get higher than 40
obviously, this task isn't for me.


Requirements
This task is best fit for native speakers of English. Your response must be grammatical, fluent English that satisfies the following criteria:

  1. Either the question or the answer contains the special word.
  2. The question contains at least one word from the sentence.
  3. The question is about the meaning of the sentence (and not, for example, the order of the words).
  4. The question is answered obviously and explicitly in the sentence.
  5. The question is open-ended: yes/no and either/or questions are not allowed.
  6. None of your question-answer pairs are redundant with each other, even for different special words.
 
  • Like
Reactions: TheresaofAlbion

Julian Michael

New Member
Requester
Joined
Mar 28, 2017
Messages
18
Reaction score
45
Points
13
Age
30
Gender
Male
6th requirement

doesn't matter, I redid the test, but can't get higher than 40
obviously, this task isn't for me.
...
6. None of your question-answer pairs are redundant with each other, even for different special words.
What requirement 6 means is that as the question writer, you should not write redundant questions. So, as the question answerer, you should mark when questions are redundant so the question writer doesn't get rewarded for that question and so it counts against the question writer's accuracy.

I'll admit that the test is actually pretty tricky and I have to stop and think when taking it. The particular issues are about 1) how redundancy is defined in the task, and 2) what it means for a sentence to "obviously answer a question" by virtue of its meaning. On the former point, you have to be careful because questions that look similar might not actually count as redundant. On the latter point it's a pretty squishy issue and hard to explicitly write out any instructions without going into linguistics—but I try to illustrate it with a bunch of examples. Even then, there is a fair amount of room for disagreement, which is why I leave room for error on the qual test (and its scoring).
 

Drwho10

Mister Gank Yo Hits
Contributor
Joined
Jan 11, 2016
Messages
12,350
Reaction score
20,568
Points
1,388
Age
31
Location
Long Beach, CA
Gender
Male
Hi everyone, I'm new to the forums (and a relatively new requester). I have been posting HITs for writing question–answer pairs about English sentences, as well as answering them. Several people working on my task recommended I check out MTurk Crowd and introduce myself.

Since I can't post links yet, the HIT type IDs for my tasks are 36SUH4ZPJUVEFKCRRRIAVB1LGOZ705 (question–answer writing) and 3XRW87W7OXAP1BEXLSFQFDKLIPNTQ0 (question answering).

I'd be happy to receive any feedback on the task design, etc. I will also post about it in the daily work thread and I'm happy to have any more people join onto the task.. :) there is a bit of a backlog on the question answering task right now so there are lots of HITs available.

I will not be able to make huge changes to the design any time soon, but feedback would be really valuable for designing future tasks.

Thanks, and nice to meet you all!
Julian
Good to see you in the forum. Your hits have been quite enjoyable and I look forward to doing more of them. The instructions are pretty straight forward and the pay is solid or could use a slight bump. Thanks for being such a communicative requester.

This is Kent M. from the emails by the way :wave:
 

Julian Michael

New Member
Requester
Joined
Mar 28, 2017
Messages
18
Reaction score
45
Points
13
Age
30
Gender
Male
This is Kent M. from the emails by the way :wave:
Hi again! You were right, coming here was a good choice—I've already gotten some helpful feedback and once the remaining batch of questions is all answered I will kick off the task with improvements based on everyone's suggestions.
 

Drwho10

Mister Gank Yo Hits
Contributor
Joined
Jan 11, 2016
Messages
12,350
Reaction score
20,568
Points
1,388
Age
31
Location
Long Beach, CA
Gender
Male
Hi again! You were right, coming here was a good choice—I've already gotten some helpful feedback and once the remaining batch of questions is all answered I will kick off the task with improvements based on everyone's suggestions.
Good to hear. This is one of the best forums in the mturk community I'm sure you'll get even better quality work now.
 

CatWhiskers

Looking In Your Fridge
Joined
Jan 27, 2016
Messages
861
Reaction score
1,897
Points
343
Location
Cat Tree
I think you probably misunderstood something. I can't think of where in the task instructions you might have thought it said not to include redundant answers. The good thing is, you can retake the qualification if you still want to.
When? I took it and got 57 so I must have misunderstood something, but I see no way to retake it.
 

T. Leela

Oh Lord...
Contributor
Joined
May 12, 2016
Messages
11,050
Reaction score
14,046
Points
1,388
Location
Colorado
Gender
Female
I've enjoyed these hits, I did the very first batch, and a few of the next, just missed the qual but am going to try again.
Right now actually.
Thank you :)
 
  • Like
Reactions: Julian Michael

T. Leela

Oh Lord...
Contributor
Joined
May 12, 2016
Messages
11,050
Reaction score
14,046
Points
1,388
Location
Colorado
Gender
Female
sorry wrong thread :oops:
 
Last edited:

sryan66611

Well-Known Member
Contributor
Joined
Feb 13, 2016
Messages
3,412
Reaction score
5,893
Points
838
Age
36
Gender
Male
Just a heads up... People doing the questions if you change the name around or leave it out and call it "what" or other generalization as per the instructions - In the what since? --> <Invalid>The question writer simply replaced a word with "what" instead of using it to form a proper English question.

I have seen this a lot the past few days and just wanted to let everyone know so they don't lose their qual.
 
Last edited: