Rep Present Question–answer pairs and question answering

Julian Michael

New Member
Requester
Joined
Mar 28, 2017
Messages
18
Reaction score
45
Points
13
Age
30
Gender
Male
Hi everyone, I'm new to the forums (and a relatively new requester). I have been posting HITs for writing question–answer pairs about English sentences, as well as answering them. Several people working on my task recommended I check out MTurk Crowd and introduce myself.

Since I can't post links yet, the HIT type IDs for my tasks are 36SUH4ZPJUVEFKCRRRIAVB1LGOZ705 (question–answer writing) and 3XRW87W7OXAP1BEXLSFQFDKLIPNTQ0 (question answering).

I'd be happy to receive any feedback on the task design, etc. I will also post about it in the daily work thread and I'm happy to have any more people join onto the task.. :) there is a bit of a backlog on the question answering task right now so there are lots of HITs available.

I will not be able to make huge changes to the design any time soon, but feedback would be really valuable for designing future tasks.

Thanks, and nice to meet you all!
Julian
 

electrolyte

The Ghost of MTurk Past
Contributor
Joined
Jan 10, 2016
Messages
19,184
Reaction score
45,978
Points
1,313
Title: Answer simple questions about a sentence | PANDA
Worker: Preview | Accept | Requester
Requester: Julian Michael [A3AA2VKV87R6PG] (Contact)
TO 1: Not Available
TO 2: Not Available
Description: Given a sentence and several questions about it,
highlight the part of the sentence that answers each question,
and mark questions that are invalid or redundant.
Time: 10 minutes
HITs Available: 346
Reward: $0.08
Qualifications: Question answering test score (%) is not less than 80; Question answering agreement rate (%) is not less than 70; HIT approval rate (%) is not less than 95; Location is US;

Title: Write question-answer pairs about a sentence's meaning | PANDA
Worker: Preview | Accept | Requester
Requester: Julian Michael [A3AA2VKV87R6PG] (Contact)
TO 1: Not Available
TO 2: Not Available
Description: Given a sentence and some words from that sentence,
write questions and answers involving each word.
Write more question-answer pairs for increasing bonuses!
Time: 10 minutes
HITs Available: 145
Reward: $0.20
Qualifications: Question-answer generation accuracy (%) is not less than 75; HIT approval rate (%) is not less than 95; Location is US;

My export doesn't have working TO links, so for reference:
TO 1 - https://turkopticon.ucsd.edu/A3AA2VKV87R6PG
TO 2 - https://turkopticon.info/requesters/A3AA2VKV87R6PG
 
Last edited:
  • Like
Reactions: kryss

electrolyte

The Ghost of MTurk Past
Contributor
Joined
Jan 10, 2016
Messages
19,184
Reaction score
45,978
Points
1,313
Hi! I see a qualification test to get a "Question answering test" score. How are you granting the "Question-answer generation accuracy" qualification?
 

Julian Michael

New Member
Requester
Joined
Mar 28, 2017
Messages
18
Reaction score
45
Points
13
Age
30
Gender
Male
Hi! I see a qualification test to get a "Question answering test" score. How are you granting the "Question-answer generation accuracy" qualification?
It's auto-granted. This seems to be a source of confusion among workers. Should I write that in the task description? I figured this is how most requesters would do it..
 

Julian Michael

New Member
Requester
Joined
Mar 28, 2017
Messages
18
Reaction score
45
Points
13
Age
30
Gender
Male
Maybe there is some nice little acronym like "(AG)" I can put in the qual name to show that it's auto-granted. Is there a common way of doing this?
 

electrolyte

The Ghost of MTurk Past
Contributor
Joined
Jan 10, 2016
Messages
19,184
Reaction score
45,978
Points
1,313
It's auto-granted. This seems to be a source of confusion among workers. Should I write that in the task description? I figured this is how most requesters would do it..
Oh, that would probably be helpful. It's not too common to have autogranted quals, so most workers will be either looking for a qualification test or a batch of test HITs, or assuming the requester grants it after reviewing some completed HITs.

I don't think anyone would know what "AG" means but you could put "autogranted qual" in the title.
 
  • Like
Reactions: kryss

Julian Michael

New Member
Requester
Joined
Mar 28, 2017
Messages
18
Reaction score
45
Points
13
Age
30
Gender
Male
Oh, that would probably be helpful. It's not too common to have autogranted quals, so most workers will be either looking for a qualification test or a batch of test HITs, or assuming the requester grants it after reviewing some completed HITs.

I don't think anyone would know what "AG" means but you could put "autogranted qual" in the title.
Ah I see, that makes sense. Ok, that is super helpful, thanks!
 

Jharkan

Internets EXPERT
Contributor
Joined
Jul 9, 2016
Messages
11,873
Reaction score
20,418
Points
1,288
Location
Earth
Gender
Male
I thought I had aced the qual test, but scored lower than 30.
what?
 
  • Like
Reactions: heeheehahaha

Julian Michael

New Member
Requester
Joined
Mar 28, 2017
Messages
18
Reaction score
45
Points
13
Age
30
Gender
Male
By the way.. how do I post one of those nicely formatted links to a HIT?
 

Julian Michael

New Member
Requester
Joined
Mar 28, 2017
Messages
18
Reaction score
45
Points
13
Age
30
Gender
Male
I thought I had aced the qual test, but scored lower than 30.
what?
It's designed to get at the issues where the task instructions might contradict your intuition. It's possible that there is some ambiguity in the instructions though. Are there any parts of it that you find confusing?
 

sryan66611

Well-Known Member
Contributor
Joined
Feb 13, 2016
Messages
3,412
Reaction score
5,893
Points
838
Age
36
Gender
Male
J @Julian Michael ur talking about this right ?/

Title: Answer simple questions about a sentence | PANDA
Worker: Preview | Accept | Requester
Requester: Julian Michael [A3AA2VKV87R6PG] (Contact)
TO: [Pay: 4.80] [Fair: 4.60] [Comm: 5.00] [Fast: 5.00] [Reviews: 6] [ToS: 0]
Description:
Given a sentence and several questions about it,
highlight the part of the sentence that answers each question,
and mark questions that are invalid or redundant.
Time: 10 minutes
HITs Available: 350
Reward: $0.08
Qualifications: Question answering test score (%) GreaterThanOrEqualTo 80; Question answering agreement rate (%) GreaterThanOrEqualTo 70; HIT approval rate (%) GreaterThanOrEqualTo 95; Location EqualTo US;
 

Jharkan

Internets EXPERT
Contributor
Joined
Jul 9, 2016
Messages
11,873
Reaction score
20,418
Points
1,288
Location
Earth
Gender
Male
It's designed to get at the issues where the task instructions might contradict your intuition. It's possible that there is some ambiguity in the instructions though. Are there any parts of it that you find confusing?
Yea, the questions' answers suggest that redundant questions are not wanted.
Given the answers I submitted, and the score I received, I can only guess that I misinterpreted the present material.

And there is no way to access the other hit that doesn't require the test, that I can see.
 

Julian Michael

New Member
Requester
Joined
Mar 28, 2017
Messages
18
Reaction score
45
Points
13
Age
30
Gender
Male
J @Julian Michael ur talking about this right ?/

Title: Answer simple questions about a sentence | PANDA
Worker: Preview | Accept | Requester
Requester: Julian Michael [A3AA2VKV87R6PG] (Contact)
TO: [Pay: 4.80] [Fair: 4.60] [Comm: 5.00] [Fast: 5.00] [Reviews: 6] [ToS: 0]
Description:
Given a sentence and several questions about it,
highlight the part of the sentence that answers each question,
and mark questions that are invalid or redundant.
Time: 10 minutes
HITs Available: 350
Reward: $0.08
Qualifications: Question answering test score (%) GreaterThanOrEqualTo 80; Question answering agreement rate (%) GreaterThanOrEqualTo 70; HIT approval rate (%) GreaterThanOrEqualTo 95; Location EqualTo US;
Yep, that's it. And thanks for the link!
 

Julian Michael

New Member
Requester
Joined
Mar 28, 2017
Messages
18
Reaction score
45
Points
13
Age
30
Gender
Male
Yea, the questions' answers suggest that redundant questions are not wanted.
Given the answers I submitted, and the score I received, I can only guess that I misinterpreted the present material.

And there is no way to access the other hit that doesn't require the test, that I can see.
Can you not preview the HIT? I thought I set it to allow previewing without qualifications. Also it has an "accuracy" qual requirement which you have to request and which will be auto-granted.

It all worked as expected when I tested it on the sandbox. Unfortunately I can't test it on production because Amazon rejected my worker account registration.
 
  • Like
Reactions: sryan66611

Jharkan

Internets EXPERT
Contributor
Joined
Jul 9, 2016
Messages
11,873
Reaction score
20,418
Points
1,288
Location
Earth
Gender
Male
Can you not preview the HIT? I thought I set it to allow previewing without qualifications. Also it has an "accuracy" qual requirement which you have to request and which will be auto-granted.

It all worked as expected when I tested it on the sandbox. Unfortunately I can't test it on production because Amazon rejected my worker account registration.
oh, er... yea, requesting it works.
Usually I don't hit the request, because nothing ever happens - it tells us that it emails the requester.
the end.
never hear from them again.
 

sryan66611

Well-Known Member
Contributor
Joined
Feb 13, 2016
Messages
3,412
Reaction score
5,893
Points
838
Age
36
Gender
Male
Can you not preview the HIT? I thought I set it to allow previewing without qualifications. Also it has an "accuracy" qual requirement which you have to request and which will be auto-granted.

It all worked as expected when I tested it on the sandbox. Unfortunately I can't test it on production because Amazon rejected my worker account registration.
Everything works as expected I took the qual yesterday and you can preview your hits without having the qual.
 
  • Like
Reactions: kryss

Julian Michael

New Member
Requester
Joined
Mar 28, 2017
Messages
18
Reaction score
45
Points
13
Age
30
Gender
Male
oh, er... yea, requesting it works.
Usually I don't hit the request, because nothing ever happens - it tells us that it emails the requester.
the end.
never hear from them again.
Makes sense. I'll add some clarifications in the obvious places and have those on new uploads.

Everything works as expected I took the qual yesterday and you can preview your hits without having the qual.
Ok, good to have that clarified, thanks!

And thank you too :)
 

Jharkan

Internets EXPERT
Contributor
Joined
Jul 9, 2016
Messages
11,873
Reaction score
20,418
Points
1,288
Location
Earth
Gender
Male
The two hits completely contradict each other, but I can't even talk about it without discussing the qualification test on the one task.