Another snippet from the article describing the dataset explaining the annotation process (https://ai.google/research/pubs/pub47761.pdf):
\n\n\n", "messages": [ { "id": "667282", "postDate": "11/07/2019 04:07:36", "content": "Annotation is performed using a custom annotation interface, by a pool of around 50 annotators, with an average annotation time of 80 seconds.\n The guidelines and tooling divide the annotation task into three conceptual stages, where all three stages are completed by a single annotator in succession. The decision flow through these is illustrated in Figure 2 and the instructions given to annotators are summarized below.\n Question Identification: contributors determine whether the given question is good or bad.\n A good question is a fact-seeking question that can be answered with an entity or explanation.\n A bad question is ambigous, incomprehensible, dependent on clear false presuppositions, opinionseeking, or not clearly a request for factual information. Annotators must make this judgment solely by the content of the question; they are not yet shown the Wikipedia page.\n Long Answer Identification: for good questions only, annotators select the earliest HTML bounding box containing enough information for a reader to completely infer the answer to the question. Bounding boxes can be paragraphs, tables, list items, or whole lists. Alternatively, annotators mark ‘no answer’ if the page does not answer the question, or if the information is present but not contained in a single one of the allowed elements.\n Short Answer Identification: for examples with long answers, annotators select the entity or set of entities within the long answer that answer the question. Alternatively, annotators can flag that the short answer is ‘yes’, ‘no’, or they can flag that no short answer is possible.
\n
Another snippet from the article describing the dataset explaining the annotation process (https://ai.google/research/pubs/pub47761.pdf):
\n\n\n", "rawMarkdown": "Another snippet from the article describing the dataset explaining the annotation process (https://ai.google/research/pubs/pub47761.pdf):\n> Annotation is performed using a custom annotation interface, by a pool of around 50 annotators, with an average annotation time of 80 seconds.\nThe guidelines and tooling divide the annotation task into three conceptual stages, where all three stages are completed by a single annotator in succession. The decision flow through these is illustrated in Figure 2 and the instructions given to annotators are summarized below.\n**Question Identification**: contributors determine whether the given question is good or bad.\nA good question is a fact-seeking question that can be answered with an entity or explanation.\nA bad question is ambigous, incomprehensible, dependent on clear false presuppositions, opinionseeking, or not clearly a request for factual information. Annotators must make this judgment solely by the content of the question; they are not yet shown the Wikipedia page.\n**Long Answer Identification**: for good questions only, annotators select the earliest HTML bounding box containing enough information for a reader to completely infer the answer to the question. Bounding boxes can be paragraphs, tables, list items, or whole lists. Alternatively, annotators mark ‘no answer’ if the page does not answer the question, or if the information is present but not contained in a single one of the allowed elements.\n**Short Answer Identification**: for examples with long answers, annotators select the entity or set of entities within the long answer that answer the question. Alternatively, annotators can flag that the short answer is ‘yes’, ‘no’, or they can flag that no short answer is possible.", "votes": null }, { "id": "698251", "postDate": "12/19/2019 01:36:52", "content": "Annotation is performed using a custom annotation interface, by a pool of around 50 annotators, with an average annotation time of 80 seconds.\n The guidelines and tooling divide the annotation task into three conceptual stages, where all three stages are completed by a single annotator in succession. The decision flow through these is illustrated in Figure 2 and the instructions given to annotators are summarized below.\n Question Identification: contributors determine whether the given question is good or bad.\n A good question is a fact-seeking question that can be answered with an entity or explanation.\n A bad question is ambigous, incomprehensible, dependent on clear false presuppositions, opinionseeking, or not clearly a request for factual information. Annotators must make this judgment solely by the content of the question; they are not yet shown the Wikipedia page.\n Long Answer Identification: for good questions only, annotators select the earliest HTML bounding box containing enough information for a reader to completely infer the answer to the question. Bounding boxes can be paragraphs, tables, list items, or whole lists. Alternatively, annotators mark ‘no answer’ if the page does not answer the question, or if the information is present but not contained in a single one of the allowed elements.\n Short Answer Identification: for examples with long answers, annotators select the entity or set of entities within the long answer that answer the question. Alternatively, annotators can flag that the short answer is ‘yes’, ‘no’, or they can flag that no short answer is possible.
\n
Thanks for sharing @thedrcat
", "rawMarkdown": "Thanks for sharing @thedrcat", "votes": null } ], "comments": [ { "id": 698251, "author_name": "rohitagarwal", "author_url": "", "post_date": "12/19/2019 01:36:52", "content": "Thanks for sharing @thedrcat
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "667282": "Another snippet from the article describing the dataset explaining the annotation process (https://ai.google/research/pubs/pub47761.pdf):\n> Annotation is performed using a custom annotation interface, by a pool of around 50 annotators, with an average annotation time of 80 seconds.\nThe guidelines and tooling divide the annotation task into three conceptual stages, where all three stages are completed by a single annotator in succession. The decision flow through these is illustrated in Figure 2 and the instructions given to annotators are summarized below.\n**Question Identification**: contributors determine whether the given question is good or bad.\nA good question is a fact-seeking question that can be answered with an entity or explanation.\nA bad question is ambigous, incomprehensible, dependent on clear false presuppositions, opinionseeking, or not clearly a request for factual information. Annotators must make this judgment solely by the content of the question; they are not yet shown the Wikipedia page.\n**Long Answer Identification**: for good questions only, annotators select the earliest HTML bounding box containing enough information for a reader to completely infer the answer to the question. Bounding boxes can be paragraphs, tables, list items, or whole lists. Alternatively, annotators mark ‘no answer’ if the page does not answer the question, or if the information is present but not contained in a single one of the allowed elements.\n**Short Answer Identification**: for examples with long answers, annotators select the entity or set of entities within the long answer that answer the question. Alternatively, annotators can flag that the short answer is ‘yes’, ‘no’, or they can flag that no short answer is possible.", "698251": "Thanks for sharing @thedrcat" }, "source": "meta" }