In evaluation page , it's written
\n\nThere may be up to five labels for long answers, and more for short. If no answer applies, leave the prediction blank/null.
In Dieter's implementation or a modified version by cfiken, it looks like long_label is assumed to have ONLY one label, but short_label have multiple labels.
Question 1. What's the actual situation for long labels? @christofhenkel , @kentaronakanishi , @kashnitsky , @boliu0 , @zhaomeng1126 , do you use multiple long labels for your local CV?
\n\nFurthermore, when I use v1.0-simplified_nq-dev-all.jsonl -- in order to do CV, after using simplify_nq_example from text_utils.py, i see something like at the bottom of this post. There are multiple annotations, and each annotation has a single long_answer and multiple short_answer.
Question 2. Suppose we do have multiple long labels for this Kaggle competition. Are the long_lables and short_labels are somehow independent? Use the example below, suppose we give long_prediction as 22:341 which matches the 1st annotation, but short_prediction as 351:353 which matches the 2nd short_answer in the 2nd annotation, but not match the short_answer in the 1st annotation. Does this count a correct prediction or not?
I hope I don't make any stupid mistake here, but I am really confused by the metric. @philculliton, could you confirm clearly how is the metric calculated, including how annotations are converted to the actual labels used for this competition, please ....
\n\n> \"question_text\": \"who wrote the song photograph by ringo starr\",\n \"example_id\": -8366545547296627039,
\n \"annotations\": [\n {\n \"annotation_id\": 647276088892962831,\n \"long_answer\": {\n \"candidate_index\": 0,\n \"end_token\": 341,\n \"start_token\": 22\n },\n \"short_answers\": [],\n \"yes_no_answer\": \"NONE\"\n },\n {\n \"annotation_id\": 10083697748159579777,\n \"long_answer\": {\n \"candidate_index\": 36,\n \"end_token\": 478,\n \"start_token\": 341\n },\n \"short_answers\": [\n {\n \"end_token\": 353,\n \"start_token\": 351\n },\n {\n \"end_token\": 373,\n \"start_token\": 371\n }\n ],\n \"yes_no_answer\": \"NONE\"\n },\n {\n \"annotation_id\": 6354110109716954275,\n \"long_answer\": {\n \"candidate_index\": 0,\n \"end_token\": 341,\n \"start_token\": 22\n },\n \"short_answers\": [\n {\n \"end_token\": 126,\n \"start_token\": 124\n },\n {\n \"end_token\": 129,\n \"start_token\": 127\n }\n ],\n \"yes_no_answer\": \"NONE\"\n },\n {\n \"annotation_id\": 7396078045607493655,\n \"long_answer\": {\n \"candidate_index\": 36,\n \"end_token\": 478,\n \"start_token\": 341\n },\n \"short_answers\": [],\n \"yes_no_answer\": \"NONE\"\n },\n {\n \"annotation_id\": 9210770612718242890,\n \"long_answer\": {\n \"candidate_index\": 36,\n \"end_token\": 478,\n \"start_token\": 341\n },\n \"short_answers\": [\n {\n \"end_token\": 353,\n \"start_token\": 351\n },\n {\n \"end_token\": 373,\n \"start_token\": 371\n }\n ],\n \"yes_no_answer\": \"NONE\"\n }\n ]
In evaluation page , it's written
\n\nThere may be up to five labels for long answers, and more for short. If no answer applies, leave the prediction blank/null.
In Dieter's implementation or a modified version by cfiken, it looks like long_label is assumed to have ONLY one label, but short_label have multiple labels.
Question 1. What's the actual situation for long labels? @christofhenkel , @kentaronakanishi , @kashnitsky , @boliu0 , @zhaomeng1126 , do you use multiple long labels for your local CV?
\n\nFurthermore, when I use v1.0-simplified_nq-dev-all.jsonl -- in order to do CV, after using simplify_nq_example from text_utils.py, i see something like at the bottom of this post. There are multiple annotations, and each annotation has a single long_answer and multiple short_answer.
Question 2. Suppose we do have multiple long labels for this Kaggle competition. Are the long_lables and short_labels are somehow independent? Use the example below, suppose we give long_prediction as 22:341 which matches the 1st annotation, but short_prediction as 351:353 which matches the 2nd short_answer in the 2nd annotation, but not match the short_answer in the 1st annotation. Does this count a correct prediction or not?
I hope I don't make any stupid mistake here, but I am really confused by the metric. @philculliton, could you confirm clearly how is the metric calculated, including how annotations are converted to the actual labels used for this competition, please ....
\n\n> \"question_text\": \"who wrote the song photograph by ringo starr\",\n \"example_id\": -8366545547296627039,
\n \"annotations\": [\n {\n \"annotation_id\": 647276088892962831,\n \"long_answer\": {\n \"candidate_index\": 0,\n \"end_token\": 341,\n \"start_token\": 22\n },\n \"short_answers\": [],\n \"yes_no_answer\": \"NONE\"\n },\n {\n \"annotation_id\": 10083697748159579777,\n \"long_answer\": {\n \"candidate_index\": 36,\n \"end_token\": 478,\n \"start_token\": 341\n },\n \"short_answers\": [\n {\n \"end_token\": 353,\n \"start_token\": 351\n },\n {\n \"end_token\": 373,\n \"start_token\": 371\n }\n ],\n \"yes_no_answer\": \"NONE\"\n },\n {\n \"annotation_id\": 6354110109716954275,\n \"long_answer\": {\n \"candidate_index\": 0,\n \"end_token\": 341,\n \"start_token\": 22\n },\n \"short_answers\": [\n {\n \"end_token\": 126,\n \"start_token\": 124\n },\n {\n \"end_token\": 129,\n \"start_token\": 127\n }\n ],\n \"yes_no_answer\": \"NONE\"\n },\n {\n \"annotation_id\": 7396078045607493655,\n \"long_answer\": {\n \"candidate_index\": 36,\n \"end_token\": 478,\n \"start_token\": 341\n },\n \"short_answers\": [],\n \"yes_no_answer\": \"NONE\"\n },\n {\n \"annotation_id\": 9210770612718242890,\n \"long_answer\": {\n \"candidate_index\": 36,\n \"end_token\": 478,\n \"start_token\": 341\n },\n \"short_answers\": [\n {\n \"end_token\": 353,\n \"start_token\": 351\n },\n {\n \"end_token\": 373,\n \"start_token\": 371\n }\n ],\n \"yes_no_answer\": \"NONE\"\n }\n ]
Question 1.\nThere is at most 1 long answer as for the training data, but more (at most five answers) for test data.\nSome of us are using the split of training data in order to compute CV, so the implementation you mentioned is assumed to have only 1 long answer (especially for my implementation).
\n\nMultiple annotations for dev dataset is made by the process of Natural Questions. There are 5 annotator for a sample and each annotation by them is one of the multiple annotation.\nAbout Natural Questions, you can check official paper for detail.
Question 2.\nI think it is computed independently but I don't have confidence.\nindependently means that it is ok to say true positive if a predicted short answer is one of the gold short answer in all multiple annotations.
Thanks for your clarification, especially for Some of us are using the split of training data in order to compute CV
Question 1.\nThere is at most 1 long answer as for the training data, but more (at most five answers) for test data.\nSome of us are using the split of training data in order to compute CV, so the implementation you mentioned is assumed to have only 1 long answer (especially for my implementation).
\n\nMultiple annotations for dev dataset is made by the process of Natural Questions. There are 5 annotator for a sample and each annotation by them is one of the multiple annotation.\nAbout Natural Questions, you can check official paper for detail.
Question 2.\nI think it is computed independently but I don't have confidence.\nindependently means that it is ok to say true positive if a predicted short answer is one of the gold short answer in all multiple annotations.
Thanks for your clarification, especially for Some of us are using the split of training data in order to compute CV