Thanks kaggle for holding this wonderful competition. Big thanks to my awesome teammate @ewrfcas and @leolemon214. It's little pity for dropping from public LB 3th place, expect fetching gold medal at future.
\n\nWe carefully read other top solutions, and still puzzling for the 4% drop of private LB, can someone help to figure out the reason for this shaking?
\n\nBelow are valid part of our solution, all the following experiments are mainly performed on offline dev containing 1600 examples, and some results have been verified in public LB.
\n\n| No | Technique | Pros | Cons | Effect |\n| ---- | --------------------------------- | ------------------------------------------------------------ | --------------------------------- | ------------------------------------- |\n| 1 | TF-IDF paragraph selection | Shorten doc resulting faster inference speed and better accuracy | May loss some context information | - dev f1 +1.8%,
- public LB f1 -1% |\n| 2 | Sample negative features till 1:1 | Balance pos and neg | Cause longer training time | dev f1 +2.248% |\n| 3 | Multi-process preprocessing | Accelerate preprocessing, especially on training data | Require multi-core CPU | xN faster (with N processes) |
| No | Model Architecture | Idea | Performance |\n| ---- | -------------------------------------------------- | ------------------------------------------------------------ | -------------------- |\n| 1 | Roberta-Large joint with long/short span extractor | 1. Jointly model:
- answer type
- long span
- short span
2. Output topk start/end logits/index | dev f1 63.986% |\n| 2 | Albert-xxlarge joint with short span extractor | Jointly model:
- answer type
- short span | def short-f1 69.364% |
All of above model architectures were pretrained on SQuAD dataset by ourselves.
\n\n| No | Trick | Effect |\n| ---- | ------------------------------------------------------------ | ---------------------------------- |\n| 1 | If answer_type is yes/no, output yes/no rather than short span | public LB f1 +6% |\n| 2 | 1. If answer_type is short, output long span and short span
2. If answer_type is long, output long span only
3. If answer_type is none, output neither long span nor short span | public LB f1 +8% |\n| 3 | Choose the best long/short answer pair from topk * topk kind of long/short answer combinations | dev f1 +0.435% |\n| 4 | long_score = summary.long_span_score - summary.long_cls_score - summary.answer_type_logits[0]short_score = summary.short_span_score - summary.short_cls_score - summary.answer_type_logits[0] | - dev f1 +2.12%
- public LB +2% |\n| 5 | Increase long [CLS] logits multiplier threshold to increase null long answer | dev long-f1 +3.491% |\n| 6 | Decrease short answer_type logits divisor threshold to increase null short answer | dev short-f1 ? |
| No | Idea | Effect |\n| ---- | ------------------------------------------------------------ | ------------------------------------------------------------ |\n| 1 | For long answer, We vote long answers of 2 Roberta-Large joint with long/short span extractor models | dev long-f1 +3.341% |\n| 2 | For short answer, use step 1 result to locate predicted long answer candidate as input, We vote short answers of 2 Roberta-Large joint with long/short span extractor models and 4 Albert-xxlarge joint with short span extractor models | - dev short-f1 +2.842%
- dev f1 67.569%, +2.635%
- public LB 71%, +5%
- private LB 67% |
Thanks kaggle for holding this wonderful competition. Big thanks to my awesome teammate @ewrfcas and @leolemon214. It's little pity for dropping from public LB 3th place, expect fetching gold medal at future.
\n\nWe carefully read other top solutions, and still puzzling for the 4% drop of private LB, can someone help to figure out the reason for this shaking?
\n\nBelow are valid part of our solution, all the following experiments are mainly performed on offline dev containing 1600 examples, and some results have been verified in public LB.
\n\n| No | Technique | Pros | Cons | Effect |\n| ---- | --------------------------------- | ------------------------------------------------------------ | --------------------------------- | ------------------------------------- |\n| 1 | TF-IDF paragraph selection | Shorten doc resulting faster inference speed and better accuracy | May loss some context information | - dev f1 +1.8%,
- public LB f1 -1% |\n| 2 | Sample negative features till 1:1 | Balance pos and neg | Cause longer training time | dev f1 +2.248% |\n| 3 | Multi-process preprocessing | Accelerate preprocessing, especially on training data | Require multi-core CPU | xN faster (with N processes) |
| No | Model Architecture | Idea | Performance |\n| ---- | -------------------------------------------------- | ------------------------------------------------------------ | -------------------- |\n| 1 | Roberta-Large joint with long/short span extractor | 1. Jointly model:
- answer type
- long span
- short span
2. Output topk start/end logits/index | dev f1 63.986% |\n| 2 | Albert-xxlarge joint with short span extractor | Jointly model:
- answer type
- short span | def short-f1 69.364% |
All of above model architectures were pretrained on SQuAD dataset by ourselves.
\n\n| No | Trick | Effect |\n| ---- | ------------------------------------------------------------ | ---------------------------------- |\n| 1 | If answer_type is yes/no, output yes/no rather than short span | public LB f1 +6% |\n| 2 | 1. If answer_type is short, output long span and short span
2. If answer_type is long, output long span only
3. If answer_type is none, output neither long span nor short span | public LB f1 +8% |\n| 3 | Choose the best long/short answer pair from topk * topk kind of long/short answer combinations | dev f1 +0.435% |\n| 4 | long_score = summary.long_span_score - summary.long_cls_score - summary.answer_type_logits[0]short_score = summary.short_span_score - summary.short_cls_score - summary.answer_type_logits[0] | - dev f1 +2.12%
- public LB +2% |\n| 5 | Increase long [CLS] logits multiplier threshold to increase null long answer | dev long-f1 +3.491% |\n| 6 | Decrease short answer_type logits divisor threshold to increase null short answer | dev short-f1 ? |
| No | Idea | Effect |\n| ---- | ------------------------------------------------------------ | ------------------------------------------------------------ |\n| 1 | For long answer, We vote long answers of 2 Roberta-Large joint with long/short span extractor models | dev long-f1 +3.341% |\n| 2 | For short answer, use step 1 result to locate predicted long answer candidate as input, We vote short answers of 2 Roberta-Large joint with long/short span extractor models and 4 Albert-xxlarge joint with short span extractor models | - dev short-f1 +2.842%
- dev f1 67.569%, +2.635%
- public LB 71%, +5%
- private LB 67% |
Wow! You guys really put the effort on it! Congratulations.
", "rawMarkdown": "Wow! You guys really put the effort on it! Congratulations.", "votes": null }, { "id": "731983", "postDate": "01/29/2020 10:32:29", "content": "Congratulations!!\nThanks for sharing your Approach & Code @mikelkl
", "rawMarkdown": "Congratulations!!\nThanks for sharing your Approach & Code @mikelkl", "votes": null }, { "id": "732221", "postDate": "01/29/2020 15:29:57", "content": "Thanks for sharing!!😄 👍
", "rawMarkdown": "Thanks for sharing!!😄 👍", "votes": null }, { "id": "732361", "postDate": "01/29/2020 18:11:13", "content": "@mikelkl My guess is that you over-relied on the answer type logits in postprocessing.. refer to my comments in my solution\nhttps://www.kaggle.com/c/tensorflow2-question-answering/discussion/127241
\n\n\n", "rawMarkdown": "mikelkl My guess is that you over-relied on the answer type logits in postprocessing.. refer to my comments in my solution\nhttps://www.kaggle.com/c/tensorflow2-question-answering/discussion/127241\n\n> I still have a very strong feeling that the \"joint\" part of bert-joint might be of little use, since we've already known that:\n1) BERT-like structure is poor at passage ranking, and to make it better we need passages at least as many as in MS-MARCO\n2) We only have like 1-3% YES/NO in our training data. Very unbalanced.\nBased on my inspection and verification experiment this might well be the case, which means the answer type classifier might just be reduced to a question type classifier (a much easier task for BERT to see), which would be a bad indicator of what type of answer the passage contains. It might not be a good idea to include it in training in the first place, and it'd be a disaster if you over-rely on the answer type logits for post-processing (since it's basically overfitting to Dev or public LB).", "votes": null }, { "id": "732901", "postDate": "01/30/2020 12:47:48", "content": "I still have a very strong feeling that the \"joint\" part of bert-joint might be of little use, since we've already known that:\n 1) BERT-like structure is poor at passage ranking, and to make it better we need passages at least as many as in MS-MARCO\n 2) We only have like 1-3% YES/NO in our training data. Very unbalanced.\n Based on my inspection and verification experiment this might well be the case, which means the answer type classifier might just be reduced to a question type classifier (a much easier task for BERT to see), which would be a bad indicator of what type of answer the passage contains. It might not be a good idea to include it in training in the first place, and it'd be a disaster if you over-rely on the answer type logits for post-processing (since it's basically overfitting to Dev or public LB).
\n
Thks for ur reply, very insightful idea. We did touch answer type logits at least 3 times on postprocessing stage, will check it. If so, do u think train an extra yes/no model with balanced resampling data is a good idea?
", "rawMarkdown": "Thks for ur reply, very insightful idea. We did touch answer type logits at least 3 times on postprocessing stage, will check it. If so, do u think train an extra yes/no model with balanced resampling data is a good idea?", "votes": null }, { "id": "733224", "postDate": "01/30/2020 21:01:30", "content": "@mikelkl That was exactly what I tried - an extra YES/NO verifier trained with ALBERT-xxlarge with only the YES/NO questions. Slight improvement on Dev (~1 pt) but not on private test. I'd say it's not the most effective way due to the lack of data (1k to 10k samples). Might be better to make use of some external knowledge base rather than NQ alone.
", "rawMarkdown": "mikelkl That was exactly what I tried - an extra YES/NO verifier trained with ALBERT-xxlarge with only the YES/NO questions. Slight improvement on Dev (~1 pt) but not on private test. I'd say it's not the most effective way due to the lack of data (1k to 10k samples). Might be better to make use of some external knowledge base rather than NQ alone.", "votes": null } ], "comments": [ { "id": 731978, "author_name": "brunhs", "author_url": "", "post_date": "01/29/2020 10:25:54", "content": "Wow! You guys really put the effort on it! Congratulations.
", "votes": null, "replies": [] }, { "id": 731983, "author_name": "veeralakrishna", "author_url": "", "post_date": "01/29/2020 10:32:29", "content": "Congratulations!!\nThanks for sharing your Approach & Code @mikelkl
", "votes": null, "replies": [] }, { "id": 732221, "author_name": "mashlyn", "author_url": "", "post_date": "01/29/2020 15:29:57", "content": "Thanks for sharing!!😄 👍
", "votes": null, "replies": [] }, { "id": 732361, "author_name": "siriuself", "author_url": "", "post_date": "01/29/2020 18:11:13", "content": "@mikelkl My guess is that you over-relied on the answer type logits in postprocessing.. refer to my comments in my solution\nhttps://www.kaggle.com/c/tensorflow2-question-answering/discussion/127241
\n\n\n", "votes": null, "replies": [ { "id": 732901, "author_name": "mikelkl", "author_url": "", "post_date": "01/30/2020 12:47:48", "content": "I still have a very strong feeling that the \"joint\" part of bert-joint might be of little use, since we've already known that:\n 1) BERT-like structure is poor at passage ranking, and to make it better we need passages at least as many as in MS-MARCO\n 2) We only have like 1-3% YES/NO in our training data. Very unbalanced.\n Based on my inspection and verification experiment this might well be the case, which means the answer type classifier might just be reduced to a question type classifier (a much easier task for BERT to see), which would be a bad indicator of what type of answer the passage contains. It might not be a good idea to include it in training in the first place, and it'd be a disaster if you over-rely on the answer type logits for post-processing (since it's basically overfitting to Dev or public LB).
\n
Thks for ur reply, very insightful idea. We did touch answer type logits at least 3 times on postprocessing stage, will check it. If so, do u think train an extra yes/no model with balanced resampling data is a good idea?
", "votes": null, "replies": [] }, { "id": 733224, "author_name": "siriuself", "author_url": "", "post_date": "01/30/2020 21:01:30", "content": "@mikelkl That was exactly what I tried - an extra YES/NO verifier trained with ALBERT-xxlarge with only the YES/NO questions. Slight improvement on Dev (~1 pt) but not on private test. I'd say it's not the most effective way due to the lack of data (1k to 10k samples). Might be better to make use of some external knowledge base rather than NQ alone.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "731886": "[Code for Solution](https://github.com/mikelkl/TF2-QA)\n\nThanks kaggle for holding this wonderful competition. Big thanks to my awesome teammate [@ewrfcas](https://www.kaggle.com/ewrfcas) and [@leolemon214](https://www.kaggle.com/leolemon214). It's little pity for dropping from public LB 3th place, expect fetching gold medal at future. \n\n**We carefully read other top solutions, and still puzzling for the 4% drop of private LB, can someone help to figure out the reason for this shaking?**\n\nBelow are valid part of our solution, all the following experiments are mainly performed on offline **dev containing 1600 examples**, and some results have been verified in public LB.\n\n## 1. Preprocessing\n\n| No | Technique | Pros | Cons | Effect |\n| ---- | --------------------------------- | ------------------------------------------------------------ | --------------------------------- | ------------------------------------- |\n| 1 | TF\\-IDF paragraph selection | Shorten doc resulting faster inference speed and better accuracy | May loss some context information | - dev f1 +1.8%,