Initially we set out to try both TF 1.15 and TF 2.0. Unfortunately TF 2.0 pipeline's scores are significantly lower (we probably didn't figure out how to correctly use the TF 2.0 API), so all our top submissions and what is described below were done in TF 1.15.
\n\nAll the experiments were done in Google cloud TPUs. This is my first time seriously using TPUs and I have to say, it feels so good. Because they are so fast and Google is generous enough to give us 5 TPUs, the experiment cycle is dramatically reduced.
\n\nWe noticed that the evaluation metric is not very stable on smaller validation set, for example better models on dev00 (1600 examples) may not be better on dev01. So we rely solely on the whole dev set (7830 examples) for validation (i.e. selecting checkpoints, selecting models, tuning thresholds, tuning ensemble weights). Same thing goes for the public LB. It only has 346 examples, even less stable than dev00.
\n\nMost of our models are based on the official implementation of joint-bert, which is surprisingly hard to beat. For each variation to joint-bert, we ran 5 training sessions on TPU simultaneously, with different batch size and learning rate. We save checkpoints every 500 or 1000 steps. Usually bs=64 and lr=4e-5 for 1 epoch gives the best scores.
\n\nWe tried the following variations to joint-bert. The ideas for many of them are drawn from this IBM paper: https://arxiv.org/pdf/1909.05286.pdf\n1) and 2) are the most important. Adding them always help. They contribute to the best single model. 3) to 6) sometimes help, sometimes don't. They depend on each other. Nevertheless, training with these variations did produce many diverse models, which are good for ensembling.
\n\nOfficial joint-bert was trained from \"BERT-Large, Uncased\", but training from \"BERT-Large, Uncased (Whole Word Masking)\" will see a big boost.
\n\nWe also tried fine-tuning joint-bert on Squad 2.0 before fine-tuning it on NQ.
\n\nOfficial joint-bert samples 2% negative examples in both answerable questions (i.e. the sliding windows that don't contain an answer) and unanswerable questions. As explained in the IBM paper, joint-bert tends to be overconfident for unanswerable questions, so 1% for answerable and 4% unanswerable seem to be better. We saw about 1 point increase in F1 doing negative sampling.
\n\nmax_seq_length, doc_strideDefault are 512 and 128 respectively. This means for answers not in the beginning of article, they appear about 4 times after pre-processing (in training the example is seen 4 times per epoch, in inference it's predicted 4 times, then the one with max logits is selected), which seems like an overkill.
\n\nSo we changed doc_stride to 256 during inference, which doesn't affect score much but reduced inference time in half.
\n\nFor training, we used default 128 as well 192 and 256. IBM paper claims 192 gives best results, but we didn't see much difference.
\n\nDefault is 48. There are some very very long wikepedia articles, so joint-bert only take first 48 paragraphs/tables/lists of each article. We tried different values like 100 and 200. Using a bigger value is tradeoff between more answer coverage v.s. more \"empty\" windows.
\n\nAlso proposed in the IBM paper: shuffling all the sentences in the paragraph containing short answers. This is an augmentation method.
\n\ndo_lower_case=False\nFor this, we generated a new vocab file by adding all the NQ special tokens into the cased BERT vocab.
Mentioned in the IBM paper as the most important change, but it didn't work for us.
\n\nIn 3 hours, we can do inference for 3 models with doc_stride=256. Luckily, n=3 happen to be the number of our best ensemble: adding a fourth model does not help anymore. The ensemble strategy is simply averaging the probability of each candidate span.
\n\nOur 2 submissions consist of the following 5 single models:\na. wwm, stride=256, dev 62.4\nb. wwm, neg sampling, pre-tuned on squad, dev 64.7 (long 69.5, short 57.8) - best single model\nc. wwm, neg sampling, max_contexts=200, dev 64.5\nd. wwm, neg sampling, stride=192, dev 63.8\ne. wwm, neg sampling, cased, dev 63.3
\n\nsub1: ensemble of a,b,c, dev 66.8 (long 71.6, short 59.8), private LB 0.69\nsub2: ensemble of c,d,e, dev 67.0 (long 71.6, short 59.9), private LB 0.69\n(note: these scores are after post-processing)
\n\nThese are tuned on dev set as well. If the yes/no logits in the answer_type_logit are over the thresholds, predict \"YES\"/\"NO\" regardless of the short span predictions. This gives 0.5 boost to dev F1.
Increase max_contexts from the default 48 to 100 or 80 can squeeze out another 0.3 F1 points, taking advantage of the leftover inference time within the 3 hour limit. For sub1 we did 100; for sub2 we only did 80 because generating features for the cased model e took a little more time.
repo: https://github.com/boliu61/tf2qa\ninference notebook and model weights: https://www.kaggle.com/boliu0/7th-place-submission
", "messages": [ { "id": 726570, "postDate": "2020-01-23T04:56:36.910Z", "content": "Initially we set out to try both TF 1.15 and TF 2.0. Unfortunately TF 2.0 pipeline's scores are significantly lower (we probably didn't figure out how to correctly use the TF 2.0 API), so all our top submissions and what is described below were done in TF 1.15.
\n\nAll the experiments were done in Google cloud TPUs. This is my first time seriously using TPUs and I have to say, it feels so good. Because they are so fast and Google is generous enough to give us 5 TPUs, the experiment cycle is dramatically reduced.
\n\nWe noticed that the evaluation metric is not very stable on smaller validation set, for example better models on dev00 (1600 examples) may not be better on dev01. So we rely solely on the whole dev set (7830 examples) for validation (i.e. selecting checkpoints, selecting models, tuning thresholds, tuning ensemble weights). Same thing goes for the public LB. It only has 346 examples, even less stable than dev00.
\n\nMost of our models are based on the official implementation of joint-bert, which is surprisingly hard to beat. For each variation to joint-bert, we ran 5 training sessions on TPU simultaneously, with different batch size and learning rate. We save checkpoints every 500 or 1000 steps. Usually bs=64 and lr=4e-5 for 1 epoch gives the best scores.
\n\nWe tried the following variations to joint-bert. The ideas for many of them are drawn from this IBM paper: https://arxiv.org/pdf/1909.05286.pdf\n1) and 2) are the most important. Adding them always help. They contribute to the best single model. 3) to 6) sometimes help, sometimes don't. They depend on each other. Nevertheless, training with these variations did produce many diverse models, which are good for ensembling.
\n\nOfficial joint-bert was trained from \"BERT-Large, Uncased\", but training from \"BERT-Large, Uncased (Whole Word Masking)\" will see a big boost.
\n\nWe also tried fine-tuning joint-bert on Squad 2.0 before fine-tuning it on NQ.
\n\nOfficial joint-bert samples 2% negative examples in both answerable questions (i.e. the sliding windows that don't contain an answer) and unanswerable questions. As explained in the IBM paper, joint-bert tends to be overconfident for unanswerable questions, so 1% for answerable and 4% unanswerable seem to be better. We saw about 1 point increase in F1 doing negative sampling.
\n\nmax_seq_length, doc_strideDefault are 512 and 128 respectively. This means for answers not in the beginning of article, they appear about 4 times after pre-processing (in training the example is seen 4 times per epoch, in inference it's predicted 4 times, then the one with max logits is selected), which seems like an overkill.
\n\nSo we changed doc_stride to 256 during inference, which doesn't affect score much but reduced inference time in half.
\n\nFor training, we used default 128 as well 192 and 256. IBM paper claims 192 gives best results, but we didn't see much difference.
\n\nDefault is 48. There are some very very long wikepedia articles, so joint-bert only take first 48 paragraphs/tables/lists of each article. We tried different values like 100 and 200. Using a bigger value is tradeoff between more answer coverage v.s. more \"empty\" windows.
\n\nAlso proposed in the IBM paper: shuffling all the sentences in the paragraph containing short answers. This is an augmentation method.
\n\ndo_lower_case=False\nFor this, we generated a new vocab file by adding all the NQ special tokens into the cased BERT vocab.
Mentioned in the IBM paper as the most important change, but it didn't work for us.
\n\nIn 3 hours, we can do inference for 3 models with doc_stride=256. Luckily, n=3 happen to be the number of our best ensemble: adding a fourth model does not help anymore. The ensemble strategy is simply averaging the probability of each candidate span.
\n\nOur 2 submissions consist of the following 5 single models:\na. wwm, stride=256, dev 62.4\nb. wwm, neg sampling, pre-tuned on squad, dev 64.7 (long 69.5, short 57.8) - best single model\nc. wwm, neg sampling, max_contexts=200, dev 64.5\nd. wwm, neg sampling, stride=192, dev 63.8\ne. wwm, neg sampling, cased, dev 63.3
\n\nsub1: ensemble of a,b,c, dev 66.8 (long 71.6, short 59.8), private LB 0.69\nsub2: ensemble of c,d,e, dev 67.0 (long 71.6, short 59.9), private LB 0.69\n(note: these scores are after post-processing)
\n\nThese are tuned on dev set as well. If the yes/no logits in the answer_type_logit are over the thresholds, predict \"YES\"/\"NO\" regardless of the short span predictions. This gives 0.5 boost to dev F1.
Increase max_contexts from the default 48 to 100 or 80 can squeeze out another 0.3 F1 points, taking advantage of the leftover inference time within the 3 hour limit. For sub1 we did 100; for sub2 we only did 80 because generating features for the cased model e took a little more time.
repo: https://github.com/boliu61/tf2qa\ninference notebook and model weights: https://www.kaggle.com/boliu0/7th-place-submission
", "rawMarkdown": "# Framework and hardware\nInitially we set out to try both TF 1.15 and TF 2.0. Unfortunately TF 2.0 pipeline's scores are significantly lower (we probably didn't figure out how to correctly use the TF 2.0 API), so all our top submissions and what is described below were done in TF 1.15. \n\nAll the experiments were done in Google cloud TPUs. This is my first time seriously using TPUs and I have to say, it feels so good. Because they are so fast and Google is generous enough to give us 5 TPUs, the experiment cycle is dramatically reduced.\n\n# Validation scheme and experiment setup\nWe noticed that the evaluation metric is not very stable on smaller validation set, for example better models on dev00 (1600 examples) may not be better on dev01. So we rely solely on the whole dev set (7830 examples) for validation (i.e. selecting checkpoints, selecting models, tuning thresholds, tuning ensemble weights). Same thing goes for the public LB. It only has 346 examples, even less stable than dev00.\n\nMost of our models are based on the official implementation of [joint-bert](https://github.com/google-research/language/tree/master/language/question_answering/bert_joint), which is surprisingly hard to beat. For each variation to joint-bert, we ran 5 training sessions on TPU simultaneously, with different batch size and learning rate. We save checkpoints every 500 or 1000 steps. Usually bs=64 and lr=4e-5 for 1 epoch gives the best scores.\n\n# Variations to official joint-bert\nWe tried the following variations to joint-bert. The ideas for many of them are drawn from this IBM paper: https://arxiv.org/pdf/1909.05286.pdf\n1) and 2) are the most important. Adding them always help. They contribute to the best single model. 3) to 6) sometimes help, sometimes don't. They depend on each other. Nevertheless, training with these variations did produce many diverse models, which are good for ensembling. \n## 1) pre-trained weights\nOfficial joint-bert was trained from \"BERT-Large, Uncased\", but training from \"BERT-Large, Uncased (Whole Word Masking)\" will see a big boost.\n\nWe also tried fine-tuning joint-bert on Squad 2.0 before fine-tuning it on NQ.\n\n## 2) negative sampling\nOfficial joint-bert samples 2% negative examples in both answerable questions (i.e. the sliding windows that don't contain an answer) and unanswerable questions. As explained in the IBM paper, joint-bert tends to be overconfident for unanswerable questions, so 1% for answerable and 4% unanswerable seem to be better. We saw about 1 point increase in F1 doing negative sampling.\n\n## 3) `max_seq_length`, `doc_stride`\nDefault are 512 and 128 respectively. This means for answers not in the beginning of article, they appear about 4 times after pre-processing (in training the example is seen 4 times per epoch, in inference it's predicted 4 times, then the one with max logits is selected), which seems like an overkill.\n\nSo we changed doc_stride to 256 during inference, which doesn't affect score much but reduced inference time in half. \n\nFor training, we used default 128 as well 192 and 256. IBM paper claims 192 gives best results, but we didn't see much difference.\n\n## 4) max_contexts\nDefault is 48. There are some very very long wikepedia articles, so joint-bert only take first 48 paragraphs/tables/lists of each article. We tried different values like 100 and 200. Using a bigger value is tradeoff between more answer coverage v.s. more \"empty\" windows.\n\n## 5) sentence order shuffling\nAlso proposed in the IBM paper: shuffling all the sentences in the paragraph containing short answers. This is an augmentation method. \n\n## 6) cased\n`do_lower_case=False`\nFor this, we generated a new vocab file by adding all the NQ special tokens into the cased BERT vocab.\n\n## 7) Attention-over-attention\nMentioned in the IBM paper as the most important change, but it didn't work for us.\n\n# Ensemble\nIn 3 hours, we can do inference for 3 models with doc_stride=256. Luckily, n=3 happen to be the number of our best ensemble: adding a fourth model does not help anymore. The ensemble strategy is simply averaging the probability of each candidate span.\n\nOur 2 submissions consist of the following 5 single models:\na. wwm, stride=256, dev 62.4\nb. wwm, neg sampling, pre-tuned on squad, **dev 64.7** (long 69.5, short 57.8) - best single model\nc. wwm, neg sampling, max_contexts=200, dev 64.5\nd. wwm, neg sampling, stride=192, dev 63.8\ne. wwm, neg sampling, cased, dev 63.3\n\nsub1: ensemble of a,b,c, dev 66.8 (long 71.6, short 59.8), private LB 0.69\nsub2: ensemble of c,d,e, **dev 67.0** (long 71.6, short 59.9), private LB 0.69\n(note: these scores are after post-processing)\n\n# Post-process\n### yes/no thresholds\nThese are tuned on dev set as well. If the yes/no logits in the `answer_type_logit` are over the thresholds, predict \"YES\"/\"NO\" regardless of the short span predictions. This gives 0.5 boost to dev F1.\n\n### max_contexts\nIncrease max_contexts from the default 48 to 100 or 80 can squeeze out another 0.3 F1 points, taking advantage of the leftover inference time within the 3 hour limit. For sub1 we did 100; for sub2 we only did 80 because generating features for the cased model `e` took a little more time.\n\n# Code\nrepo: https://github.com/boliu61/tf2qa\ninference notebook and model weights: https://www.kaggle.com/boliu0/7th-place-submission", "votes": 52 }, { "id": 728686, "postDate": "2020-01-25T05:00:22.847Z", "content": "Update: code is available at https://github.com/boliu61/tf2qa
\n\ninference notebook and model weights: https://www.kaggle.com/boliu0/7th-place-submission
", "rawMarkdown": "**Update**: code is available at https://github.com/boliu61/tf2qa\n\ninference notebook and model weights: https://www.kaggle.com/boliu0/7th-place-submission", "votes": 1 }, { "id": 728501, "postDate": "2020-01-24T20:02:08.713Z", "content": "Congrats & Thanks for sharing your solutionsπ π π
", "rawMarkdown": "Congrats & Thanks for sharing your solutionsπ π π ", "votes": 1 }, { "id": 728364, "postDate": "2020-01-24T16:40:08.753Z", "content": "Congratulations @boliu0, I really like the experiments you've conducted. Great write up and I will be waiting for the code. π
", "rawMarkdown": "Congratulations @boliu0, I really like the experiments you've conducted. Great write up and I will be waiting for the code. π ", "votes": 1 }, { "id": 727053, "postDate": "2020-01-23T12:17:02.920Z", "content": "Thanks for the details explanation.\n\"7. Attention-over-attention didn't work\" mean that you had implemented it but did not give any improvement, or you are unable to implement it?
", "rawMarkdown": "Thanks for the details explanation.\n\"7. Attention-over-attention didn't work\" mean that you had implemented it but did not give any improvement, or you are unable to implement it? ", "votes": 1, "replies": [ { "id": 727230, "postDate": "2020-01-23T14:47:40.083Z", "content": "We implemented it, but it didn't improve the score.
", "rawMarkdown": "We implemented it, but it didn't improve the score.", "votes": 2 }, { "id": 727597, "postDate": "2020-01-23T21:18:03.077Z", "content": "Would you mind to share to me the code with the Attention-over-attention implementation? I am interested to research further on this particular algorithm.
", "rawMarkdown": "Would you mind to share to me the code with the Attention-over-attention implementation? I am interested to research further on this particular algorithm. " }, { "id": 728688, "postDate": "2020-01-25T05:02:13.997Z", "content": "Sure. Our attempted implementation is at https://github.com/boliu61/tf2qa/blob/master/jb_train_tpu.py#L1054
", "rawMarkdown": "Sure. Our attempted implementation is at https://github.com/boliu61/tf2qa/blob/master/jb_train_tpu.py#L1054" } ] }, { "id": 726689, "postDate": "2020-01-23T07:04:17.793Z", "content": "Thanks for sharing, I learned a lot from your post.
", "rawMarkdown": "Thanks for sharing, I learned a lot from your post. ", "votes": 1 }, { "id": 726617, "postDate": "2020-01-23T06:03:54.097Z", "content": "Congratulations Bo and thanks for sharing :)
", "rawMarkdown": "Congratulations Bo and thanks for sharing :)", "votes": 1 }, { "id": 726644, "postDate": "2020-01-23T06:27:59.360Z", "content": "Congratulations\nThanks for Sharing your Approach & Insights!! @boliu0
", "rawMarkdown": "Congratulations\nThanks for Sharing your Approach & Insights!! @boliu0 ", "votes": -2 }, { "id": 739426, "postDate": "2020-02-07T20:37:36.873Z", "content": "@boliu0, congratulations! Thanks for sharing your approaches and magic! I added your post to my collection of the best Kaggle kernels and posts of winners of NLP Prize Competitions: [Data Science with DL & NLP: Advanced Techniques] (https://www.kaggle.com/vbmokin/data-science-with-dl-nlp-advanced-techniques)
", "rawMarkdown": "@boliu0, congratulations! Thanks for sharing your approaches and magic! I added your post to my collection of the best Kaggle kernels and posts of winners of NLP Prize Competitions: [Data Science with DL & NLP: Advanced Techniques] (https://www.kaggle.com/vbmokin/data-science-with-dl-nlp-advanced-techniques) " }, { "id": 726591, "postDate": "2020-01-23T05:38:41.050Z", "content": "Congrats and thanks for sharing! We discovered this paper too late to really benefit from it.
", "rawMarkdown": "Congrats and thanks for sharing! We discovered this paper too late to really benefit from it.", "replies": [ { "id": 726613, "postDate": "2020-01-23T06:02:54.640Z", "content": "Thanks and congrats to you too. Looking forward to your solution.
\n\nWe may have discovered that paper (which claimed SOTA) too early and spent too much time on it π
", "rawMarkdown": "Thanks and congrats to you too. Looking forward to your solution. \n\nWe may have discovered that paper (which claimed SOTA) too early and spent too much time on it π", "votes": 5 } ] }, { "id": 794593, "postDate": "2020-04-01T23:29:40.910Z", "rawMarkdown": "", "isDeleted": true }, { "id": 726589, "postDate": "2020-01-23T05:34:33.500Z", "content": "Congratulations and thanks for sharing!
", "rawMarkdown": "Congratulations and thanks for sharing!", "votes": 1 }, { "id": 733307, "postDate": "2020-01-31T00:18:55.643Z", "content": "Congrats and thank you for sharing!
", "rawMarkdown": "Congrats and thank you for sharing!" } ], "comments": [ { "id": 728686, "author_name": "Bo", "author_url": "", "post_date": "2020-01-25T05:00:22.847000", "content": "Update: code is available at https://github.com/boliu61/tf2qa
\n\ninference notebook and model weights: https://www.kaggle.com/boliu0/7th-place-submission
", "votes": 1, "replies": [] }, { "id": 728501, "author_name": "Miyabon", "author_url": "", "post_date": "2020-01-24T20:02:08.713000", "content": "Congrats & Thanks for sharing your solutionsπ π π
", "votes": 1, "replies": [] }, { "id": 728364, "author_name": "Mourad", "author_url": "", "post_date": "2020-01-24T16:40:08.753000", "content": "Congratulations @boliu0, I really like the experiments you've conducted. Great write up and I will be waiting for the code. π
", "votes": 1, "replies": [] }, { "id": 727053, "author_name": "Chew Kok Wah", "author_url": "", "post_date": "2020-01-23T12:17:02.920000", "content": "Thanks for the details explanation.\n\"7. Attention-over-attention didn't work\" mean that you had implemented it but did not give any improvement, or you are unable to implement it?
", "votes": 1, "replies": [ { "id": 727230, "author_name": "Bo", "author_url": "", "post_date": "2020-01-23T14:47:40.083000", "content": "We implemented it, but it didn't improve the score.
", "votes": 2, "replies": [] }, { "id": 727597, "author_name": "Chew Kok Wah", "author_url": "", "post_date": "2020-01-23T21:18:03.077000", "content": "Would you mind to share to me the code with the Attention-over-attention implementation? I am interested to research further on this particular algorithm.
", "votes": 0, "replies": [] }, { "id": 728688, "author_name": "Bo", "author_url": "", "post_date": "2020-01-25T05:02:13.997000", "content": "Sure. Our attempted implementation is at https://github.com/boliu61/tf2qa/blob/master/jb_train_tpu.py#L1054
", "votes": 0, "replies": [] } ] }, { "id": 726689, "author_name": "SchenbergZ", "author_url": "", "post_date": "2020-01-23T07:04:17.793000", "content": "Thanks for sharing, I learned a lot from your post.
", "votes": 1, "replies": [] }, { "id": 726617, "author_name": "Ram Ramrakhya", "author_url": "", "post_date": "2020-01-23T06:03:54.097000", "content": "Congratulations Bo and thanks for sharing :)
", "votes": 1, "replies": [] }, { "id": 726644, "author_name": "Ailurophile", "author_url": "", "post_date": "2020-01-23T06:27:59.360000", "content": "Congratulations\nThanks for Sharing your Approach & Insights!! @boliu0
", "votes": -2, "replies": [] }, { "id": 739426, "author_name": "Vitalii Mokin", "author_url": "", "post_date": "2020-02-07T20:37:36.873000", "content": "@boliu0, congratulations! Thanks for sharing your approaches and magic! I added your post to my collection of the best Kaggle kernels and posts of winners of NLP Prize Competitions: [Data Science with DL & NLP: Advanced Techniques] (https://www.kaggle.com/vbmokin/data-science-with-dl-nlp-advanced-techniques)
", "votes": 0, "replies": [] }, { "id": 726591, "author_name": "CPMP", "author_url": "", "post_date": "2020-01-23T05:38:41.050000", "content": "Congrats and thanks for sharing! We discovered this paper too late to really benefit from it.
", "votes": 0, "replies": [ { "id": 726613, "author_name": "Bo", "author_url": "", "post_date": "2020-01-23T06:02:54.640000", "content": "Thanks and congrats to you too. Looking forward to your solution.
\n\nWe may have discovered that paper (which claimed SOTA) too early and spent too much time on it π
", "votes": 5, "replies": [] } ] }, { "id": 794593, "author_name": "", "author_url": "", "post_date": "2020-04-01T23:29:40.910000", "content": "", "votes": 0, "replies": [] }, { "id": 726589, "author_name": "higepon", "author_url": "", "post_date": "2020-01-23T05:34:33.500000", "content": "Congratulations and thanks for sharing!
", "votes": 1, "replies": [] }, { "id": 733307, "author_name": "corochann", "author_url": "", "post_date": "2020-01-31T00:18:55.643000", "content": "Congrats and thank you for sharing!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "726570": "# Framework and hardware\nInitially we set out to try both TF 1.15 and TF 2.0. Unfortunately TF 2.0 pipeline's scores are significantly lower (we probably didn't figure out how to correctly use the TF 2.0 API), so all our top submissions and what is described below were done in TF 1.15. \n\nAll the experiments were done in Google cloud TPUs. This is my first time seriously using TPUs and I have to say, it feels so good. Because they are so fast and Google is generous enough to give us 5 TPUs, the experiment cycle is dramatically reduced.\n\n# Validation scheme and experiment setup\nWe noticed that the evaluation metric is not very stable on smaller validation set, for example better models on dev00 (1600 examples) may not be better on dev01. So we rely solely on the whole dev set (7830 examples) for validation (i.e. selecting checkpoints, selecting models, tuning thresholds, tuning ensemble weights). Same thing goes for the public LB. It only has 346 examples, even less stable than dev00.\n\nMost of our models are based on the official implementation of [joint-bert](https://github.com/google-research/language/tree/master/language/question_answering/bert_joint), which is surprisingly hard to beat. For each variation to joint-bert, we ran 5 training sessions on TPU simultaneously, with different batch size and learning rate. We save checkpoints every 500 or 1000 steps. Usually bs=64 and lr=4e-5 for 1 epoch gives the best scores.\n\n# Variations to official joint-bert\nWe tried the following variations to joint-bert. The ideas for many of them are drawn from this IBM paper: https://arxiv.org/pdf/1909.05286.pdf\n1) and 2) are the most important. Adding them always help. They contribute to the best single model. 3) to 6) sometimes help, sometimes don't. They depend on each other. Nevertheless, training with these variations did produce many diverse models, which are good for ensembling. \n## 1) pre-trained weights\nOfficial joint-bert was trained from \"BERT-Large, Uncased\", but training from \"BERT-Large, Uncased (Whole Word Masking)\" will see a big boost.\n\nWe also tried fine-tuning joint-bert on Squad 2.0 before fine-tuning it on NQ.\n\n## 2) negative sampling\nOfficial joint-bert samples 2% negative examples in both answerable questions (i.e. the sliding windows that don't contain an answer) and unanswerable questions. As explained in the IBM paper, joint-bert tends to be overconfident for unanswerable questions, so 1% for answerable and 4% unanswerable seem to be better. We saw about 1 point increase in F1 doing negative sampling.\n\n## 3) `max_seq_length`, `doc_stride`\nDefault are 512 and 128 respectively. This means for answers not in the beginning of article, they appear about 4 times after pre-processing (in training the example is seen 4 times per epoch, in inference it's predicted 4 times, then the one with max logits is selected), which seems like an overkill.\n\nSo we changed doc_stride to 256 during inference, which doesn't affect score much but reduced inference time in half. \n\nFor training, we used default 128 as well 192 and 256. IBM paper claims 192 gives best results, but we didn't see much difference.\n\n## 4) max_contexts\nDefault is 48. There are some very very long wikepedia articles, so joint-bert only take first 48 paragraphs/tables/lists of each article. We tried different values like 100 and 200. Using a bigger value is tradeoff between more answer coverage v.s. more \"empty\" windows.\n\n## 5) sentence order shuffling\nAlso proposed in the IBM paper: shuffling all the sentences in the paragraph containing short answers. This is an augmentation method. \n\n## 6) cased\n`do_lower_case=False`\nFor this, we generated a new vocab file by adding all the NQ special tokens into the cased BERT vocab.\n\n## 7) Attention-over-attention\nMentioned in the IBM paper as the most important change, but it didn't work for us.\n\n# Ensemble\nIn 3 hours, we can do inference for 3 models with doc_stride=256. Luckily, n=3 happen to be the number of our best ensemble: adding a fourth model does not help anymore. The ensemble strategy is simply averaging the probability of each candidate span.\n\nOur 2 submissions consist of the following 5 single models:\na. wwm, stride=256, dev 62.4\nb. wwm, neg sampling, pre-tuned on squad, **dev 64.7** (long 69.5, short 57.8) - best single model\nc. wwm, neg sampling, max_contexts=200, dev 64.5\nd. wwm, neg sampling, stride=192, dev 63.8\ne. wwm, neg sampling, cased, dev 63.3\n\nsub1: ensemble of a,b,c, dev 66.8 (long 71.6, short 59.8), private LB 0.69\nsub2: ensemble of c,d,e, **dev 67.0** (long 71.6, short 59.9), private LB 0.69\n(note: these scores are after post-processing)\n\n# Post-process\n### yes/no thresholds\nThese are tuned on dev set as well. If the yes/no logits in the `answer_type_logit` are over the thresholds, predict \"YES\"/\"NO\" regardless of the short span predictions. This gives 0.5 boost to dev F1.\n\n### max_contexts\nIncrease max_contexts from the default 48 to 100 or 80 can squeeze out another 0.3 F1 points, taking advantage of the leftover inference time within the 3 hour limit. For sub1 we did 100; for sub2 we only did 80 because generating features for the cased model `e` took a little more time.\n\n# Code\nrepo: https://github.com/boliu61/tf2qa\ninference notebook and model weights: https://www.kaggle.com/boliu0/7th-place-submission", "728686": "**Update**: code is available at https://github.com/boliu61/tf2qa\n\ninference notebook and model weights: https://www.kaggle.com/boliu0/7th-place-submission", "728501": "Congrats & Thanks for sharing your solutionsπ π π ", "728364": "Congratulations @boliu0, I really like the experiments you've conducted. Great write up and I will be waiting for the code. π ", "727053": "Thanks for the details explanation.\n\"7. Attention-over-attention didn't work\" mean that you had implemented it but did not give any improvement, or you are unable to implement it? ", "726689": "Thanks for sharing, I learned a lot from your post. ", "726617": "Congratulations Bo and thanks for sharing :)", "726644": "Congratulations\nThanks for Sharing your Approach & Insights!! @boliu0 ", "739426": "@boliu0, congratulations! Thanks for sharing your approaches and magic! I added your post to my collection of the best Kaggle kernels and posts of winners of NLP Prize Competitions: [Data Science with DL & NLP: Advanced Techniques] (https://www.kaggle.com/vbmokin/data-science-with-dl-nlp-advanced-techniques) ", "726591": "Congrats and thanks for sharing! We discovered this paper too late to really benefit from it.", "794593": "", "726589": "Congratulations and thanks for sharing!", "733307": "Congrats and thank you for sharing!" } }