Firstly, thanks Kaggle for great challenging competition, and congrats winners.\nThis is my first time to handle QA task, so I could learn a lot of things from you.
\n\nSecondly, thanks you all kagglers those had many discussions and ideas here, especially @christofhenkel, @boliu0, @higepon and @kashnitsky .
\n\nThis is my brief solution. I welcome questions and advises.
\n\nCombining 1 ranker model & 4 span prediction ensemble model.
\n\n1) compute passage score for all long answer candidates on test dataset\n2) select top 10 score passages for each record\n3) feed selected passage into span prediction models\n4) get averaged score by each model
\n\nOne of problems is that NQ dataset has so many candidates for long answer. These include obviously negative passages and takes much time to predict for all.
\n\nI used bert-base-uncased pre-trained and construct binary classification model to predict the passage is including long/short answer or not.\nIt get abount 0.98 recall@10 score on my validation dataset.\nIt takes about 5 minutes for public test dataset.\nOther settings are same as span prediction model.\nBy this model, I make ranker-selected dataset for train and test data by selecting top 10 score candidate for each record.
I used 4 models for ensemble:
\n\nbert-large-uncased-squad1 pre-trained + 1 epoch on NQ datasetbert-large-uncased-squad2 pre-trained + 1 epoch on NQ datasetspanbert-large-cased-squad2 pre-trained + 1 epoch on NQ datasetbert-large-uncased-squad2 pre-trained + 1 epoch on ranker-selected NQ datasetAll models are bert-joint based model.
\n\nFor 1st~3rd models, I use whole NQ dataset.\nIn training, as reported in Frustratingly Easy Natural Question Answering, I use 196 as stride, different down sampling rate for answerable and non-answerable question (each 0.01, 0.04).\nBatch size is 32, max learning rate is 3e-5.\nFor 4th model, I used ranker-selected NQ training dataset and adjust sampling rate to 0.03 for answerable and 0.12 for unanswerable.
I used only ranker-selected test dataset. This makes slight improvement on val score than predicting all candidates, and what is more important, this makes faster prediction.\nIt takes about 3 minute for each model prediction.\nI get 0.65 private LB score by single model, 0.67 private LB score by ensemble model.
Thanks.
", "messages": [ { "id": "727216", "postDate": "01/23/2020 14:34:35", "content": "Firstly, thanks Kaggle for great challenging competition, and congrats winners.\nThis is my first time to handle QA task, so I could learn a lot of things from you.
\n\nSecondly, thanks you all kagglers those had many discussions and ideas here, especially @christofhenkel, @boliu0, @higepon and @kashnitsky .
\n\nThis is my brief solution. I welcome questions and advises.
\n\nCombining 1 ranker model & 4 span prediction ensemble model.
\n\n1) compute passage score for all long answer candidates on test dataset\n2) select top 10 score passages for each record\n3) feed selected passage into span prediction models\n4) get averaged score by each model
\n\nOne of problems is that NQ dataset has so many candidates for long answer. These include obviously negative passages and takes much time to predict for all.
\n\nI used bert-base-uncased pre-trained and construct binary classification model to predict the passage is including long/short answer or not.\nIt get abount 0.98 recall@10 score on my validation dataset.\nIt takes about 5 minutes for public test dataset.\nOther settings are same as span prediction model.\nBy this model, I make ranker-selected dataset for train and test data by selecting top 10 score candidate for each record.
I used 4 models for ensemble:
\n\nbert-large-uncased-squad1 pre-trained + 1 epoch on NQ datasetbert-large-uncased-squad2 pre-trained + 1 epoch on NQ datasetspanbert-large-cased-squad2 pre-trained + 1 epoch on NQ datasetbert-large-uncased-squad2 pre-trained + 1 epoch on ranker-selected NQ datasetAll models are bert-joint based model.
\n\nFor 1st~3rd models, I use whole NQ dataset.\nIn training, as reported in Frustratingly Easy Natural Question Answering, I use 196 as stride, different down sampling rate for answerable and non-answerable question (each 0.01, 0.04).\nBatch size is 32, max learning rate is 3e-5.\nFor 4th model, I used ranker-selected NQ training dataset and adjust sampling rate to 0.03 for answerable and 0.12 for unanswerable.
I used only ranker-selected test dataset. This makes slight improvement on val score than predicting all candidates, and what is more important, this makes faster prediction.\nIt takes about 3 minute for each model prediction.\nI get 0.65 private LB score by single model, 0.67 private LB score by ensemble model.
Thanks.
", "rawMarkdown": "Firstly, thanks Kaggle for great challenging competition, and congrats winners.\nThis is my first time to handle QA task, so I could learn a lot of things from you.\n\nSecondly, thanks you all kagglers those had many discussions and ideas here, especially @christofhenkel, @boliu0, @higepon and @kashnitsky .\n\nThis is my brief solution. I welcome questions and advises.\n\n# My Solution\n\nCombining 1 ranker model & 4 span prediction ensemble model.\n\n## Whole Prediction\n\n1) compute passage score for all long answer candidates on test dataset\n2) select top 10 score passages for each record\n3) feed selected passage into span prediction models\n4) get averaged score by each model\n\n## Ranker Model\n\nOne of problems is that NQ dataset has so many candidates for long answer. These include obviously negative passages and takes much time to predict for all.\n\nI used `bert-base-uncased` pre-trained and construct binary classification model to predict the passage is including long/short answer or not.\nIt get abount 0.98 recall@10 score on my validation dataset.\nIt takes about 5 minutes for public test dataset.\nOther settings are same as span prediction model.\nBy this model, I make `ranker-selected` dataset for train and test data by selecting top 10 score candidate for each record.\n\n## Span Prediction Model\n\nI used 4 models for ensemble:\n\n1. `bert-large-uncased-squad1` pre-trained + 1 epoch on NQ dataset\n2. `bert-large-uncased-squad2` pre-trained + 1 epoch on NQ dataset\n3. `spanbert-large-cased-squad2` pre-trained + 1 epoch on NQ dataset\n4. `bert-large-uncased-squad2` pre-trained + 1 epoch on `ranker-selected` NQ dataset\n\nAll models are bert-joint based model.\n\n### Training\n\nFor 1st~3rd models, I use whole NQ dataset.\nIn training, as reported in [Frustratingly Easy Natural Question Answering](https://arxiv.org/abs/1909.05286), I use 196 as stride, different down sampling rate for answerable and non-answerable question (each 0.01, 0.04).\nBatch size is 32, max learning rate is 3e-5.\nFor 4th model, I used `ranker-selected` NQ training dataset and adjust sampling rate to 0.03 for answerable and 0.12 for unanswerable.\n\n### Inference\n\nI used only `ranker-selected` test dataset. This makes slight improvement on val score than predicting all candidates, and what is more important, this makes faster prediction.\nIt takes about 3 minute for each model prediction.\nI get 0.65 private LB score by single model, 0.67 private LB score by ensemble model.\n\n# Trials which didn't works for me\n\n- using albert, xlnet didn't improve scores. maybe I need more tuning.\n- Attention over Attention didn't affect positively. But I don't have confidence for implementation.\n- BERT layer combination on last 2, 4, 8, 12 layers. It slightly improve but pre-trained by squad was better.\n- kinds of dropout on dense layer.\n- label smoothing on start and end position.\n- combine ranker model score into span prediction lead worse result.\n- dividing short and long span prediction, or only predict short spans get worse result.\n- kinds of preprocessing\n - no special token\n - partly use special token in BERT-joint\n- kinds of postprocessing\n - use only max context position as score\n - get all logits score and obtain top k candidates\n\nThanks.", "votes": null }, { "id": "727752", "postDate": "01/24/2020 02:22:05", "content": "Congratulations\nThanks for Sharing your Approach & Insights!! @kentaronakanishi
", "rawMarkdown": "Congratulations\nThanks for Sharing your Approach & Insights!! @kentaronakanishi", "votes": null }, { "id": "728288", "postDate": "01/24/2020 15:10:33", "content": "Congrats & Thanks for sharing your solutions🎉 😄 👍
", "rawMarkdown": "Congrats & Thanks for sharing your solutions🎉 😄 👍", "votes": null }, { "id": "733312", "postDate": "01/31/2020 00:23:03", "content": "Thanks for sharing
", "rawMarkdown": "Thanks for sharing", "votes": null } ], "comments": [ { "id": 727752, "author_name": "veeralakrishna", "author_url": "", "post_date": "01/24/2020 02:22:05", "content": "Congratulations\nThanks for Sharing your Approach & Insights!! @kentaronakanishi
", "votes": null, "replies": [] }, { "id": 728288, "author_name": "mashlyn", "author_url": "", "post_date": "01/24/2020 15:10:33", "content": "Congrats & Thanks for sharing your solutions🎉 😄 👍
", "votes": null, "replies": [] }, { "id": 733312, "author_name": "corochann", "author_url": "", "post_date": "01/31/2020 00:23:03", "content": "Thanks for sharing
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "727216": "Firstly, thanks Kaggle for great challenging competition, and congrats winners.\nThis is my first time to handle QA task, so I could learn a lot of things from you.\n\nSecondly, thanks you all kagglers those had many discussions and ideas here, especially @christofhenkel, @boliu0, @higepon and @kashnitsky .\n\nThis is my brief solution. I welcome questions and advises.\n\n# My Solution\n\nCombining 1 ranker model & 4 span prediction ensemble model.\n\n## Whole Prediction\n\n1) compute passage score for all long answer candidates on test dataset\n2) select top 10 score passages for each record\n3) feed selected passage into span prediction models\n4) get averaged score by each model\n\n## Ranker Model\n\nOne of problems is that NQ dataset has so many candidates for long answer. These include obviously negative passages and takes much time to predict for all.\n\nI used `bert-base-uncased` pre-trained and construct binary classification model to predict the passage is including long/short answer or not.\nIt get abount 0.98 recall@10 score on my validation dataset.\nIt takes about 5 minutes for public test dataset.\nOther settings are same as span prediction model.\nBy this model, I make `ranker-selected` dataset for train and test data by selecting top 10 score candidate for each record.\n\n## Span Prediction Model\n\nI used 4 models for ensemble:\n\n1. `bert-large-uncased-squad1` pre-trained + 1 epoch on NQ dataset\n2. `bert-large-uncased-squad2` pre-trained + 1 epoch on NQ dataset\n3. `spanbert-large-cased-squad2` pre-trained + 1 epoch on NQ dataset\n4. `bert-large-uncased-squad2` pre-trained + 1 epoch on `ranker-selected` NQ dataset\n\nAll models are bert-joint based model.\n\n### Training\n\nFor 1st~3rd models, I use whole NQ dataset.\nIn training, as reported in [Frustratingly Easy Natural Question Answering](https://arxiv.org/abs/1909.05286), I use 196 as stride, different down sampling rate for answerable and non-answerable question (each 0.01, 0.04).\nBatch size is 32, max learning rate is 3e-5.\nFor 4th model, I used `ranker-selected` NQ training dataset and adjust sampling rate to 0.03 for answerable and 0.12 for unanswerable.\n\n### Inference\n\nI used only `ranker-selected` test dataset. This makes slight improvement on val score than predicting all candidates, and what is more important, this makes faster prediction.\nIt takes about 3 minute for each model prediction.\nI get 0.65 private LB score by single model, 0.67 private LB score by ensemble model.\n\n# Trials which didn't works for me\n\n- using albert, xlnet didn't improve scores. maybe I need more tuning.\n- Attention over Attention didn't affect positively. But I don't have confidence for implementation.\n- BERT layer combination on last 2, 4, 8, 12 layers. It slightly improve but pre-trained by squad was better.\n- kinds of dropout on dense layer.\n- label smoothing on start and end position.\n- combine ranker model score into span prediction lead worse result.\n- dividing short and long span prediction, or only predict short spans get worse result.\n- kinds of preprocessing\n - no special token\n - partly use special token in BERT-joint\n- kinds of postprocessing\n - use only max context position as score\n - get all logits score and obtain top k candidates\n\nThanks.", "727752": "Congratulations\nThanks for Sharing your Approach & Insights!! @kentaronakanishi", "728288": "Congrats & Thanks for sharing your solutions🎉 😄 👍", "733312": "Thanks for sharing" }, "source": "meta" }