Congrats to all winners!
\n\nI've made available my solution (public 0.65 private 0.67).\nhttps://www.kaggle.com/siriuself/tf-qa-wwm-verifier-forked
\n\nhttps://www.kaggle.com/siriuself/bert-disjoint-fn-builder\nhttps://www.kaggle.com/siriuself/bert-disjoint-modeling\nhttps://www.kaggle.com/siriuself/bert-disjoint-utils\nhttps://www.kaggle.com/siriuself/albert-yes-no-fn-builder\nhttps://www.kaggle.com/siriuself/albert-yes-no-modeling\nhttps://www.kaggle.com/siriuself/albert-yes-no-utils\nhttps://www.kaggle.com/siriuself/tokenization\nhttps://www.kaggle.com/siriuself/create-submission
\n\nMy model is simple, BERT large whole-word-masking uncased, retrained using the start/end logit loss only without the answer type loss. Using provided nq train tf record with the following setting:\nbatch_size: 32\nepoch: 2\nalpha: 2e-5, but use ckpt-15000 (so stop at around 1 epoch)\nNote that because of learning rate warmup/decay, this is different from training with 2e-5 for 1 epoch
\n\nThen totally disregard answer type classification (since I don't have it), and rely on threshold setting for long and short questions, tuned on the dev set with nq_eval. And yes I let go all the YES/NO questions.
\n\nI did try adding an ALBERT xxlarge yes/no verifier after the BERT stage, which seemed to improve for like 1 pt on dev, but apparently not on the LBs somehow. My kernel includes the ALBERT part too. There isn't much insight in the utility scripts, except the modifications I made in order to restore the checkpoint into some contrib layers forced to be re-written in keras (e.g. LayerNormalization). It was a nightmare..
\n\n****************some reflections/insights************\nI started the competition way too late and was on the wrong ALBERT-taking-too-long direction for quite a while, ending up with little time for tuning. I still have a very strong feeling that the \"joint\" part of bert-joint might be of little use, since we've already known that:\n1) BERT-like structure is poor at passage ranking, and to make it better we need passages at least as many as in MS-MARCO\n2) We only have like 1-3% YES/NO in our training data. Very unbalanced.\nBased on my inspection and verification experiment this might well be the case, which means the answer type classifier might just be reduced to a question type classifier (a much easier task for BERT to see), which would be a bad indicator of what type of answer the passage contains. It might not be a good idea to include it in training in the first place, and it'd be a disaster if you over-rely on the answer type logits for post-processing.
\n\nLet me know if you have similar/opposite findings. It's just my feeling anyway, with a bit of confirmation of my score, obtained by doing nothing other than removing the answer type loss in training. This is almost higher than my ALBERT-xxlarge-joint model too.. I almost felt like doing more experiments and writing a paper on the classification power reduction, but guess it's too trivial.
\n\nLastly, we are hiring intermediate/senior NLP engineer/researcher/scientist, with possibility of sponsorship to come to our Canadian headquarter in Waterloo, Ontario for strong candidates. Bilingualism in English and Mandarin is a plus.
", "messages": [ { "id": "726450", "postDate": "01/23/2020 02:34:02", "content": "Congrats to all winners!
\n\nI've made available my solution (public 0.65 private 0.67).\nhttps://www.kaggle.com/siriuself/tf-qa-wwm-verifier-forked
\n\nhttps://www.kaggle.com/siriuself/bert-disjoint-fn-builder\nhttps://www.kaggle.com/siriuself/bert-disjoint-modeling\nhttps://www.kaggle.com/siriuself/bert-disjoint-utils\nhttps://www.kaggle.com/siriuself/albert-yes-no-fn-builder\nhttps://www.kaggle.com/siriuself/albert-yes-no-modeling\nhttps://www.kaggle.com/siriuself/albert-yes-no-utils\nhttps://www.kaggle.com/siriuself/tokenization\nhttps://www.kaggle.com/siriuself/create-submission
\n\nMy model is simple, BERT large whole-word-masking uncased, retrained using the start/end logit loss only without the answer type loss. Using provided nq train tf record with the following setting:\nbatch_size: 32\nepoch: 2\nalpha: 2e-5, but use ckpt-15000 (so stop at around 1 epoch)\nNote that because of learning rate warmup/decay, this is different from training with 2e-5 for 1 epoch
\n\nThen totally disregard answer type classification (since I don't have it), and rely on threshold setting for long and short questions, tuned on the dev set with nq_eval. And yes I let go all the YES/NO questions.
\n\nI did try adding an ALBERT xxlarge yes/no verifier after the BERT stage, which seemed to improve for like 1 pt on dev, but apparently not on the LBs somehow. My kernel includes the ALBERT part too. There isn't much insight in the utility scripts, except the modifications I made in order to restore the checkpoint into some contrib layers forced to be re-written in keras (e.g. LayerNormalization). It was a nightmare..
\n\n****************some reflections/insights************\nI started the competition way too late and was on the wrong ALBERT-taking-too-long direction for quite a while, ending up with little time for tuning. I still have a very strong feeling that the \"joint\" part of bert-joint might be of little use, since we've already known that:\n1) BERT-like structure is poor at passage ranking, and to make it better we need passages at least as many as in MS-MARCO\n2) We only have like 1-3% YES/NO in our training data. Very unbalanced.\nBased on my inspection and verification experiment this might well be the case, which means the answer type classifier might just be reduced to a question type classifier (a much easier task for BERT to see), which would be a bad indicator of what type of answer the passage contains. It might not be a good idea to include it in training in the first place, and it'd be a disaster if you over-rely on the answer type logits for post-processing.
\n\nLet me know if you have similar/opposite findings. It's just my feeling anyway, with a bit of confirmation of my score, obtained by doing nothing other than removing the answer type loss in training. This is almost higher than my ALBERT-xxlarge-joint model too.. I almost felt like doing more experiments and writing a paper on the classification power reduction, but guess it's too trivial.
\n\nLastly, we are hiring intermediate/senior NLP engineer/researcher/scientist, with possibility of sponsorship to come to our Canadian headquarter in Waterloo, Ontario for strong candidates. Bilingualism in English and Mandarin is a plus.
", "rawMarkdown": "Congrats to all winners!\n\nI've made available my solution (public 0.65 private 0.67).\nhttps://www.kaggle.com/siriuself/tf-qa-wwm-verifier-forked\n\nhttps://www.kaggle.com/siriuself/bert-disjoint-fn-builder\nhttps://www.kaggle.com/siriuself/bert-disjoint-modeling\nhttps://www.kaggle.com/siriuself/bert-disjoint-utils\nhttps://www.kaggle.com/siriuself/albert-yes-no-fn-builder\nhttps://www.kaggle.com/siriuself/albert-yes-no-modeling\nhttps://www.kaggle.com/siriuself/albert-yes-no-utils\nhttps://www.kaggle.com/siriuself/tokenization\nhttps://www.kaggle.com/siriuself/create-submission\n\nMy model is simple, BERT large whole-word-masking uncased, retrained using the start/end logit loss only without the answer type loss. Using provided nq train tf record with the following setting:\n**batch_size**: 32\n**epoch**: 2\n**alpha**: 2e-5, but use ckpt-15000 (so stop at around 1 epoch)\nNote that because of learning rate warmup/decay, this is different from training with 2e-5 for 1 epoch\n\nThen totally disregard answer type classification (since I don't have it), and rely on threshold setting for long and short questions, tuned on the dev set with nq_eval. And yes I let go all the YES/NO questions.\n\nI did try adding an ALBERT xxlarge yes/no verifier after the BERT stage, which seemed to improve for like 1 pt on dev, but apparently not on the LBs somehow. My kernel includes the ALBERT part too. There isn't much insight in the utility scripts, except the modifications I made in order to restore the checkpoint into some contrib layers forced to be re-written in keras (e.g. LayerNormalization). It was a nightmare..\n\n*********************************some reflections/insights*************************\nI started the competition way too late and was on the wrong ALBERT-taking-too-long direction for quite a while, ending up with little time for tuning. I still have a very strong feeling that the \"joint\" part of bert-joint might be of little use, since we've already known that:\n1) BERT-like structure is poor at passage ranking, and to make it better we need passages at least as many as in MS-MARCO\n2) We only have like 1-3% YES/NO in our training data. Very unbalanced.\nBased on my inspection and verification experiment this might well be the case, which means the answer type classifier might just be reduced to a question type classifier (a much easier task for BERT to see), which would be a bad indicator of what type of answer the passage contains. It might not be a good idea to include it in training in the first place, and it'd be a disaster if you over-rely on the answer type logits for post-processing.\n\nLet me know if you have similar/opposite findings. It's just my feeling anyway, with a bit of confirmation of my score, obtained by doing nothing other than removing the answer type loss in training. This is almost higher than my ALBERT-xxlarge-joint model too.. I almost felt like doing more experiments and writing a paper on the classification power reduction, but guess it's too trivial.\n\nLastly, we are hiring intermediate/senior NLP engineer/researcher/scientist, with possibility of sponsorship to come to our Canadian headquarter in Waterloo, Ontario for strong candidates. Bilingualism in English and Mandarin is a plus.", "votes": null }, { "id": "726459", "postDate": "01/23/2020 02:44:28", "content": "Great Job!
", "rawMarkdown": "Great Job!", "votes": null }, { "id": "728480", "postDate": "01/24/2020 19:29:57", "content": "Congrats & Thanks for sharing your solutions🎉 😄 👍
", "rawMarkdown": "Congrats & Thanks for sharing your solutions🎉 😄 👍", "votes": null }, { "id": "733310", "postDate": "01/31/2020 00:22:14", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!", "votes": null }, { "id": "734165", "postDate": "02/01/2020 03:26:55", "content": "Nice! Thanks for the open-source!
", "rawMarkdown": "Nice! Thanks for the open-source!", "votes": null } ], "comments": [ { "id": 726459, "author_name": "renxingkai", "author_url": "", "post_date": "01/23/2020 02:44:28", "content": "Great Job!
", "votes": null, "replies": [] }, { "id": 728480, "author_name": "mashlyn", "author_url": "", "post_date": "01/24/2020 19:29:57", "content": "Congrats & Thanks for sharing your solutions🎉 😄 👍
", "votes": null, "replies": [] }, { "id": 733310, "author_name": "corochann", "author_url": "", "post_date": "01/31/2020 00:22:14", "content": "Thanks for sharing!
", "votes": null, "replies": [] }, { "id": 734165, "author_name": "adityaecdrid", "author_url": "", "post_date": "02/01/2020 03:26:55", "content": "Nice! Thanks for the open-source!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "726450": "Congrats to all winners!\n\nI've made available my solution (public 0.65 private 0.67).\nhttps://www.kaggle.com/siriuself/tf-qa-wwm-verifier-forked\n\nhttps://www.kaggle.com/siriuself/bert-disjoint-fn-builder\nhttps://www.kaggle.com/siriuself/bert-disjoint-modeling\nhttps://www.kaggle.com/siriuself/bert-disjoint-utils\nhttps://www.kaggle.com/siriuself/albert-yes-no-fn-builder\nhttps://www.kaggle.com/siriuself/albert-yes-no-modeling\nhttps://www.kaggle.com/siriuself/albert-yes-no-utils\nhttps://www.kaggle.com/siriuself/tokenization\nhttps://www.kaggle.com/siriuself/create-submission\n\nMy model is simple, BERT large whole-word-masking uncased, retrained using the start/end logit loss only without the answer type loss. Using provided nq train tf record with the following setting:\n**batch_size**: 32\n**epoch**: 2\n**alpha**: 2e-5, but use ckpt-15000 (so stop at around 1 epoch)\nNote that because of learning rate warmup/decay, this is different from training with 2e-5 for 1 epoch\n\nThen totally disregard answer type classification (since I don't have it), and rely on threshold setting for long and short questions, tuned on the dev set with nq_eval. And yes I let go all the YES/NO questions.\n\nI did try adding an ALBERT xxlarge yes/no verifier after the BERT stage, which seemed to improve for like 1 pt on dev, but apparently not on the LBs somehow. My kernel includes the ALBERT part too. There isn't much insight in the utility scripts, except the modifications I made in order to restore the checkpoint into some contrib layers forced to be re-written in keras (e.g. LayerNormalization). It was a nightmare..\n\n*********************************some reflections/insights*************************\nI started the competition way too late and was on the wrong ALBERT-taking-too-long direction for quite a while, ending up with little time for tuning. I still have a very strong feeling that the \"joint\" part of bert-joint might be of little use, since we've already known that:\n1) BERT-like structure is poor at passage ranking, and to make it better we need passages at least as many as in MS-MARCO\n2) We only have like 1-3% YES/NO in our training data. Very unbalanced.\nBased on my inspection and verification experiment this might well be the case, which means the answer type classifier might just be reduced to a question type classifier (a much easier task for BERT to see), which would be a bad indicator of what type of answer the passage contains. It might not be a good idea to include it in training in the first place, and it'd be a disaster if you over-rely on the answer type logits for post-processing.\n\nLet me know if you have similar/opposite findings. It's just my feeling anyway, with a bit of confirmation of my score, obtained by doing nothing other than removing the answer type loss in training. This is almost higher than my ALBERT-xxlarge-joint model too.. I almost felt like doing more experiments and writing a paper on the classification power reduction, but guess it's too trivial.\n\nLastly, we are hiring intermediate/senior NLP engineer/researcher/scientist, with possibility of sponsorship to come to our Canadian headquarter in Waterloo, Ontario for strong candidates. Bilingualism in English and Mandarin is a plus.", "726459": "Great Job!", "728480": "Congrats & Thanks for sharing your solutions🎉 😄 👍", "733310": "Thanks for sharing!", "734165": "Nice! Thanks for the open-source!" }, "source": "meta" }