It seems no hints, no words about models-usage other than BERT such as ROBERTA, ALBERT, GPT2, XLNET in Discussion and Notebook sessions... SO are there any kagglers trying these models?
", "messages": [ { "id": "701225", "postDate": "12/23/2019 08:59:38", "content": "It seems no hints, no words about models-usage other than BERT such as ROBERTA, ALBERT, GPT2, XLNET in Discussion and Notebook sessions... SO are there any kagglers trying these models?
", "rawMarkdown": "It seems no hints, no words about models-usage other than BERT such as ROBERTA, ALBERT, GPT2, XLNET in Discussion and Notebook sessions... SO are there any kagglers trying these models?", "votes": null }, { "id": "704363", "postDate": "12/27/2019 11:35:37", "content": "I started looking into BERT at first. But since QA models are relatively new for me...and the run times of training a BERT model on my 1070 Ti are way to slow I started looking around for some other models.
\n\nI'am currently modifying a QA-Net which used to be more or less one of the top models before BERT and his family appeared.\nIt is absolutely not material to win the contest..but it is very enjoyable to update the complete framework for this competition and to get a fundamental understanding of how it works.
\n\nAlso since it is CNN based it is fast to train and try out stuff compared to a BERT-like model.
", "rawMarkdown": "I started looking into BERT at first. But since QA models are relatively new for me...and the run times of training a BERT model on my 1070 Ti are way to slow I started looking around for some other models.\n\nI'am currently modifying a QA-Net which used to be more or less one of the top models before BERT and his family appeared.\nIt is absolutely not material to win the contest..but it is very enjoyable to update the complete framework for this competition and to get a fundamental understanding of how it works.\n\nAlso since it is CNN based it is fast to train and try out stuff compared to a BERT-like model.", "votes": null }, { "id": "711168", "postDate": "01/05/2020 19:11:39", "content": "Cool to hear CNN methods. So have you tried combine bert force with CNN output head?
", "rawMarkdown": "Cool to hear CNN methods. So have you tried combine bert force with CNN output head?", "votes": null }, { "id": "711258", "postDate": "01/05/2020 21:14:11", "content": "Since I am new to NLP in general, I've only tried DrQA Reader(Bidirectional LSTM) beside BERT and ALBERT.
", "rawMarkdown": "Since I am new to NLP in general, I've only tried [DrQA Reader](https://www.kaggle.com/msheriey/starter-drqa-reader)(Bidirectional LSTM) beside BERT and ALBERT.", "votes": null }, { "id": "711300", "postDate": "01/05/2020 22:45:40", "content": "Hey @httpwwwfszyc No I haven't tried that but it sounds as something cool to try. I basically spent the last 4 weeks with diving into the whole QA stuff. I used one of the QANet frameworks from Github completely worked through it and modified it for this competition. Very learnfull and fun to do :-)
\n\nFor the last few days I was finally able to create submissions with it...but the score was constantly 0. After finding out and fixing the last issue it just gave me a score of 0.15.
\n\nNot bad for a test run where I trained from scratch in less than 2 hours :-)
\n\nSo I will dive into BERT...but the current QA-Net is just to much fun to stop with now ;-)
", "rawMarkdown": "Hey @httpwwwfszyc No I haven't tried that but it sounds as something cool to try. I basically spent the last 4 weeks with diving into the whole QA stuff. I used one of the QANet frameworks from Github completely worked through it and modified it for this competition. Very learnfull and fun to do :-)\n\nFor the last few days I was finally able to create submissions with it...but the score was constantly 0. After finding out and fixing the last issue it just gave me a score of 0.15.\n\nNot bad for a test run where I trained from scratch in less than 2 hours :-)\n\nSo I will dive into BERT...but the current QA-Net is just to much fun to stop with now ;-)", "votes": null } ], "comments": [ { "id": 704363, "author_name": "rsmits", "author_url": "", "post_date": "12/27/2019 11:35:37", "content": "I started looking into BERT at first. But since QA models are relatively new for me...and the run times of training a BERT model on my 1070 Ti are way to slow I started looking around for some other models.
\n\nI'am currently modifying a QA-Net which used to be more or less one of the top models before BERT and his family appeared.\nIt is absolutely not material to win the contest..but it is very enjoyable to update the complete framework for this competition and to get a fundamental understanding of how it works.
\n\nAlso since it is CNN based it is fast to train and try out stuff compared to a BERT-like model.
", "votes": null, "replies": [ { "id": 711168, "author_name": "httpwwwfszyc", "author_url": "", "post_date": "01/05/2020 19:11:39", "content": "Cool to hear CNN methods. So have you tried combine bert force with CNN output head?
", "votes": null, "replies": [] }, { "id": 711300, "author_name": "rsmits", "author_url": "", "post_date": "01/05/2020 22:45:40", "content": "Hey @httpwwwfszyc No I haven't tried that but it sounds as something cool to try. I basically spent the last 4 weeks with diving into the whole QA stuff. I used one of the QANet frameworks from Github completely worked through it and modified it for this competition. Very learnfull and fun to do :-)
\n\nFor the last few days I was finally able to create submissions with it...but the score was constantly 0. After finding out and fixing the last issue it just gave me a score of 0.15.
\n\nNot bad for a test run where I trained from scratch in less than 2 hours :-)
\n\nSo I will dive into BERT...but the current QA-Net is just to much fun to stop with now ;-)
", "votes": null, "replies": [] } ] }, { "id": 711258, "author_name": "msheriey", "author_url": "", "post_date": "01/05/2020 21:14:11", "content": "Since I am new to NLP in general, I've only tried DrQA Reader(Bidirectional LSTM) beside BERT and ALBERT.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "701225": "It seems no hints, no words about models-usage other than BERT such as ROBERTA, ALBERT, GPT2, XLNET in Discussion and Notebook sessions... SO are there any kagglers trying these models?", "704363": "I started looking into BERT at first. But since QA models are relatively new for me...and the run times of training a BERT model on my 1070 Ti are way to slow I started looking around for some other models.\n\nI'am currently modifying a QA-Net which used to be more or less one of the top models before BERT and his family appeared.\nIt is absolutely not material to win the contest..but it is very enjoyable to update the complete framework for this competition and to get a fundamental understanding of how it works.\n\nAlso since it is CNN based it is fast to train and try out stuff compared to a BERT-like model.", "711168": "Cool to hear CNN methods. So have you tried combine bert force with CNN output head?", "711258": "Since I am new to NLP in general, I've only tried [DrQA Reader](https://www.kaggle.com/msheriey/starter-drqa-reader)(Bidirectional LSTM) beside BERT and ALBERT.", "711300": "Hey @httpwwwfszyc No I haven't tried that but it sounds as something cool to try. I basically spent the last 4 weeks with diving into the whole QA stuff. I used one of the QANet frameworks from Github completely worked through it and modified it for this competition. Very learnfull and fun to do :-)\n\nFor the last few days I was finally able to create submissions with it...but the score was constantly 0. After finding out and fixing the last issue it just gave me a score of 0.15.\n\nNot bad for a test run where I trained from scratch in less than 2 hours :-)\n\nSo I will dive into BERT...but the current QA-Net is just to much fun to stop with now ;-)" }, "source": "meta" }