![\"Synonyms](\"https://i.imgur.com/IrA2CmI.png\")
Hello everyone.
\n\nThis was my first real experience in kaggle.
\n\nBriefly, I want to say that I'm very glad that my team achieved 30th place in this competition. I want to thank my teammates @xiaokangwang @xiao-xiao for all the help and work during this competition. Also thanks for all the answers given during the competition explaining everything and giving awesome ideas!
\n\nWithout further ado, the solution mainly focused on three things:\n-> Fine-tuning Albert Large\n-> DataAugmentation\n-> Understanding thresholds.
\n\nThe model used was Albert xLarge version 2 trained on SQUAD2.0 and Fine-tuned on the tiny-dev. One of the important things was the DataAugmentation used for training, changing the document_text replacing words by synonyms using WordNet corpus.
\n\n
In here there wasn't no much of magic. We also tried xxLarge only able to run with a huge doc_stride giving bad results.
\n\nRemoving html tags helped as well.
\n\nUnderstand the data and the output of the model was an important step. \nThe long_answers appeared 50% of the times\nThe short_answers appeared 35% of the times.
\n\n![\"Output](\"https://i.imgur.com/eHTopaB.png\")
In this regard, we used threshold values based on the outputs using 50% of all the results for the long_answers e 35% for the short_answers. This was achieve by ordering the list of results, select the number that was in the middle and for the short_answer, 35%. For example, given an output list [1,2,3,4,5] we selected the threshold 3 for the long_answer and 2 for short_answer.
\n\nI hope this gives good ideas for future work.
\n\nBest Regards,\nPedro Azevedo
", "messages": [ { "id": 732090, "postDate": "2020-01-29T13:39:47.380Z", "content": "Hello everyone.
\n\nThis was my first real experience in kaggle.
\n\nBriefly, I want to say that I'm very glad that my team achieved 30th place in this competition. I want to thank my teammates @xiaokangwang @xiao-xiao for all the help and work during this competition. Also thanks for all the answers given during the competition explaining everything and giving awesome ideas!
\n\nWithout further ado, the solution mainly focused on three things:\n-> Fine-tuning Albert Large\n-> DataAugmentation\n-> Understanding thresholds.
\n\nThe model used was Albert xLarge version 2 trained on SQUAD2.0 and Fine-tuned on the tiny-dev. One of the important things was the DataAugmentation used for training, changing the document_text replacing words by synonyms using WordNet corpus.
\n\n
In here there wasn't no much of magic. We also tried xxLarge only able to run with a huge doc_stride giving bad results.
\n\nRemoving html tags helped as well.
\n\nUnderstand the data and the output of the model was an important step. \nThe long_answers appeared 50% of the times\nThe short_answers appeared 35% of the times.
\n\n
In this regard, we used threshold values based on the outputs using 50% of all the results for the long_answers e 35% for the short_answers. This was achieve by ordering the list of results, select the number that was in the middle and for the short_answer, 35%. For example, given an output list [1,2,3,4,5] we selected the threshold 3 for the long_answer and 2 for short_answer.
\n\nI hope this gives good ideas for future work.
\n\nBest Regards,\nPedro Azevedo
", "rawMarkdown": "Hello everyone.\n\nThis was my first real experience in kaggle.\n\nBriefly, I want to say that I'm very glad that my team achieved 30th place in this competition. I want to thank my teammates @xiaokangwang @xiao-xiao for all the help and work during this competition. Also thanks for all the answers given during the competition explaining everything and giving awesome ideas!\n\nWithout further ado, the solution mainly focused on three things:\n-> Fine-tuning Albert Large\n-> DataAugmentation\n-> Understanding thresholds.\n\nThe model used was **Albert xLarge version 2** trained on SQUAD2.0 and Fine-tuned on the tiny-dev. One of the important things was the **DataAugmentation** used for training, changing the document_text replacing words by synonyms using WordNet corpus.\n\n\n\nIn here there wasn't no much of magic. We also tried xxLarge only able to run with a huge doc_stride giving bad results. \n\nRemoving html tags helped as well.\n\nUnderstand the data and the output of the model was an important step. \nThe long_answers appeared 50% of the times\nThe short_answers appeared 35% of the times.\n\n\n\nIn this regard, we used threshold values based on the outputs using 50% of all the results for the long_answers e 35% for the short_answers. This was achieve by ordering the list of results, select the number that was in the middle and for the short_answer, 35%. For example, given an output list [1,2,3,4,5] we selected the threshold 3 for the long_answer and 2 for short_answer.\n\nI hope this gives good ideas for future work. \n\nBest Regards,\nPedro Azevedo", "votes": 8 }, { "id": 732879, "postDate": "2020-01-30T12:18:40.023Z", "content": "Congrats!
", "rawMarkdown": "Congrats!", "votes": 4 }, { "id": 733206, "postDate": "2020-01-30T20:26:35.663Z", "content": "Congratulations!!
", "rawMarkdown": "Congratulations!!" }, { "id": 732446, "postDate": "2020-01-29T20:45:14.160Z", "content": "Sorry, the images were not showing. Fixed the bug!
", "rawMarkdown": "Sorry, the images were not showing. Fixed the bug! " }, { "id": 732350, "postDate": "2020-01-29T18:01:58.433Z", "content": "Congratulations!!\nThanks for sharing your Approach & Insights @pedrojlazevedo
", "rawMarkdown": "Congratulations!!\nThanks for sharing your Approach & Insights @pedrojlazevedo " }, { "id": 732142, "postDate": "2020-01-29T14:30:02.400Z", "content": "Thanks for your sharing!\nI have a question about how to train Albert xLarge on SQUAD2.0 ?
", "rawMarkdown": "Thanks for your sharing!\nI have a question about how to train Albert xLarge on SQUAD2.0 ?", "replies": [ { "id": 732154, "postDate": "2020-01-29T14:37:10.780Z", "content": "I hope that you liked.
\n\nYou can see in their github:ALBERT\nIn the read.me you can see all the instructions.
\n\nAlso check the article that the Albert development team did:\nArticle
", "rawMarkdown": "I hope that you liked.\n\nYou can see in their github:[ALBERT](https://github.com/google-research/ALBERT/)\nIn the read.me you can see all the instructions.\n\nAlso check the article that the Albert development team did:\n[Article](https://arxiv.org/pdf/1909.11942.pdf)", "votes": 2 }, { "id": 732185, "postDate": "2020-01-29T14:51:14.300Z", "content": "Thanks for your explanation.\nIt's very helpful for me.
", "rawMarkdown": "Thanks for your explanation.\nIt's very helpful for me.", "votes": 1 } ] }, { "id": 732447, "postDate": "2020-01-29T20:46:20.427Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 732879, "author_name": "", "author_url": "", "post_date": "2020-01-30T12:18:40.023000", "content": "Congrats!
", "votes": 4, "replies": [] }, { "id": 733206, "author_name": "Manish Nayak", "author_url": "", "post_date": "2020-01-30T20:26:35.663000", "content": "Congratulations!!
", "votes": 0, "replies": [] }, { "id": 732446, "author_name": "Pedro Azevedo", "author_url": "", "post_date": "2020-01-29T20:45:14.160000", "content": "Sorry, the images were not showing. Fixed the bug!
", "votes": 0, "replies": [] }, { "id": 732350, "author_name": "Ailurophile", "author_url": "", "post_date": "2020-01-29T18:01:58.433000", "content": "Congratulations!!\nThanks for sharing your Approach & Insights @pedrojlazevedo
", "votes": 0, "replies": [] }, { "id": 732142, "author_name": "Dean", "author_url": "", "post_date": "2020-01-29T14:30:02.400000", "content": "Thanks for your sharing!\nI have a question about how to train Albert xLarge on SQUAD2.0 ?
", "votes": 0, "replies": [ { "id": 732154, "author_name": "Pedro Azevedo", "author_url": "", "post_date": "2020-01-29T14:37:10.780000", "content": "I hope that you liked.
\n\nYou can see in their github:ALBERT\nIn the read.me you can see all the instructions.
\n\nAlso check the article that the Albert development team did:\nArticle
", "votes": 2, "replies": [] }, { "id": 732185, "author_name": "Dean", "author_url": "", "post_date": "2020-01-29T14:51:14.300000", "content": "Thanks for your explanation.\nIt's very helpful for me.
", "votes": 1, "replies": [] } ] }, { "id": 732447, "author_name": "", "author_url": "", "post_date": "2020-01-29T20:46:20.427000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "732090": "Hello everyone.\n\nThis was my first real experience in kaggle.\n\nBriefly, I want to say that I'm very glad that my team achieved 30th place in this competition. I want to thank my teammates @xiaokangwang @xiao-xiao for all the help and work during this competition. Also thanks for all the answers given during the competition explaining everything and giving awesome ideas!\n\nWithout further ado, the solution mainly focused on three things:\n-> Fine-tuning Albert Large\n-> DataAugmentation\n-> Understanding thresholds.\n\nThe model used was **Albert xLarge version 2** trained on SQUAD2.0 and Fine-tuned on the tiny-dev. One of the important things was the **DataAugmentation** used for training, changing the document_text replacing words by synonyms using WordNet corpus.\n\n\n\nIn here there wasn't no much of magic. We also tried xxLarge only able to run with a huge doc_stride giving bad results. \n\nRemoving html tags helped as well.\n\nUnderstand the data and the output of the model was an important step. \nThe long_answers appeared 50% of the times\nThe short_answers appeared 35% of the times.\n\n\n\nIn this regard, we used threshold values based on the outputs using 50% of all the results for the long_answers e 35% for the short_answers. This was achieve by ordering the list of results, select the number that was in the middle and for the short_answer, 35%. For example, given an output list [1,2,3,4,5] we selected the threshold 3 for the long_answer and 2 for short_answer.\n\nI hope this gives good ideas for future work. \n\nBest Regards,\nPedro Azevedo", "732879": "Congrats!", "733206": "Congratulations!!", "732446": "Sorry, the images were not showing. Fixed the bug! ", "732350": "Congratulations!!\nThanks for sharing your Approach & Insights @pedrojlazevedo ", "732142": "Thanks for your sharing!\nI have a question about how to train Albert xLarge on SQUAD2.0 ?", "732447": "" } }