![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F37166%2F14ef07f17e3b1eda7b98ebac1f6cb46b%2Fdownload.png?generation=1592897946838419&alt=media\")
Thanks a lot to Kaggle and Jigsaw to host this competition. I am very happy to achieve my second solo gold medal within a week :)
\n\nMy main goal when starting this competition was to learn how to use TPUs on Kaggle. I started with Pytorch, switched to Tensorflow, and then switched back to Pytorch. I think overall the usability is still very shaky, and 3 hour limit is also limiting things quite a bit. There is constant progress on it though, and I hope that they will be better usable in the future. I made a separate topic before.
\n\nThe whole test set was visible in this competition, so subbing was very easy and blending many models was very important. Unfortunately, I could not select my best sub because it included a blend with a kernel where no training script was available. I am happy I followed my heart to not select it even though I would have ranked a few spots higher with it. I really would encourage people to stop posting inference only kernels, because logically people will use them if they score well, but the learning potential is quite limited imho.
\n\nThere is not much differences in the models I fit to those that have been posted throughout the competition. I could not get any better results compared to using XLM-Roberta-Large. I also tried uni-language models without any improvements over XLM-Roberta.
\n\nI also do not believe that any advanced sampling technique, label smoothing, augmentation, or other stuff floating around was giving any improvements.
\n\nThe only real important thing was to translate the training data to the languages at hand. This is in line with the experiments reported in the original XLM-Roberta paper.
\n\nI blended models by doing rank-averaged blending.
\n\nI only used provided data and translations of it.\nI think the most untapped potential was to use talk pages from Wikipedia dumps to further train the models. I did not have time to explore this, but I hope someone else did and can report on it.
\n\nI actually think one of the main issues of the provided English data was that it included barely any Bot comments, but the test data contained quite many of them, specifically Turkish which is also why I think the score is better there than for others.
\n\nAfter blending, post-processing was quite important for me to reach high scores. As the metric is a global AUC over six different languages, it became quite quickly obvious to me that there should be room to optimize the global score by shifting predictions of individual languages up or down.
\n\nI tested this on CV with the three languages and managed to get very good boosts there. I further tested how a simulated public and private dataset would behave in terms of optimization, assuming a random split between public and private. So what I did was to repeatedly select random 30% of CV, optimize the thresholds for the 30% and apply them to the remaining 70% and check how far off they would be from the optimal thresholds of the 70%. Overall, I saw that there was not much difference, which is why I was quite confident that I can optimize the thresholds on public LB and they should work well for private LB. Optimizing on CV alone does not work because CV only has 3 languages. This is an example plot of such an analysis:
\n\n
The x-axis is the scaling factor for the language at hand, the y-axis is the improvement of the score over no scaling. The blue line is the improvement of the 30% sample and the orange of the 70% sample. Note that the improvement is not 0 for factor 1.0 in this case as another language has already been optimized.
\n\nAdditionally, I optimized a range of values for which I optimize the scaling factors. I found that not scaling the top predictions improved scores, so I only scaled predictions between 0.25 and 0.875 (rank average values). You can get some further minor improvements by averaging the predictions for similar texts.
\n\nMy whole PP routine boosted scores by roughly 0.004, so quite a lot.
", "messages": [ { "id": "897962", "postDate": "06/23/2020 07:42:23", "content": "Thanks a lot to Kaggle and Jigsaw to host this competition. I am very happy to achieve my second solo gold medal within a week :)
\n\nMy main goal when starting this competition was to learn how to use TPUs on Kaggle. I started with Pytorch, switched to Tensorflow, and then switched back to Pytorch. I think overall the usability is still very shaky, and 3 hour limit is also limiting things quite a bit. There is constant progress on it though, and I hope that they will be better usable in the future. I made a separate topic before.
\n\nThe whole test set was visible in this competition, so subbing was very easy and blending many models was very important. Unfortunately, I could not select my best sub because it included a blend with a kernel where no training script was available. I am happy I followed my heart to not select it even though I would have ranked a few spots higher with it. I really would encourage people to stop posting inference only kernels, because logically people will use them if they score well, but the learning potential is quite limited imho.
\n\nThere is not much differences in the models I fit to those that have been posted throughout the competition. I could not get any better results compared to using XLM-Roberta-Large. I also tried uni-language models without any improvements over XLM-Roberta.
\n\nI also do not believe that any advanced sampling technique, label smoothing, augmentation, or other stuff floating around was giving any improvements.
\n\nThe only real important thing was to translate the training data to the languages at hand. This is in line with the experiments reported in the original XLM-Roberta paper.
\n\nI blended models by doing rank-averaged blending.
\n\nI only used provided data and translations of it.\nI think the most untapped potential was to use talk pages from Wikipedia dumps to further train the models. I did not have time to explore this, but I hope someone else did and can report on it.
\n\nI actually think one of the main issues of the provided English data was that it included barely any Bot comments, but the test data contained quite many of them, specifically Turkish which is also why I think the score is better there than for others.
\n\nAfter blending, post-processing was quite important for me to reach high scores. As the metric is a global AUC over six different languages, it became quite quickly obvious to me that there should be room to optimize the global score by shifting predictions of individual languages up or down.
\n\nI tested this on CV with the three languages and managed to get very good boosts there. I further tested how a simulated public and private dataset would behave in terms of optimization, assuming a random split between public and private. So what I did was to repeatedly select random 30% of CV, optimize the thresholds for the 30% and apply them to the remaining 70% and check how far off they would be from the optimal thresholds of the 70%. Overall, I saw that there was not much difference, which is why I was quite confident that I can optimize the thresholds on public LB and they should work well for private LB. Optimizing on CV alone does not work because CV only has 3 languages. This is an example plot of such an analysis:
\n\n
The x-axis is the scaling factor for the language at hand, the y-axis is the improvement of the score over no scaling. The blue line is the improvement of the 30% sample and the orange of the 70% sample. Note that the improvement is not 0 for factor 1.0 in this case as another language has already been optimized.
\n\nAdditionally, I optimized a range of values for which I optimize the scaling factors. I found that not scaling the top predictions improved scores, so I only scaled predictions between 0.25 and 0.875 (rank average values). You can get some further minor improvements by averaging the predictions for similar texts.
\n\nMy whole PP routine boosted scores by roughly 0.004, so quite a lot.
", "rawMarkdown": "Thanks a lot to Kaggle and Jigsaw to host this competition. I am very happy to achieve my second solo gold medal within a week :)\n\nMy main goal when starting this competition was to learn how to use TPUs on Kaggle. I started with Pytorch, switched to Tensorflow, and then switched back to Pytorch. I think overall the usability is still very shaky, and 3 hour limit is also limiting things quite a bit. There is constant progress on it though, and I hope that they will be better usable in the future. I made a separate [topic](https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/159723) before.\n\nThe whole test set was visible in this competition, so subbing was very easy and blending many models was very important. Unfortunately, I could not select my best sub because it included a blend with a [kernel](https://www.kaggle.com/shonenkov/tpu-inference-super-fast-xlmroberta) where no training script was available. I am happy I followed my heart to not select it even though I would have ranked a few spots higher with it. I really would encourage people to stop posting inference only kernels, because logically people will use them if they score well, but the learning potential is quite limited imho.\n\n### Models\n\nThere is not much differences in the models I fit to those that have been posted throughout the competition. I could not get any better results compared to using XLM-Roberta-Large. I also tried uni-language models without any improvements over XLM-Roberta.\n\nI also do not believe that any advanced sampling technique, label smoothing, augmentation, or other stuff floating around was giving any improvements. \n\nThe only real important thing was to translate the training data to the languages at hand. This is in line with the experiments reported in the original [XLM-Roberta paper](https://arxiv.org/pdf/1911.02116.pdf).\n\nI blended models by doing rank-averaged blending.\n\n### Data\n\nI only used provided data and translations of it.\nI think the most untapped potential was to use talk pages from Wikipedia dumps to further train the models. I did not have time to explore this, but I hope someone else did and can report on it.\n\nI actually think one of the main issues of the provided English data was that it included barely any Bot comments, but the test data contained quite many of them, specifically Turkish which is also why I think the score is better there than for others.\n\n### Post-processing\n\nAfter blending, post-processing was quite important for me to reach high scores. As the metric is a global AUC over six different languages, it became quite quickly obvious to me that there should be room to optimize the global score by shifting predictions of individual languages up or down. \n\nI tested this on CV with the three languages and managed to get very good boosts there. I further tested how a simulated public and private dataset would behave in terms of optimization, assuming a random split between public and private. So what I did was to repeatedly select random 30% of CV, optimize the thresholds for the 30% and apply them to the remaining 70% and check how far off they would be from the optimal thresholds of the 70%. Overall, I saw that there was not much difference, which is why I was quite confident that I can optimize the thresholds on public LB and they should work well for private LB. Optimizing on CV alone does not work because CV only has 3 languages. This is an example plot of such an analysis:\n\n\n\nThe x-axis is the scaling factor for the language at hand, the y-axis is the improvement of the score over no scaling. The blue line is the improvement of the 30% sample and the orange of the 70% sample. Note that the improvement is not 0 for factor 1.0 in this case as another language has already been optimized.\n\nAdditionally, I optimized a range of values for which I optimize the scaling factors. I found that not scaling the top predictions improved scores, so I only scaled predictions between 0.25 and 0.875 (rank average values). You can get some further minor improvements by averaging the predictions for similar texts.\n\nMy whole PP routine boosted scores by roughly 0.004, so quite a lot.", "votes": null }, { "id": "897979", "postDate": "06/23/2020 07:54:23", "content": "@philippsinger congrats!
", "rawMarkdown": "philippsinger congrats!", "votes": null }, { "id": "897997", "postDate": "06/23/2020 08:08:14", "content": "Whoa, that's one killer PP! Thank you for sharing.
", "rawMarkdown": "Whoa, that's one killer PP! Thank you for sharing.", "votes": null }, { "id": "898002", "postDate": "06/23/2020 08:11:36", "content": "Always a pleasure reading your solutions. One question: have you tried subsampling the 0 class? \nAnd again, congratulations on your solo gold. 💪
", "rawMarkdown": "Always a pleasure reading your solutions. One question: have you tried subsampling the 0 class? \nAnd again, congratulations on your solo gold. 💪", "votes": null }, { "id": "898003", "postDate": "06/23/2020 08:11:51", "content": "\n\n\nI started with Pytorch, switched to Tensorflow, and then switched back to Pytorch.
\n
Similar experience, TPU is hard ! Tensorflow -> Pytorch -> Tensorflow -> Pytorch -> ...\nThanks for writeup and congrats!
", "rawMarkdown": "> I started with Pytorch, switched to Tensorflow, and then switched back to Pytorch.\n\nSimilar experience, TPU is hard ! Tensorflow -> Pytorch -> Tensorflow -> Pytorch -> ...\nThanks for writeup and congrats!", "votes": null }, { "id": "898007", "postDate": "06/23/2020 08:18:53", "content": "Congrats on your second solo gold in a week! We used a very similar post processing that explains our boost during last day of competition.
", "rawMarkdown": "Congrats on your second solo gold in a week! We used a very similar post processing that explains our boost during last day of competition.", "votes": null }, { "id": "898035", "postDate": "06/23/2020 08:37:37", "content": "Congratulations! Which CV strategy did you use? Wasn't a 5-fold very computationally expensive?
", "rawMarkdown": "Congratulations! Which CV strategy did you use? Wasn't a 5-fold very computationally expensive?", "votes": null }, { "id": "898039", "postDate": "06/23/2020 08:40:17", "content": "In the end only using public LB.
", "rawMarkdown": "In the end only using public LB.", "votes": null }, { "id": "898044", "postDate": "06/23/2020 08:41:05", "content": "I tried a few sampling techniques and as said I dont think it has any huge impact. I rather did some downsampling in the end to speed up training.
", "rawMarkdown": "I tried a few sampling techniques and as said I dont think it has any huge impact. I rather did some downsampling in the end to speed up training.", "votes": null }, { "id": "898239", "postDate": "06/23/2020 11:50:16", "content": "Congratulations. Seems that almost all of the top teams working with nice post-processing methods.
", "rawMarkdown": "Congratulations. Seems that almost all of the top teams working with nice post-processing methods.", "votes": null }, { "id": "898294", "postDate": "06/23/2020 12:32:31", "content": "Congrats @philippsinger and thanks for sharing.
", "rawMarkdown": "Congrats @philippsinger and thanks for sharing.", "votes": null }, { "id": "898540", "postDate": "06/23/2020 15:21:25", "content": "Congrats, Great job. You went from no solo Gold to two solo Gold in one week. Amazing!
\n\nI like your PP trick. That's clever.
", "rawMarkdown": "Congrats, Great job. You went from no solo Gold to two solo Gold in one week. Amazing! \n\nI like your PP trick. That's clever.", "votes": null }, { "id": "898559", "postDate": "06/23/2020 15:32:37", "content": "Same for us.
", "rawMarkdown": "Same for us.", "votes": null }, { "id": "899924", "postDate": "06/24/2020 14:18:54", "content": "Nice
", "rawMarkdown": "Nice", "votes": null }, { "id": "900043", "postDate": "06/24/2020 15:32:53", "content": "Congratulations :-)
", "rawMarkdown": "Congratulations :-)", "votes": null } ], "comments": [ { "id": 897979, "author_name": "rohitsingh9990", "author_url": "", "post_date": "06/23/2020 07:54:23", "content": "@philippsinger congrats!
", "votes": null, "replies": [] }, { "id": 897997, "author_name": "ilhamfp31", "author_url": "", "post_date": "06/23/2020 08:08:14", "content": "Whoa, that's one killer PP! Thank you for sharing.
", "votes": null, "replies": [] }, { "id": 898002, "author_name": "yassinealouini", "author_url": "", "post_date": "06/23/2020 08:11:36", "content": "Always a pleasure reading your solutions. One question: have you tried subsampling the 0 class? \nAnd again, congratulations on your solo gold. 💪
", "votes": null, "replies": [ { "id": 898044, "author_name": "philippsinger", "author_url": "", "post_date": "06/23/2020 08:41:05", "content": "I tried a few sampling techniques and as said I dont think it has any huge impact. I rather did some downsampling in the end to speed up training.
", "votes": null, "replies": [] } ] }, { "id": 898003, "author_name": "mzr2017", "author_url": "", "post_date": "06/23/2020 08:11:51", "content": "\n\n\nI started with Pytorch, switched to Tensorflow, and then switched back to Pytorch.
\n
Similar experience, TPU is hard ! Tensorflow -> Pytorch -> Tensorflow -> Pytorch -> ...\nThanks for writeup and congrats!
", "votes": null, "replies": [] }, { "id": 898007, "author_name": "cpmpml", "author_url": "", "post_date": "06/23/2020 08:18:53", "content": "Congrats on your second solo gold in a week! We used a very similar post processing that explains our boost during last day of competition.
", "votes": null, "replies": [] }, { "id": 898035, "author_name": "rftexas", "author_url": "", "post_date": "06/23/2020 08:37:37", "content": "Congratulations! Which CV strategy did you use? Wasn't a 5-fold very computationally expensive?
", "votes": null, "replies": [ { "id": 898039, "author_name": "philippsinger", "author_url": "", "post_date": "06/23/2020 08:40:17", "content": "In the end only using public LB.
", "votes": null, "replies": [] }, { "id": 898559, "author_name": "rftexas", "author_url": "", "post_date": "06/23/2020 15:32:37", "content": "Same for us.
", "votes": null, "replies": [] } ] }, { "id": 898239, "author_name": "tonyxu", "author_url": "", "post_date": "06/23/2020 11:50:16", "content": "Congratulations. Seems that almost all of the top teams working with nice post-processing methods.
", "votes": null, "replies": [] }, { "id": 898294, "author_name": "sheriytm", "author_url": "", "post_date": "06/23/2020 12:32:31", "content": "Congrats @philippsinger and thanks for sharing.
", "votes": null, "replies": [] }, { "id": 898540, "author_name": "cdeotte", "author_url": "", "post_date": "06/23/2020 15:21:25", "content": "Congrats, Great job. You went from no solo Gold to two solo Gold in one week. Amazing!
\n\nI like your PP trick. That's clever.
", "votes": null, "replies": [] }, { "id": 899924, "author_name": "", "author_url": "", "post_date": "06/24/2020 14:18:54", "content": "Nice
", "votes": null, "replies": [] }, { "id": 900043, "author_name": "aaroha33", "author_url": "", "post_date": "06/24/2020 15:32:53", "content": "Congratulations :-)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "897962": "Thanks a lot to Kaggle and Jigsaw to host this competition. I am very happy to achieve my second solo gold medal within a week :)\n\nMy main goal when starting this competition was to learn how to use TPUs on Kaggle. I started with Pytorch, switched to Tensorflow, and then switched back to Pytorch. I think overall the usability is still very shaky, and 3 hour limit is also limiting things quite a bit. There is constant progress on it though, and I hope that they will be better usable in the future. I made a separate [topic](https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/159723) before.\n\nThe whole test set was visible in this competition, so subbing was very easy and blending many models was very important. Unfortunately, I could not select my best sub because it included a blend with a [kernel](https://www.kaggle.com/shonenkov/tpu-inference-super-fast-xlmroberta) where no training script was available. I am happy I followed my heart to not select it even though I would have ranked a few spots higher with it. I really would encourage people to stop posting inference only kernels, because logically people will use them if they score well, but the learning potential is quite limited imho.\n\n### Models\n\nThere is not much differences in the models I fit to those that have been posted throughout the competition. I could not get any better results compared to using XLM-Roberta-Large. I also tried uni-language models without any improvements over XLM-Roberta.\n\nI also do not believe that any advanced sampling technique, label smoothing, augmentation, or other stuff floating around was giving any improvements. \n\nThe only real important thing was to translate the training data to the languages at hand. This is in line with the experiments reported in the original [XLM-Roberta paper](https://arxiv.org/pdf/1911.02116.pdf).\n\nI blended models by doing rank-averaged blending.\n\n### Data\n\nI only used provided data and translations of it.\nI think the most untapped potential was to use talk pages from Wikipedia dumps to further train the models. I did not have time to explore this, but I hope someone else did and can report on it.\n\nI actually think one of the main issues of the provided English data was that it included barely any Bot comments, but the test data contained quite many of them, specifically Turkish which is also why I think the score is better there than for others.\n\n### Post-processing\n\nAfter blending, post-processing was quite important for me to reach high scores. As the metric is a global AUC over six different languages, it became quite quickly obvious to me that there should be room to optimize the global score by shifting predictions of individual languages up or down. \n\nI tested this on CV with the three languages and managed to get very good boosts there. I further tested how a simulated public and private dataset would behave in terms of optimization, assuming a random split between public and private. So what I did was to repeatedly select random 30% of CV, optimize the thresholds for the 30% and apply them to the remaining 70% and check how far off they would be from the optimal thresholds of the 70%. Overall, I saw that there was not much difference, which is why I was quite confident that I can optimize the thresholds on public LB and they should work well for private LB. Optimizing on CV alone does not work because CV only has 3 languages. This is an example plot of such an analysis:\n\n\n\nThe x-axis is the scaling factor for the language at hand, the y-axis is the improvement of the score over no scaling. The blue line is the improvement of the 30% sample and the orange of the 70% sample. Note that the improvement is not 0 for factor 1.0 in this case as another language has already been optimized.\n\nAdditionally, I optimized a range of values for which I optimize the scaling factors. I found that not scaling the top predictions improved scores, so I only scaled predictions between 0.25 and 0.875 (rank average values). You can get some further minor improvements by averaging the predictions for similar texts.\n\nMy whole PP routine boosted scores by roughly 0.004, so quite a lot.", "897979": "philippsinger congrats!", "897997": "Whoa, that's one killer PP! Thank you for sharing.", "898002": "Always a pleasure reading your solutions. One question: have you tried subsampling the 0 class? \nAnd again, congratulations on your solo gold. 💪", "898003": "> I started with Pytorch, switched to Tensorflow, and then switched back to Pytorch.\n\nSimilar experience, TPU is hard ! Tensorflow -> Pytorch -> Tensorflow -> Pytorch -> ...\nThanks for writeup and congrats!", "898007": "Congrats on your second solo gold in a week! We used a very similar post processing that explains our boost during last day of competition.", "898035": "Congratulations! Which CV strategy did you use? Wasn't a 5-fold very computationally expensive?", "898039": "In the end only using public LB.", "898044": "I tried a few sampling techniques and as said I dont think it has any huge impact. I rather did some downsampling in the end to speed up training.", "898239": "Congratulations. Seems that almost all of the top teams working with nice post-processing methods.", "898294": "Congrats @philippsinger and thanks for sharing.", "898540": "Congrats, Great job. You went from no solo Gold to two solo Gold in one week. Amazing! \n\nI like your PP trick. That's clever.", "898559": "Same for us.", "899924": "Nice", "900043": "Congratulations :-)" }, "source": "meta" }