![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1424766%2Fd03ded105462fb5ba610393ba97e4ad8%2Fjigsaw%20arch2.001.jpeg?generation=1592909197036996&alt=media\")
Thanks to kaggle for hosting such an interesting and challenging competition! Multilingual NLP is something very interesting yet difficult. In hindsight I (Dieter) wish Iâd spend less time on the tweet sentiment extraction competition and more on this one.
\n\nOur solution is a simple blend of several transformer models (mainly xlm-roberta-large) paired with some post-processing. The architecture was the same as in public available kernels with the classification head taking either max+mean pooling of hidden states or the hidden state of the CLS token. We used a 3-step approach for training our models, where starting from 7 languages we fine-tuned to 3 languages and finished at a single language. As this 3-step approach results in distortion of global predictions we shifted languages individually by a factor as a post-processing step.
\n\nWe are still astonished by the great result in such a short time. While @aerdem4 worked a bit longer on this competition and already had a good understanding of specific challenges @cpmpml and @christofhenkel joined right after the tweet sentiment competition a few hours before team merger deadline. Two days ago we were still at a 100+ spot and were hoping for a silver medal at best. But with persistence and the right amount of intuition what ideas to proceed with, and of course some luck we managed to climb right to the upper gold position.
\n\nNone. That's NLP in 2020 :D
\n\nWe did two things. One was to downsample negative samples to get a more balanced dataset and a smaller dataset at the same time. If you only use as many negative as positive then dataset is reduced 5x roughly, same for time to run one epoch. Some of our models were trained that way.
\n\nAnother speedup came from padding by batch. The main idea is to limit the amount of padding to what is necessary for the current batch. Inputs are padded to the length of the longest input in the batch rather than a fixed length, say 512. This is now a well known technique, and it has been used in previous competitions. It accelerates training significantly.
\nWe refined the idea by sorting the samples by their length so that the samples in a given batch have similar length. This reduces even further the need for padding. If all inputs in the batch have the same length then there is no padding at all.
Given samples are sorted, we cannot shuffle them in training mode. We rather shuffled batches. This yields an extra 2x speedup compared to the original padding by batch. Training one epoch for xlm-roberta-large on the first train set takes about 17 minutes on a V100 GPU.
\n\nAs time was short and hence we did not have many submissions to spare for getting LB feedback we worked on getting a reliable cross-validation. We used a mix of Group-3fold per language and simple 3fold of the validation data to represent the unknown languages ru,pt,fr as well as the known languages it, es and tr. which results in a 6fold scheme. So do for example fold1 represent what if es would not be in valid.csv which should behave the same as pt is not in valid. Taking the mean AUC of all folds was a very good proxy for Public and Private LB.
\n\nAs most teams we used xlm-roberta large as the main backbone with a simple classification head that either uses max+mean pooling or the hidden states of the CLS token. In total we used the following backbones with their approximate weighting in our final ensemble in brackets.
\n\nOne main ingredient for our good result is a stepwise finetuning towards single language models for tr, it and es. Let us illustrate the whole approach before explaining:
\n\n
As input data we used the english jigsaw-toxic-comment-train.csv and its six translations available as public datasets which combined are roughly 1.4M comments. I realized that training a model directly on the combined dataset is not optimal, as each comment appears 7x in each epoch (although in different languages) and the model overfits on that. So I divided the combined dataset into 7 stratified parts, where each part contains a comment only once. For each fold we then finetuned a transformer in a 3step manner:
\n\nWe then use the step1 model for predicting ru, the step2 model for predicting pt and fr and the respective step3 models for tr, it and es. Using the step2 model for pt and fr gave a significant boost compared to using step1 model for those due. Most likely due to the language similarity between it, es, fr and pt.
\n\nWe used mainly a batchsize of 32 using gradient accumulation and a learning rate of 3e-6 with linear decay with AdamW optimizer. One thing worth mentioning is that we train on a max sequence length of 512 due to the dataloader mentioned above which sorts the data by length and then randomly serves batches of same length comments. Only the huge speed-up paired with the non padding of shorter comments made using a 512 max length reasonable.
\n\nApart from one model which was built on top of a public kernel, all models were trained using pytorch and GPU.
\n\nAs for ensembling we did team member individual ensembling and then combined the models of the team members and public kernels by weighted sum of rank percentile
\n\nAnother key ingredient, which we luckily found on the last day is post processing of the final predictions. Although languages are derived from individual models, the competition metric is sensitive to the global rank of all predictions and not language specific. So we took care of the languages having a correct relation to each other, by shifting the predictions of each language individually. In our final submission for example we used the following factors
\n\n\ntest_df.loc[test_df[\"lang\"] == \"es\", \"toxic\"] *= 1.06\ntest_df.loc[test_df[\"lang\"] == \"fr\", \"toxic\"] *= 1.04\ntest_df.loc[test_df[\"lang\"] == \"it\", \"toxic\"] *= 0.97\ntest_df.loc[test_df[\"lang\"] == \"pt\", \"toxic\"] *= 0.96\ntest_df.loc[test_df[\"lang\"] == \"tr\", \"toxic\"] *= 0.98\n
We derived the factors by matching the test prediction mean with the public LB mean for each language individually. We had already made 6 submissions for probing the test set by setting one language to 1 and the rest to 0. Then we made sure that means are aligned to give similar experimental AUC. That posprocessing moved us from 14th place to 4th place on public LB!
\n\nWhile this postprocessing might also help other teams, we think it specifically fixes the divergence of global prediction distributions introduced by having 5 models to predict 6 languages, and might not help much if a team used a single model approach.
\n\nThanks for reading. Questions welcome.
", "messages": [ { "id": "898155", "postDate": "06/23/2020 10:48:17", "content": "Thanks to kaggle for hosting such an interesting and challenging competition! Multilingual NLP is something very interesting yet difficult. In hindsight I (Dieter) wish Iâd spend less time on the tweet sentiment extraction competition and more on this one.
\n\nOur solution is a simple blend of several transformer models (mainly xlm-roberta-large) paired with some post-processing. The architecture was the same as in public available kernels with the classification head taking either max+mean pooling of hidden states or the hidden state of the CLS token. We used a 3-step approach for training our models, where starting from 7 languages we fine-tuned to 3 languages and finished at a single language. As this 3-step approach results in distortion of global predictions we shifted languages individually by a factor as a post-processing step.
\n\nWe are still astonished by the great result in such a short time. While @aerdem4 worked a bit longer on this competition and already had a good understanding of specific challenges @cpmpml and @christofhenkel joined right after the tweet sentiment competition a few hours before team merger deadline. Two days ago we were still at a 100+ spot and were hoping for a silver medal at best. But with persistence and the right amount of intuition what ideas to proceed with, and of course some luck we managed to climb right to the upper gold position.
\n\nNone. That's NLP in 2020 :D
\n\nWe did two things. One was to downsample negative samples to get a more balanced dataset and a smaller dataset at the same time. If you only use as many negative as positive then dataset is reduced 5x roughly, same for time to run one epoch. Some of our models were trained that way.
\n\nAnother speedup came from padding by batch. The main idea is to limit the amount of padding to what is necessary for the current batch. Inputs are padded to the length of the longest input in the batch rather than a fixed length, say 512. This is now a well known technique, and it has been used in previous competitions. It accelerates training significantly.
\nWe refined the idea by sorting the samples by their length so that the samples in a given batch have similar length. This reduces even further the need for padding. If all inputs in the batch have the same length then there is no padding at all.
Given samples are sorted, we cannot shuffle them in training mode. We rather shuffled batches. This yields an extra 2x speedup compared to the original padding by batch. Training one epoch for xlm-roberta-large on the first train set takes about 17 minutes on a V100 GPU.
\n\nAs time was short and hence we did not have many submissions to spare for getting LB feedback we worked on getting a reliable cross-validation. We used a mix of Group-3fold per language and simple 3fold of the validation data to represent the unknown languages ru,pt,fr as well as the known languages it, es and tr. which results in a 6fold scheme. So do for example fold1 represent what if es would not be in valid.csv which should behave the same as pt is not in valid. Taking the mean AUC of all folds was a very good proxy for Public and Private LB.
\n\nAs most teams we used xlm-roberta large as the main backbone with a simple classification head that either uses max+mean pooling or the hidden states of the CLS token. In total we used the following backbones with their approximate weighting in our final ensemble in brackets.
\n\nOne main ingredient for our good result is a stepwise finetuning towards single language models for tr, it and es. Let us illustrate the whole approach before explaining:
\n\n
As input data we used the english jigsaw-toxic-comment-train.csv and its six translations available as public datasets which combined are roughly 1.4M comments. I realized that training a model directly on the combined dataset is not optimal, as each comment appears 7x in each epoch (although in different languages) and the model overfits on that. So I divided the combined dataset into 7 stratified parts, where each part contains a comment only once. For each fold we then finetuned a transformer in a 3step manner:
\n\nWe then use the step1 model for predicting ru, the step2 model for predicting pt and fr and the respective step3 models for tr, it and es. Using the step2 model for pt and fr gave a significant boost compared to using step1 model for those due. Most likely due to the language similarity between it, es, fr and pt.
\n\nWe used mainly a batchsize of 32 using gradient accumulation and a learning rate of 3e-6 with linear decay with AdamW optimizer. One thing worth mentioning is that we train on a max sequence length of 512 due to the dataloader mentioned above which sorts the data by length and then randomly serves batches of same length comments. Only the huge speed-up paired with the non padding of shorter comments made using a 512 max length reasonable.
\n\nApart from one model which was built on top of a public kernel, all models were trained using pytorch and GPU.
\n\nAs for ensembling we did team member individual ensembling and then combined the models of the team members and public kernels by weighted sum of rank percentile
\n\nAnother key ingredient, which we luckily found on the last day is post processing of the final predictions. Although languages are derived from individual models, the competition metric is sensitive to the global rank of all predictions and not language specific. So we took care of the languages having a correct relation to each other, by shifting the predictions of each language individually. In our final submission for example we used the following factors
\n\n\ntest_df.loc[test_df[\"lang\"] == \"es\", \"toxic\"] *= 1.06\ntest_df.loc[test_df[\"lang\"] == \"fr\", \"toxic\"] *= 1.04\ntest_df.loc[test_df[\"lang\"] == \"it\", \"toxic\"] *= 0.97\ntest_df.loc[test_df[\"lang\"] == \"pt\", \"toxic\"] *= 0.96\ntest_df.loc[test_df[\"lang\"] == \"tr\", \"toxic\"] *= 0.98\n
We derived the factors by matching the test prediction mean with the public LB mean for each language individually. We had already made 6 submissions for probing the test set by setting one language to 1 and the rest to 0. Then we made sure that means are aligned to give similar experimental AUC. That posprocessing moved us from 14th place to 4th place on public LB!
\n\nWhile this postprocessing might also help other teams, we think it specifically fixes the divergence of global prediction distributions introduced by having 5 models to predict 6 languages, and might not help much if a team used a single model approach.
\n\nThanks for reading. Questions welcome.
", "rawMarkdown": "Thanks to kaggle for hosting such an interesting and challenging competition! Multilingual NLP is something very interesting yet difficult. In hindsight I (Dieter) wish Iâd spend less time on the tweet sentiment extraction competition and more on this one. \n\n# Brief Summary\nOur solution is a simple blend of several transformer models (mainly xlm-roberta-large) paired with some post-processing. The architecture was the same as in public available kernels with the classification head taking either max+mean pooling of hidden states or the hidden state of the CLS token. We used a 3-step approach for training our models, where starting from 7 languages we fine-tuned to 3 languages and finished at a single language. As this 3-step approach results in distortion of global predictions we shifted languages individually by a factor as a post-processing step. \n\n# Detailed Summary\nWe are still astonished by the great result in such a short time. While @aerdem4 worked a bit longer on this competition and already had a good understanding of specific challenges @cpmpml and @christofhenkel joined right after the tweet sentiment competition a few hours before team merger deadline. Two days ago we were still at a 100+ spot and were hoping for a silver medal at best. But with persistence and the right amount of intuition what ideas to proceed with, and of course some luck we managed to climb right to the upper gold position.\n\n## Preprocessing\nNone. That's NLP in 2020 :D\n\n## Speeding up training\nWe did two things. One was to downsample negative samples to get a more balanced dataset and a smaller dataset at the same time. If you only use as many negative as positive then dataset is reduced 5x roughly, same for time to run one epoch. Some of our models were trained that way.\n\nAnother speedup came from padding by batch. The main idea is to limit the amount of padding to what is necessary for the current batch. Inputs are padded to the length of the longest input in the batch rather than a fixed length, say 512. This is now a well known technique, and it has been used in previous competitions. It accelerates training significantly. \nWe refined the idea by sorting the samples by their length so that the samples in a given batch have similar length. This reduces even further the need for padding. If all inputs in the batch have the same length then there is no padding at all. \n\nGiven samples are sorted, we cannot shuffle them in training mode. We rather shuffled batches. This yields an extra 2x speedup compared to the original padding by batch. Training one epoch for xlm-roberta-large on the first train set takes about 17 minutes on a V100 GPU.\n\n## Cross-validation\nAs time was short and hence we did not have many submissions to spare for getting LB feedback we worked on getting a reliable cross-validation. We used a mix of Group-3fold per language and simple 3fold of the validation data to represent the unknown languages ru,pt,fr as well as the known languages it, es and tr. which results in a 6fold scheme. So do for example fold1 represent what if es would not be in valid.csv which should behave the same as pt is not in valid. Taking the mean AUC of all folds was a very good proxy for Public and Private LB.\n\n## Architectures\nAs most teams we used xlm-roberta large as the main backbone with a simple classification head that either uses max+mean pooling or the hidden states of the CLS token. In total we used the following backbones with their approximate weighting in our final ensemble in brackets.\n \n- 5x xlm-roberta-large (85%)\n- 1x xlm-roberta-base (5%)\n- 1x mBart-large (10%)\n\n## Training strategy\nOne main ingredient for our good result is a stepwise finetuning towards single language models for tr, it and es. Let us illustrate the whole approach before explaining:\n\n\n\nAs input data we used the english jigsaw-toxic-comment-train.csv and its six translations available as public datasets which combined are roughly 1.4M comments. I realized that training a model directly on the combined dataset is not optimal, as each comment appears 7x in each epoch (although in different languages) and the model overfits on that. So I divided the combined dataset into 7 stratified parts, where each part contains a comment only once. For each fold we then finetuned a transformer in a 3step manner:\n\n\n- Step 1: finetune to all 7 languages for 2 epochs\n- Step 2: finetune only to the full valid.csv which only has tr, it and es\n- Step 3: 3x finetune to each language in valid resulting in 3 models\n\nWe then use the step1 model for predicting ru, the step2 model for predicting pt and fr and the respective step3 models for tr, it and es. Using the step2 model for pt and fr gave a significant boost compared to using step1 model for those due. Most likely due to the language similarity between it, es, fr and pt.\n\nWe used mainly a batchsize of 32 using gradient accumulation and a learning rate of 3e-6 with linear decay with AdamW optimizer. One thing worth mentioning is that we train on a max sequence length of 512 due to the dataloader mentioned above which sorts the data by length and then randomly serves batches of same length comments. Only the huge speed-up paired with the non padding of shorter comments made using a 512 max length reasonable. \n\nApart from one model which was built on top of a public kernel, all models were trained using pytorch and GPU.\n\n## Ensembling\nAs for ensembling we did team member individual ensembling and then combined the models of the team members and public kernels by weighted sum of rank percentile \n\n## Post-processing\nAnother key ingredient, which we luckily found on the last day is post processing of the final predictions. Although languages are derived from individual models, the competition metric is sensitive to the global rank of all predictions and not language specific. So we took care of the languages having a correct relation to each other, by shifting the predictions of each language individually. In our final submission for example we used the following factors \n\n```\ntest_df.loc[test_df[\"lang\"] == \"es\", \"toxic\"] *= 1.06\ntest_df.loc[test_df[\"lang\"] == \"fr\", \"toxic\"] *= 1.04\ntest_df.loc[test_df[\"lang\"] == \"it\", \"toxic\"] *= 0.97\ntest_df.loc[test_df[\"lang\"] == \"pt\", \"toxic\"] *= 0.96\ntest_df.loc[test_df[\"lang\"] == \"tr\", \"toxic\"] *= 0.98\n```\n\n\nWe derived the factors by matching the test prediction mean with the public LB mean for each language individually. We had already made 6 submissions for probing the test set by setting one language to 1 and the rest to 0. Then we made sure that means are aligned to give similar experimental AUC. That posprocessing moved us from 14th place to 4th place on public LB! \n\nWhile this postprocessing might also help other teams, we think it specifically fixes the divergence of global prediction distributions introduced by having 5 models to predict 6 languages, and might not help much if a team used a single model approach.\n\nThanks for reading. Questions welcome.", "votes": null }, { "id": "898189", "postDate": "06/23/2020 11:16:22", "content": "Post Proc ... Genius! Thank you for your exp!
", "rawMarkdown": "Post Proc ... Genius! Thank you for your exp!", "votes": null }, { "id": "898201", "postDate": "06/23/2020 11:21:02", "content": "Question: so, how was the architecture graph done? ð \nMore seriously: what hardware have you used, i.e. your own or cloud? Also, have you tried TPU (I guess not since you only mention V100)? \nGreat achievement and a lot to learn from!
", "rawMarkdown": "Question: so, how was the architecture graph done? ð \nMore seriously: what hardware have you used, i.e. your own or cloud? Also, have you tried TPU (I guess not since you only mention V100)? \nGreat achievement and a lot to learn from!", "votes": null }, { "id": "898221", "postDate": "06/23/2020 11:32:46", "content": "Thanks for sharing your great approach!\nI used the normal 1 model to predict 6 languages approach with xlmr-large models. I got +0.008 improvement with your post processing.
\n| | public score| private score |\n| --- | --- | --- |\n|xlmr-large my blending | 0.9487 |0.9471 |\n| xlmr-large my blending + your post processing | 0.9494|0.9479|
Amazing work...I never thought your team would boost from 9500 to 9522 in the last dayð . Can you share some details about how your team found this post-processing trick?
", "rawMarkdown": "Amazing work...I never thought your team would boost from 9500 to 9522 in the last dayð . Can you share some details about how your team found this post-processing trick?", "votes": null }, { "id": "898233", "postDate": "06/23/2020 11:46:23", "content": "To me it made sense that global ranking matters, and especially the 'ru' prediction should be off as the only thing the step1 model has seen were translations from en, which I think are weaker in terms of toxicity as original russian comments. Hence we expected those predictions to be in average too low. Then we checked shifting the predictions on the validation set, and it gave a significant improvement in score. So we worked on a method to shift test predictions too...
", "rawMarkdown": "To me it made sense that global ranking matters, and especially the 'ru' prediction *should* be off as the only thing the step1 model has seen were translations from en, which I think are weaker in terms of toxicity as original russian comments. Hence we expected those predictions to be in average too low. Then we checked shifting the predictions on the validation set, and it gave a significant improvement in score. So we worked on a method to shift test predictions too...", "votes": null }, { "id": "898247", "postDate": "06/23/2020 11:54:54", "content": "Congrats for one week journey and anther gold medal!\nWe also find each language need calibration in the last day, but we have got no idea about the coef in only 5 submissions, we just try some lucky coef and get a small boost on post processing.\nAmazing work and thanks for detailed writeup!
", "rawMarkdown": "Congrats for one week journey and anther gold medal!\nWe also find each language need calibration in the last day, but we have got no idea about the coef in only 5 submissions, we just try some lucky coef and get a small boost on post processing.\nAmazing work and thanks for detailed writeup!", "votes": null }, { "id": "898248", "postDate": "06/23/2020 11:55:52", "content": "Thank you for your explanation!!!\nHope to learn more from you
", "rawMarkdown": "Thank you for your explanation!!!\nHope to learn more from you", "votes": null }, { "id": "898250", "postDate": "06/23/2020 11:56:01", "content": "The post-processing is real magic! My model got +0.006 and +0.011 improvement when I just multiple Spanish by 1.2 .
", "rawMarkdown": "The post-processing is real magic! My model got +0.006 and +0.011 improvement when I just multiple Spanish by 1.2 .", "votes": null }, { "id": "898275", "postDate": "06/23/2020 12:20:12", "content": "Congrats @christofhenkel, @cpmpml, @aerdem4 and thanks for sharing a very good solution writeup.
", "rawMarkdown": "Congrats @christofhenkel, @cpmpml, @aerdem4 and thanks for sharing a very good solution writeup.", "votes": null }, { "id": "898285", "postDate": "06/23/2020 12:23:45", "content": "So many brilliant ideas! Thanks for sharing.
\n\nOn the ensembling, is the weighted sum of rank percentile the method of choice to try first, from experience, or just one from a toolbox ?
You forget @aerdem4 ;)
", "rawMarkdown": "You forget @aerdem4 ;)", "votes": null }, { "id": "898290", "postDate": "06/23/2020 12:27:53", "content": "@cpmpml, amended :-)
", "rawMarkdown": "cpmpml, amended :-)", "votes": null }, { "id": "898292", "postDate": "06/23/2020 12:30:07", "content": "@cpmpml, it must be the Tweet and Jigsaw exhaustion. @aerdem4 may agree :-)
", "rawMarkdown": "cpmpml, it must be the Tweet and Jigsaw exhaustion. @aerdem4 may agree :-)", "votes": null }, { "id": "898373", "postDate": "06/23/2020 13:41:24", "content": "Dieter and team, very discipline and intelligent move indeed. Watching you and your team come after us from behind is akin watching a horror movie. What a great chase. Thanks very much for sharing the insights. Dr.
", "rawMarkdown": "Dieter and team, very discipline and intelligent move indeed. Watching you and your team come after us from behind is akin watching a horror movie. What a great chase. Thanks very much for sharing the insights. Dr.", "votes": null }, { "id": "898388", "postDate": "06/23/2020 13:50:51", "content": "Ahmet used TPU, Dieter and I used GPU (V100). The former used Keras/TF while the latter used Pytorch. There may be a correlation there ;)
", "rawMarkdown": "Ahmet used TPU, Dieter and I used GPU (V100). The former used Keras/TF while the latter used Pytorch. There may be a correlation there ;)", "votes": null }, { "id": "898529", "postDate": "06/23/2020 15:14:41", "content": "Great model, congrats on building such a great model in a short time.
\n\nWow, nice post post processing trick. I just applied it to my final model. It increased both public and private LB by 0.008 and increased my private rank from 61st to 24th. Wow.
", "rawMarkdown": "Great model, congrats on building such a great model in a short time. \n\nWow, nice post post processing trick. I just applied it to my final model. It increased both public and private LB by 0.008 and increased my private rank from 61st to 24th. Wow.", "votes": null }, { "id": "898586", "postDate": "06/23/2020 15:52:59", "content": "Makes sense. :)
", "rawMarkdown": "Makes sense. :)", "votes": null }, { "id": "898597", "postDate": "06/23/2020 16:02:47", "content": "I am sure if you would optimize it further on public part you could get even higher. The optimal scaling factors can differ quite a bit depending on how your predictions look like.
", "rawMarkdown": "I am sure if you would optimize it further on public part you could get even higher. The optimal scaling factors can differ quite a bit depending on how your predictions look like.", "votes": null }, { "id": "899129", "postDate": "06/24/2020 02:29:35", "content": "Great Post-processing Guys !
", "rawMarkdown": "Great Post-processing Guys !", "votes": null }, { "id": "899200", "postDate": "06/24/2020 04:44:14", "content": "Congratulations sir.
", "rawMarkdown": "Congratulations sir.", "votes": null }, { "id": "899414", "postDate": "06/24/2020 08:18:06", "content": "Congrats and thanks for sharing! All of the methods you've shared are so cool and effective, which have opened my eyes! I also tried to apply your post-processing to my final model. It increased my private rank from 23rd to 13thïŒI'd never thought of such a fast but effective trick to improve score before. That's really awesome! \nThere's so much more I need to learn.\nMy takeway from recent deep learning competitions is that postprocessing is the best way to improve score, more than NN architecture. I'm serious.
", "rawMarkdown": "My takeway from recent deep learning competitions is that postprocessing is the best way to improve score, more than NN architecture. I'm serious.", "votes": null }, { "id": "899668", "postDate": "06/24/2020 11:20:26", "content": "@cpmpml happened for both jigsaw and twitter. And suprisingly both were expected to have big shakeups and ended up having nothing.
", "rawMarkdown": "cpmpml happened for both jigsaw and twitter. And suprisingly both were expected to have big shakeups and ended up having nothing.", "votes": null }, { "id": "899676", "postDate": "06/24/2020 11:33:56", "content": "When everyone is working off the same set of pretrained models and knowledge of common architectural tweaks, postprocessing is one of the few ways to distinguish your models.
\n\nThe main downside with PP is that it's easy to do it at the last minute so people can see their leads evaporate literally overnight.
", "rawMarkdown": "When everyone is working off the same set of pretrained models and knowledge of common architectural tweaks, postprocessing is one of the few ways to distinguish your models. \n\nThe main downside with PP is that it's easy to do it at the last minute so people can see their leads evaporate literally overnight.", "votes": null }, { "id": "899680", "postDate": "06/24/2020 11:38:00", "content": "@shahules it also happened in Bengali
\n\n@leecming sometimes the pp is tricky. In Tweet sentiment I didn't found it nor did @philippsinger for instance.
", "rawMarkdown": "shahules it also happened in Bengali\n\n@leecming sometimes the pp is tricky. In Tweet sentiment I didn't found it nor did @philippsinger for instance.", "votes": null }, { "id": "899682", "postDate": "06/24/2020 11:38:53", "content": "You shoudl try many methods and see their effect. But when roc-auc is the metric, only rank matter, hence using rank makes sense.
", "rawMarkdown": "You shoudl try many methods and see their effect. But when roc-auc is the metric, only rank matter, hence using rank makes sense.", "votes": null }, { "id": "899704", "postDate": "06/24/2020 11:58:00", "content": "Yeah, I found/profited from it here, but did not in Tweets.
", "rawMarkdown": "Yeah, I found/profited from it here, but did not in Tweets.", "votes": null }, { "id": "899713", "postDate": "06/24/2020 12:03:48", "content": "And to be fair, if we would have had more time, I would have tried to incorporate fixing the disturbed individual language distributions in the modelling, instead of fixing it via pp. Often post-processing fixes the symptoms but does not solve the root cause.
", "rawMarkdown": "And to be fair, if we would have had more time, I would have tried to incorporate fixing the disturbed individual language distributions in the modelling, instead of fixing it via pp. Often post-processing fixes the symptoms but does not solve the root cause.", "votes": null }, { "id": "899741", "postDate": "06/24/2020 12:24:07", "content": "You still need to adjust for test distributions. So in both ways you are doing some form of test pp, even if it is from a different perspective.
", "rawMarkdown": "You still need to adjust for test distributions. So in both ways you are doing some form of test pp, even if it is from a different perspective.", "votes": null }, { "id": "899747", "postDate": "06/24/2020 12:28:51", "content": "We wouldn't postprocess without LB probing. If Kaggle splits the public/private not random, it could be much different. In the past, Kaggle had more tricky splits.
", "rawMarkdown": "We wouldn't postprocess without LB probing. If Kaggle splits the public/private not random, it could be much different. In the past, Kaggle had more tricky splits.", "votes": null }, { "id": "899884", "postDate": "06/24/2020 13:56:34", "content": "Yes, I wanted to make a longer post about it but pub LB should probably not have included all languages, and private test should have been completely hidden. I still dont understand the technical argument of why TPUs need to have the full test data available.
", "rawMarkdown": "Yes, I wanted to make a longer post about it but pub LB should probably not have included all languages, and private test should have been completely hidden. I still dont understand the technical argument of why TPUs need to have the full test data available.", "votes": null }, { "id": "904323", "postDate": "06/27/2020 14:31:15", "content": "I got 0.9499 on private LB by just applying the PP trick to my (not chosen) best model ( from 0.9478 to 0.9499 )
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F974295%2Fb81459e076acd5500bd0df2ce56ea854%2F11Capture.PNG?generation=1593268260582472&alt=media\")
yeah, its a good pp trick :D
", "rawMarkdown": "yeah, its a good pp trick :D", "votes": null }, { "id": "910784", "postDate": "07/01/2020 11:22:15", "content": "Congratulation! Super solid solution and really loved it, may I ask you how you implemented the padding by batch trick in detail? It would be great if you can share any links for refs, sort of new to NLP and BERT so need some help. By the way around the last 2 days, we were at 14th and then you guys jumped up over us, it makes losing a gold medal not that suffering if it was you guys who outplayed our team LOL, thanks for sharing your cool ideas.
", "rawMarkdown": "Congratulation! Super solid solution and really loved it, may I ask you how you implemented the padding by batch trick in detail? It would be great if you can share any links for refs, sort of new to NLP and BERT so need some help. By the way around the last 2 days, we were at 14th and then you guys jumped up over us, it makes losing a gold medal not that suffering if it was you guys who outplayed our team LOL, thanks for sharing your cool ideas.", "votes": null }, { "id": "916408", "postDate": "07/05/2020 15:50:36", "content": "Congrats! \nAnd sorry for late asking.\nI'm still confused about the way from 6 single language submissions to pp factors, could you sharing more details?
", "rawMarkdown": "Congrats! \nAnd sorry for late asking.\nI'm still confused about the way from 6 single language submissions to pp factors, could you sharing more details?", "votes": null }, { "id": "1495839", "postDate": "08/29/2021 20:37:19", "content": "Thanks for sharing!! very helpful
", "rawMarkdown": "Thanks for sharing!! very helpful", "votes": null }, { "id": "1574214", "postDate": "11/07/2021 10:43:22", "content": "Is your BERT model able to detect negation and sarcasm?
", "rawMarkdown": "Is your BERT model able to detect negation and sarcasm?", "votes": null }, { "id": "1590621", "postDate": "11/21/2021 13:37:26", "content": "æ¥æ¬èªèš³ã§ã
\nBrief Summary
\nç§ãã¡ã®ãœãªã¥ãŒã·ã§ã³ã¯ãããã€ãã®åŸåŠçãšçµã¿åãããããã€ãã®ãã©ã³ã¹ã¢ãã«ïŒäž»ã«xlm-roberta-largeïŒã®åçŽãªãã¬ã³ãã§ãã ã¢ãŒããã¯ãã£ã¯å
¬éãããŠããã«ãŒãã«ãšåãã§ãåé¡ãããã¯é衚瀺ç¶æ
ã®æ倧+å¹³åããŒãªã³ã°ãŸãã¯CLSããŒã¯ã³ã®é衚瀺ç¶æ
ã®ãããããåããŸãã ã¢ãã«ã®ãã¬ãŒãã³ã°ã«ã¯3段éã®ã¢ãããŒãã䜿çšããŸãããããã§ã¯ã7ã€ã®èšèªããå§ããŠã3ã€ã®èšèªã«åŸ®èª¿æŽãã1ã€ã®èšèªã§çµäºããŸããã ãã®3ã¹ãããã®ã¢ãããŒãã§ã¯ã°ããŒãã«äºæž¬ãæªããããåŸåŠçã¹ããããšããŠèšèªã1ã€ã®èŠå ã§åå¥ã«ã·ããããŸããã
Detailed Summary
\nãããªã«çãæéã§çŽ æŽãããçµæãåŸãããããšã«ãç§ãã¡ã¯ä»ã§ãé©ããŠããŸãã @ aerdem4ã¯ãã®ã³ã³ãã¹ãã§ããå°ãé·ãåããç¹å®ã®èª²é¡ããã§ã«ããç解ããŠããŸãããã@ cpmpmlãš@christofhenkelã¯ãããŒã ã®å䜵æéã®æ°æéåã®ãã€ãŒãææ
ã³ã³ãã¹ãã®çŽåŸã«åå ããŸããã 2æ¥åãç§ãã¡ã¯ãŸã 100以äžã®å Žæã«ããŠãããããéã¡ãã«ãæãã§ããŸããã ããããç²ã匷ããšé©åãªéã®çŽæã§ãã©ã®ã¢ã€ãã¢ãé²ãããããããŠãã¡ããéãè¯ããã°ãç§ãã¡ã¯éã®äžäœã«ç»ãããšãã§ããŸããã
Preprocessing
\nNone. That's NLP in 2020 :D
Speeding up training
\nç§ãã¡ã¯2ã€ã®ããšãããŸããã 1ã€ã¯ããã¬ãã£ããµã³ãã«ãããŠã³ãµã³ããªã³ã°ããŠããããã©ã³ã¹ã®åããããŒã¿ã»ãããšããå°ããªããŒã¿ã»ãããåæã«ååŸããããšã§ããã ããžãã£ããšãã¬ãã£ãã®æ°ã ãã䜿çšããå ŽåãããŒã¿ã»ããã¯çŽ5åã®1ã«åæžãããŸããããã¯ã1ã€ã®ãšããã¯ãå®è¡ããæéãšåãã§ãã ç§ãã¡ã®ã¢ãã«ã®ããã€ãã¯ãã®ããã«èšç·ŽãããŸããã
ãã1ã€ã®ã¹ããŒãã¢ããã¯ããããã«ããããã£ã³ã°ã«ãããã®ã§ãã äž»ãªã¢ã€ãã¢ã¯ãããã£ã³ã°ã®éãçŸåšã®ãããã«å¿
èŠãªéã«å¶éããããšã§ãã å
¥åã¯ãåºå®é·ïŒ512ãªã©ïŒã§ã¯ãªãããããå
ã®æé·å
¥åã®é·ãã«ããã£ã³ã°ãããŸããããã¯çŸåšããç¥ãããŠããææ³ã§ããã以åã®ã³ã³ãã¹ãã§äœ¿çšãããŠããŸãã ãã¬ãŒãã³ã°ã倧å¹
ã«å éããŸãã
\nç¹å®ã®ãããã®ãµã³ãã«ã®é·ããåãã«ãªãããã«ããµã³ãã«ãé·ãã§äžŠã¹æ¿ããããšã§ã¢ã€ãã¢ãæŽç·ŽããŸããã ããã«ãããããã£ã³ã°ã®å¿
èŠæ§ãããã«æžå°ããŸãã ãããå
ã®ãã¹ãŠã®å
¥åã®é·ããåãã§ããå Žåãããã£ã³ã°ã¯ãŸã£ãããããŸããã
ãµã³ãã«ããœãŒããããŠããå Žåããã¬ãŒãã³ã°ã¢ãŒãã§ããããã·ã£ããã«ããããšã¯ã§ããŸããã ãããããããã·ã£ããã«ããŸããã ããã«ããããããã«ããå ã®ããã£ã³ã°ãšæ¯èŒããŠãããã«2åã®ã¹ããŒãã¢ãããåŸãããŸãã æåã®ãã¬ã€ã³ã»ããã§xlm-roberta-largeã®1ã€ã®ãšããã¯ããã¬ãŒãã³ã°ããã«ã¯ãV100GPUã§çŽ17åããããŸãã
\nCross-validation
\næéãçããLBãã£ãŒãããã¯ãååŸããããã®æåºç©ãããŸããªãã£ããããä¿¡é Œæ§ã®é«ãçžäºæ€èšŒã®ååŸã«åãçµã¿ãŸããã æªç¥ã®èšèªruãptãfrãããã³æ¢ç¥ã®èšèªitãesãtrãè¡šãããã«ãèšèªããšã«3åã®ã°ã«ãŒããšæ€èšŒããŒã¿ã®åçŽãª3åã®çµã¿åããã䜿çšããŸããã ãã®çµæã6åã®ã¹ããŒã ã«ãªããŸãã ãããã£ãŠãããšãã°fold1ã¯ãesãvalid.csvã«ãªãå Žåãptãç¡å¹ã§ããã®ãšåãããã«åäœããã¯ãã§ãã ãã¹ãŠã®ãã©ãŒã«ãã®å¹³åAUCãåãããšã¯ããããªãã¯ããã³ãã©ã€ããŒãLBã®éåžžã«åªãããããã·ã§ããã
Architectures
\nã»ãšãã©ã®ããŒã ãšããŠãmax + meanããŒãªã³ã°ãŸãã¯CLSããŒã¯ã³ã®é衚瀺ç¶æ
ã®ããããã䜿çšããåçŽãªåé¡ããããåããã¡ã€ã³ããã¯ããŒã³ãšããŠxlm-robertalargeã䜿çšããŸãããåèšã§ã次ã®ããã¯ããŒã³ã䜿çšããæ¬åŒ§å
ã®æçµçãªã¢ã³ãµã³ãã«ã§ããããã®éã¿ãä»ããŸããã
5x xlm-roberta-largeïŒ85ïŒ
ïŒ
\n1x xlm-roberta-baseïŒ5ïŒ
ïŒ
\n1x mBart-largeïŒ10ïŒ
ïŒ
\nãã¬ãŒãã³ã°æŠç¥
\nç§ãã¡ã®è¯ãçµæã®äž»ãªèŠçŽ ã®1ã€ã¯ãtrãitãããã³esã®åäžèšèªã¢ãã«ã«åãã段éçãªåŸ®èª¿æŽã§ãã説æããåã«ãã¢ãããŒãå
šäœã説æããŸãããã
å ¥åããŒã¿ãšããŠãè±èªã®jigsaw-toxic-comment-train.csvã䜿çšããå ¬éããŒã¿ã»ãããšããŠå©çšå¯èœãª6ã€ã®ç¿»èš³ãçµã¿åããããšãçŽ140äžä»¶ã®ã³ã¡ã³ãã«ãªããŸããçµåãããããŒã¿ã»ããã§ã¢ãã«ãçŽæ¥ãã¬ãŒãã³ã°ããããšã¯æé©ã§ã¯ãªãããšã«æ°ä»ããŸãããåã³ã¡ã³ãã¯åãšããã¯ã§7åã«è¡šç€ºããïŒèšèªã¯ç°ãªããŸããïŒãã¢ãã«ã¯ããã«é©åããããŸããããã§ãçµåããããŒã¿ã»ããã7ã€ã®éå±€åãããéšåã«åå²ããŸãããåéšåã«ã¯ãã³ã¡ã³ãã1åã ãå«ãŸããŠããŸãã次ã«ãæãç³ã¿ããšã«ããã©ã³ã¹ãã©ãŒããŒã3段éã§åŸ®èª¿æŽããŸããã
\nã¹ããã1ïŒ2ã€ã®ãšããã¯ã§7ã€ã®èšèªãã¹ãŠã«åŸ®èª¿æŽãã
\nã¹ããã2ïŒtrãitãesã®ã¿ãå«ãå®å
šãªvalid.csvã«ã®ã¿åŸ®èª¿æŽããŸã
\nã¹ããã3ïŒæå¹ãªåèšèªãžã®3åã®åŸ®èª¿æŽã«ããã3ã€ã®ã¢ãã«ãäœæãããŸã
\n次ã«ãruãäºæž¬ããããã®step1ã¢ãã«ãptãšfrãäºæž¬ããããã®step2ã¢ãã«ãããã³trãitãesã®ããããã®step3ã¢ãã«ã䜿çšããŸãã ptãšfrã«step2ã¢ãã«ã䜿çšãããšãæéã®ãããã®ã«step1ã¢ãã«ã䜿çšããå Žåã«æ¯ã¹ãŠå€§å¹
ã«åäžããŸããããããããesãfrãptã®éã®èšèªã®é¡äŒŒæ§ãåå ã§ãã
åŸé 环ç©ã䜿çšãã32ã®ããããµã€ãºãšãAdamWãªããã£ãã€ã¶ãŒã«ããç·åœ¢æžè¡°ã䌎ã3e-6ã®åŠç¿çãäž»ã«äœ¿çšããŸãããèšåãã䟡å€ã®ããããšã®1ã€ã¯ãããŒã¿ãé·ãã§ãœãŒãããåãé·ãã®ã³ã¡ã³ãã®ããããã©ã³ãã ã«æäŸããäžèšã®ããŒã¿ããŒããŒã®ããã«ãæ倧ã·ãŒã±ã³ã¹é·512ã§ãã¬ãŒãã³ã°ããããšã§ããæ倧512ã®é·ãã䜿çšããŠäœæãããçãã³ã¡ã³ãã®éããã£ã³ã°ãšçµã¿åãããå€§å¹ ãªã¹ããŒãã¢ããã®ã¿ã劥åœã§ãã
\nãããªãã¯ã«ãŒãã«äžã«æ§ç¯ããã1ã€ã®ã¢ãã«ãé€ããŠããã¹ãŠã®ã¢ãã«ã¯pytorchãšGPUã䜿çšããŠãã¬ãŒãã³ã°ãããŸããã
\nEnsembling
\nã¢ã³ãµã³ãã«ã«ã€ããŠã¯ãããŒã ã¡ã³ããŒã®åå¥ã¢ã³ãµã³ãã«ãè¡ããã©ã³ã¯ããŒã»ã³ã¿ã€ã«ã®å éåã§ããŒã ã¡ã³ããŒãšãããªãã¯ã«ãŒãã«ã®ã¢ãã«ãçµã¿åãããŸããã
Post-processing
\næçµæ¥ã«å¹žéã«ãèŠã€ãããã1ã€ã®éèŠãªèŠçŽ ã¯ãæçµäºæž¬ã®åŸåŠçã§ããèšèªã¯åã
ã®ã¢ãã«ãã掟çããŠããŸããã競åææšã¯ãã¹ãŠã®äºæž¬ã®ã°ããŒãã«ã©ã³ã¯ã«ææã§ãããèšèªåºæã§ã¯ãããŸãããããã§ãåèšèªã®äºæž¬ãåå¥ã«ã·ããããããšã§ãçžäºã«æ£ããé¢ä¿ãæã€èšèªãåŠçããŸãããããšãã°ãæçµçãªæåºã§ã¯ã次ã®èŠçŽ ã䜿çšããŸãã
test_df.loc [test_df [\"lang\"] == \"es\"ã \"toxic\"] * = 1.06
\ntest_df.loc [test_df [\"lang\"] == \"fr\"ã \"toxic\"] * = 1.04
\ntest_df.loc [test_df [\"lang\"] == \"it\"ã \"toxic\"] * = 0.97
\ntest_df.loc [test_df [\"lang\"] == \"pt\"ã \"toxic\"] * = 0.96
\ntest_df.loc [test_df [\"lang\"] == \"tr\"ã \"toxic\"] * = 0.98
\nãã¹ãäºæž¬å¹³åãåèšèªã®å
¬éLBå¹³åãšåå¥ã«ç
§åããããšã«ãããå åãå°ãåºããŸããã 1ã€ã®èšèªã1ã«èšå®ããæ®ãã0ã«èšå®ããŠããã¹ãã»ããã調æ»ããããã«ããã§ã«6ã€ã®æåºãè¡ããŸããã次ã«ãåæ§ã®å®éšçãªAUCãåŸãããããã«å¹³åã調æŽãããŠããããšã確èªããŸããããã®posprocessingã¯ç§éãå
Œ
±LBã®14äœãã4äœã«åãããŸããïŒ
ãã®åŸåŠçã¯ä»ã®ããŒã ã«ã圹ç«ã€å¯èœæ§ããããŸããã6ã€ã®èšèªãäºæž¬ããããã«5ã€ã®ã¢ãã«ãæã€ããšã«ãã£ãŠå°å ¥ãããã°ããŒãã«äºæž¬ååžã®çžéãå ·äœçã«ä¿®æ£ããããŒã ãåäžã®ã¢ãã«ã¢ãããŒãã䜿çšããå Žåã¯ããŸã圹ã«ç«ããªãå¯èœæ§ããããŸãã
\nThanks for reading. Questions welcome.
", "rawMarkdown": "æ¥æ¬èªèš³ã§ã\nBrief Summary\nç§ãã¡ã®ãœãªã¥ãŒã·ã§ã³ã¯ãããã€ãã®åŸåŠçãšçµã¿åãããããã€ãã®ãã©ã³ã¹ã¢ãã«ïŒäž»ã«xlm-roberta-largeïŒã®åçŽãªãã¬ã³ãã§ãã ã¢ãŒããã¯ãã£ã¯å ¬éãããŠããã«ãŒãã«ãšåãã§ãåé¡ãããã¯é衚瀺ç¶æ ã®æ倧+å¹³åããŒãªã³ã°ãŸãã¯CLSããŒã¯ã³ã®é衚瀺ç¶æ ã®ãããããåããŸãã ã¢ãã«ã®ãã¬ãŒãã³ã°ã«ã¯3段éã®ã¢ãããŒãã䜿çšããŸãããããã§ã¯ã7ã€ã®èšèªããå§ããŠã3ã€ã®èšèªã«åŸ®èª¿æŽãã1ã€ã®èšèªã§çµäºããŸããã ãã®3ã¹ãããã®ã¢ãããŒãã§ã¯ã°ããŒãã«äºæž¬ãæªããããåŸåŠçã¹ããããšããŠèšèªã1ã€ã®èŠå ã§åå¥ã«ã·ããããŸããã\n\nDetailed Summary\nãããªã«çãæéã§çŽ æŽãããçµæãåŸãããããšã«ãç§ãã¡ã¯ä»ã§ãé©ããŠããŸãã @ aerdem4ã¯ãã®ã³ã³ãã¹ãã§ããå°ãé·ãåããç¹å®ã®èª²é¡ããã§ã«ããç解ããŠããŸãããã@ cpmpmlãš@christofhenkelã¯ãããŒã ã®å䜵æéã®æ°æéåã®ãã€ãŒãææ ã³ã³ãã¹ãã®çŽåŸã«åå ããŸããã 2æ¥åãç§ãã¡ã¯ãŸã 100以äžã®å Žæã«ããŠãããããéã¡ãã«ãæãã§ããŸããã ããããç²ã匷ããšé©åãªéã®çŽæã§ãã©ã®ã¢ã€ãã¢ãé²ãããããããŠãã¡ããéãè¯ããã°ãç§ãã¡ã¯éã®äžäœã«ç»ãããšãã§ããŸããã\n\nPreprocessing\nNone. That's NLP in 2020 :D\n\nSpeeding up training\nç§ãã¡ã¯2ã€ã®ããšãããŸããã 1ã€ã¯ããã¬ãã£ããµã³ãã«ãããŠã³ãµã³ããªã³ã°ããŠããããã©ã³ã¹ã®åããããŒã¿ã»ãããšããå°ããªããŒã¿ã»ãããåæã«ååŸããããšã§ããã ããžãã£ããšãã¬ãã£ãã®æ°ã ãã䜿çšããå ŽåãããŒã¿ã»ããã¯çŽ5åã®1ã«åæžãããŸããããã¯ã1ã€ã®ãšããã¯ãå®è¡ããæéãšåãã§ãã ç§ãã¡ã®ã¢ãã«ã®ããã€ãã¯ãã®ããã«èšç·ŽãããŸããã\n\nãã1ã€ã®ã¹ããŒãã¢ããã¯ããããã«ããããã£ã³ã°ã«ãããã®ã§ãã äž»ãªã¢ã€ãã¢ã¯ãããã£ã³ã°ã®éãçŸåšã®ãããã«å¿ èŠãªéã«å¶éããããšã§ãã å ¥åã¯ãåºå®é·ïŒ512ãªã©ïŒã§ã¯ãªãããããå ã®æé·å ¥åã®é·ãã«ããã£ã³ã°ãããŸããããã¯çŸåšããç¥ãããŠããææ³ã§ããã以åã®ã³ã³ãã¹ãã§äœ¿çšãããŠããŸãã ãã¬ãŒãã³ã°ãå€§å¹ ã«å éããŸãã\nç¹å®ã®ãããã®ãµã³ãã«ã®é·ããåãã«ãªãããã«ããµã³ãã«ãé·ãã§äžŠã¹æ¿ããããšã§ã¢ã€ãã¢ãæŽç·ŽããŸããã ããã«ãããããã£ã³ã°ã®å¿ èŠæ§ãããã«æžå°ããŸãã ãããå ã®ãã¹ãŠã®å ¥åã®é·ããåãã§ããå Žåãããã£ã³ã°ã¯ãŸã£ãããããŸããã\n\nãµã³ãã«ããœãŒããããŠããå Žåããã¬ãŒãã³ã°ã¢ãŒãã§ããããã·ã£ããã«ããããšã¯ã§ããŸããã ãããããããã·ã£ããã«ããŸããã ããã«ããããããã«ããå ã®ããã£ã³ã°ãšæ¯èŒããŠãããã«2åã®ã¹ããŒãã¢ãããåŸãããŸãã æåã®ãã¬ã€ã³ã»ããã§xlm-roberta-largeã®1ã€ã®ãšããã¯ããã¬ãŒãã³ã°ããã«ã¯ãV100GPUã§çŽ17åããããŸãã\n\nCross-validation\næéãçããLBãã£ãŒãããã¯ãååŸããããã®æåºç©ãããŸããªãã£ããããä¿¡é Œæ§ã®é«ãçžäºæ€èšŒã®ååŸã«åãçµã¿ãŸããã æªç¥ã®èšèªruãptãfrãããã³æ¢ç¥ã®èšèªitãesãtrãè¡šãããã«ãèšèªããšã«3åã®ã°ã«ãŒããšæ€èšŒããŒã¿ã®åçŽãª3åã®çµã¿åããã䜿çšããŸããã ãã®çµæã6åã®ã¹ããŒã ã«ãªããŸãã ãããã£ãŠãããšãã°fold1ã¯ãesãvalid.csvã«ãªãå Žåãptãç¡å¹ã§ããã®ãšåãããã«åäœããã¯ãã§ãã ãã¹ãŠã®ãã©ãŒã«ãã®å¹³åAUCãåãããšã¯ããããªãã¯ããã³ãã©ã€ããŒãLBã®éåžžã«åªãããããã·ã§ããã\n\nArchitectures\nã»ãšãã©ã®ããŒã ãšããŠãmax + meanããŒãªã³ã°ãŸãã¯CLSããŒã¯ã³ã®é衚瀺ç¶æ ã®ããããã䜿çšããåçŽãªåé¡ããããåããã¡ã€ã³ããã¯ããŒã³ãšããŠxlm-robertalargeã䜿çšããŸãããåèšã§ã次ã®ããã¯ããŒã³ã䜿çšããæ¬åŒ§å ã®æçµçãªã¢ã³ãµã³ãã«ã§ããããã®éã¿ãä»ããŸããã\n\n5x xlm-roberta-largeïŒ85ïŒ ïŒ\n1x xlm-roberta-baseïŒ5ïŒ ïŒ\n1x mBart-largeïŒ10ïŒ ïŒ\nãã¬ãŒãã³ã°æŠç¥\nç§ãã¡ã®è¯ãçµæã®äž»ãªèŠçŽ ã®1ã€ã¯ãtrãitãããã³esã®åäžèšèªã¢ãã«ã«åãã段éçãªåŸ®èª¿æŽã§ãã説æããåã«ãã¢ãããŒãå šäœã説æããŸãããã\n\n\n\nå ¥åããŒã¿ãšããŠãè±èªã®jigsaw-toxic-comment-train.csvã䜿çšããå ¬éããŒã¿ã»ãããšããŠå©çšå¯èœãª6ã€ã®ç¿»èš³ãçµã¿åããããšãçŽ140äžä»¶ã®ã³ã¡ã³ãã«ãªããŸããçµåãããããŒã¿ã»ããã§ã¢ãã«ãçŽæ¥ãã¬ãŒãã³ã°ããããšã¯æé©ã§ã¯ãªãããšã«æ°ä»ããŸãããåã³ã¡ã³ãã¯åãšããã¯ã§7åã«è¡šç€ºããïŒèšèªã¯ç°ãªããŸããïŒãã¢ãã«ã¯ããã«é©åããããŸããããã§ãçµåããããŒã¿ã»ããã7ã€ã®éå±€åãããéšåã«åå²ããŸãããåéšåã«ã¯ãã³ã¡ã³ãã1åã ãå«ãŸããŠããŸãã次ã«ãæãç³ã¿ããšã«ããã©ã³ã¹ãã©ãŒããŒã3段éã§åŸ®èª¿æŽããŸããã\n\nã¹ããã1ïŒ2ã€ã®ãšããã¯ã§7ã€ã®èšèªãã¹ãŠã«åŸ®èª¿æŽãã\nã¹ããã2ïŒtrãitãesã®ã¿ãå«ãå®å šãªvalid.csvã«ã®ã¿åŸ®èª¿æŽããŸã\nã¹ããã3ïŒæå¹ãªåèšèªãžã®3åã®åŸ®èª¿æŽã«ããã3ã€ã®ã¢ãã«ãäœæãããŸã\n次ã«ãruãäºæž¬ããããã®step1ã¢ãã«ãptãšfrãäºæž¬ããããã®step2ã¢ãã«ãããã³trãitãesã®ããããã®step3ã¢ãã«ã䜿çšããŸãã ptãšfrã«step2ã¢ãã«ã䜿çšãããšãæéã®ãããã®ã«step1ã¢ãã«ã䜿çšããå Žåã«æ¯ã¹ãŠå€§å¹ ã«åäžããŸããããããããesãfrãptã®éã®èšèªã®é¡äŒŒæ§ãåå ã§ãã\n\nåŸé 环ç©ã䜿çšãã32ã®ããããµã€ãºãšãAdamWãªããã£ãã€ã¶ãŒã«ããç·åœ¢æžè¡°ã䌎ã3e-6ã®åŠç¿çãäž»ã«äœ¿çšããŸãããèšåãã䟡å€ã®ããããšã®1ã€ã¯ãããŒã¿ãé·ãã§ãœãŒãããåãé·ãã®ã³ã¡ã³ãã®ããããã©ã³ãã ã«æäŸããäžèšã®ããŒã¿ããŒããŒã®ããã«ãæ倧ã·ãŒã±ã³ã¹é·512ã§ãã¬ãŒãã³ã°ããããšã§ããæ倧512ã®é·ãã䜿çšããŠäœæãããçãã³ã¡ã³ãã®éããã£ã³ã°ãšçµã¿åãããå€§å¹ ãªã¹ããŒãã¢ããã®ã¿ã劥åœã§ãã\n\nãããªãã¯ã«ãŒãã«äžã«æ§ç¯ããã1ã€ã®ã¢ãã«ãé€ããŠããã¹ãŠã®ã¢ãã«ã¯pytorchãšGPUã䜿çšããŠãã¬ãŒãã³ã°ãããŸããã\n\nEnsembling\nã¢ã³ãµã³ãã«ã«ã€ããŠã¯ãããŒã ã¡ã³ããŒã®åå¥ã¢ã³ãµã³ãã«ãè¡ããã©ã³ã¯ããŒã»ã³ã¿ã€ã«ã®å éåã§ããŒã ã¡ã³ããŒãšãããªãã¯ã«ãŒãã«ã®ã¢ãã«ãçµã¿åãããŸããã\n\nPost-processing\næçµæ¥ã«å¹žéã«ãèŠã€ãããã1ã€ã®éèŠãªèŠçŽ ã¯ãæçµäºæž¬ã®åŸåŠçã§ããèšèªã¯åã ã®ã¢ãã«ãã掟çããŠããŸããã競åææšã¯ãã¹ãŠã®äºæž¬ã®ã°ããŒãã«ã©ã³ã¯ã«ææã§ãããèšèªåºæã§ã¯ãããŸãããããã§ãåèšèªã®äºæž¬ãåå¥ã«ã·ããããããšã§ãçžäºã«æ£ããé¢ä¿ãæã€èšèªãåŠçããŸãããããšãã°ãæçµçãªæåºã§ã¯ã次ã®èŠçŽ ã䜿çšããŸãã\n\ntest_df.loc [test_df [\"lang\"] == \"es\"ã \"toxic\"] * = 1.06\ntest_df.loc [test_df [\"lang\"] == \"fr\"ã \"toxic\"] * = 1.04\ntest_df.loc [test_df [\"lang\"] == \"it\"ã \"toxic\"] * = 0.97\ntest_df.loc [test_df [\"lang\"] == \"pt\"ã \"toxic\"] * = 0.96\ntest_df.loc [test_df [\"lang\"] == \"tr\"ã \"toxic\"] * = 0.98\nãã¹ãäºæž¬å¹³åãåèšèªã®å ¬éLBå¹³åãšåå¥ã«ç §åããããšã«ãããå åãå°ãåºããŸããã 1ã€ã®èšèªã1ã«èšå®ããæ®ãã0ã«èšå®ããŠããã¹ãã»ããã調æ»ããããã«ããã§ã«6ã€ã®æåºãè¡ããŸããã次ã«ãåæ§ã®å®éšçãªAUCãåŸãããããã«å¹³åã調æŽãããŠããããšã確èªããŸããããã®posprocessingã¯ç§éãå ¬å ±LBã®14äœãã4äœã«åãããŸããïŒ\n\nãã®åŸåŠçã¯ä»ã®ããŒã ã«ã圹ç«ã€å¯èœæ§ããããŸããã6ã€ã®èšèªãäºæž¬ããããã«5ã€ã®ã¢ãã«ãæã€ããšã«ãã£ãŠå°å ¥ãããã°ããŒãã«äºæž¬ååžã®çžéãå ·äœçã«ä¿®æ£ããããŒã ãåäžã®ã¢ãã«ã¢ãããŒãã䜿çšããå Žåã¯ããŸã圹ã«ç«ããªãå¯èœæ§ããããŸãã\n\nThanks for reading. Questions welcome.", "votes": null } ], "comments": [ { "id": 1495839, "author_name": "top10chi3nthan", "author_url": "", "post_date": "08/29/2021 20:37:19", "content": "Thanks for sharing!! very helpful
", "votes": null, "replies": [] }, { "id": 1574214, "author_name": "shubheshswain", "author_url": "", "post_date": "11/07/2021 10:43:22", "content": "Is your BERT model able to detect negation and sarcasm?
", "votes": null, "replies": [] }, { "id": 1590621, "author_name": "pixyz0130", "author_url": "", "post_date": "11/21/2021 13:37:26", "content": "æ¥æ¬èªèš³ã§ã
\nBrief Summary
\nç§ãã¡ã®ãœãªã¥ãŒã·ã§ã³ã¯ãããã€ãã®åŸåŠçãšçµã¿åãããããã€ãã®ãã©ã³ã¹ã¢ãã«ïŒäž»ã«xlm-roberta-largeïŒã®åçŽãªãã¬ã³ãã§ãã ã¢ãŒããã¯ãã£ã¯å
¬éãããŠããã«ãŒãã«ãšåãã§ãåé¡ãããã¯é衚瀺ç¶æ
ã®æ倧+å¹³åããŒãªã³ã°ãŸãã¯CLSããŒã¯ã³ã®é衚瀺ç¶æ
ã®ãããããåããŸãã ã¢ãã«ã®ãã¬ãŒãã³ã°ã«ã¯3段éã®ã¢ãããŒãã䜿çšããŸãããããã§ã¯ã7ã€ã®èšèªããå§ããŠã3ã€ã®èšèªã«åŸ®èª¿æŽãã1ã€ã®èšèªã§çµäºããŸããã ãã®3ã¹ãããã®ã¢ãããŒãã§ã¯ã°ããŒãã«äºæž¬ãæªããããåŸåŠçã¹ããããšããŠèšèªã1ã€ã®èŠå ã§åå¥ã«ã·ããããŸããã
Detailed Summary
\nãããªã«çãæéã§çŽ æŽãããçµæãåŸãããããšã«ãç§ãã¡ã¯ä»ã§ãé©ããŠããŸãã @ aerdem4ã¯ãã®ã³ã³ãã¹ãã§ããå°ãé·ãåããç¹å®ã®èª²é¡ããã§ã«ããç解ããŠããŸãããã@ cpmpmlãš@christofhenkelã¯ãããŒã ã®å䜵æéã®æ°æéåã®ãã€ãŒãææ
ã³ã³ãã¹ãã®çŽåŸã«åå ããŸããã 2æ¥åãç§ãã¡ã¯ãŸã 100以äžã®å Žæã«ããŠãããããéã¡ãã«ãæãã§ããŸããã ããããç²ã匷ããšé©åãªéã®çŽæã§ãã©ã®ã¢ã€ãã¢ãé²ãããããããŠãã¡ããéãè¯ããã°ãç§ãã¡ã¯éã®äžäœã«ç»ãããšãã§ããŸããã
Preprocessing
\nNone. That's NLP in 2020 :D
Speeding up training
\nç§ãã¡ã¯2ã€ã®ããšãããŸããã 1ã€ã¯ããã¬ãã£ããµã³ãã«ãããŠã³ãµã³ããªã³ã°ããŠããããã©ã³ã¹ã®åããããŒã¿ã»ãããšããå°ããªããŒã¿ã»ãããåæã«ååŸããããšã§ããã ããžãã£ããšãã¬ãã£ãã®æ°ã ãã䜿çšããå ŽåãããŒã¿ã»ããã¯çŽ5åã®1ã«åæžãããŸããããã¯ã1ã€ã®ãšããã¯ãå®è¡ããæéãšåãã§ãã ç§ãã¡ã®ã¢ãã«ã®ããã€ãã¯ãã®ããã«èšç·ŽãããŸããã
ãã1ã€ã®ã¹ããŒãã¢ããã¯ããããã«ããããã£ã³ã°ã«ãããã®ã§ãã äž»ãªã¢ã€ãã¢ã¯ãããã£ã³ã°ã®éãçŸåšã®ãããã«å¿
èŠãªéã«å¶éããããšã§ãã å
¥åã¯ãåºå®é·ïŒ512ãªã©ïŒã§ã¯ãªãããããå
ã®æé·å
¥åã®é·ãã«ããã£ã³ã°ãããŸããããã¯çŸåšããç¥ãããŠããææ³ã§ããã以åã®ã³ã³ãã¹ãã§äœ¿çšãããŠããŸãã ãã¬ãŒãã³ã°ã倧å¹
ã«å éããŸãã
\nç¹å®ã®ãããã®ãµã³ãã«ã®é·ããåãã«ãªãããã«ããµã³ãã«ãé·ãã§äžŠã¹æ¿ããããšã§ã¢ã€ãã¢ãæŽç·ŽããŸããã ããã«ãããããã£ã³ã°ã®å¿
èŠæ§ãããã«æžå°ããŸãã ãããå
ã®ãã¹ãŠã®å
¥åã®é·ããåãã§ããå Žåãããã£ã³ã°ã¯ãŸã£ãããããŸããã
ãµã³ãã«ããœãŒããããŠããå Žåããã¬ãŒãã³ã°ã¢ãŒãã§ããããã·ã£ããã«ããããšã¯ã§ããŸããã ãããããããã·ã£ããã«ããŸããã ããã«ããããããã«ããå ã®ããã£ã³ã°ãšæ¯èŒããŠãããã«2åã®ã¹ããŒãã¢ãããåŸãããŸãã æåã®ãã¬ã€ã³ã»ããã§xlm-roberta-largeã®1ã€ã®ãšããã¯ããã¬ãŒãã³ã°ããã«ã¯ãV100GPUã§çŽ17åããããŸãã
\nCross-validation
\næéãçããLBãã£ãŒãããã¯ãååŸããããã®æåºç©ãããŸããªãã£ããããä¿¡é Œæ§ã®é«ãçžäºæ€èšŒã®ååŸã«åãçµã¿ãŸããã æªç¥ã®èšèªruãptãfrãããã³æ¢ç¥ã®èšèªitãesãtrãè¡šãããã«ãèšèªããšã«3åã®ã°ã«ãŒããšæ€èšŒããŒã¿ã®åçŽãª3åã®çµã¿åããã䜿çšããŸããã ãã®çµæã6åã®ã¹ããŒã ã«ãªããŸãã ãããã£ãŠãããšãã°fold1ã¯ãesãvalid.csvã«ãªãå Žåãptãç¡å¹ã§ããã®ãšåãããã«åäœããã¯ãã§ãã ãã¹ãŠã®ãã©ãŒã«ãã®å¹³åAUCãåãããšã¯ããããªãã¯ããã³ãã©ã€ããŒãLBã®éåžžã«åªãããããã·ã§ããã
Architectures
\nã»ãšãã©ã®ããŒã ãšããŠãmax + meanããŒãªã³ã°ãŸãã¯CLSããŒã¯ã³ã®é衚瀺ç¶æ
ã®ããããã䜿çšããåçŽãªåé¡ããããåããã¡ã€ã³ããã¯ããŒã³ãšããŠxlm-robertalargeã䜿çšããŸãããåèšã§ã次ã®ããã¯ããŒã³ã䜿çšããæ¬åŒ§å
ã®æçµçãªã¢ã³ãµã³ãã«ã§ããããã®éã¿ãä»ããŸããã
5x xlm-roberta-largeïŒ85ïŒ
ïŒ
\n1x xlm-roberta-baseïŒ5ïŒ
ïŒ
\n1x mBart-largeïŒ10ïŒ
ïŒ
\nãã¬ãŒãã³ã°æŠç¥
\nç§ãã¡ã®è¯ãçµæã®äž»ãªèŠçŽ ã®1ã€ã¯ãtrãitãããã³esã®åäžèšèªã¢ãã«ã«åãã段éçãªåŸ®èª¿æŽã§ãã説æããåã«ãã¢ãããŒãå
šäœã説æããŸãããã
å ¥åããŒã¿ãšããŠãè±èªã®jigsaw-toxic-comment-train.csvã䜿çšããå ¬éããŒã¿ã»ãããšããŠå©çšå¯èœãª6ã€ã®ç¿»èš³ãçµã¿åããããšãçŽ140äžä»¶ã®ã³ã¡ã³ãã«ãªããŸããçµåãããããŒã¿ã»ããã§ã¢ãã«ãçŽæ¥ãã¬ãŒãã³ã°ããããšã¯æé©ã§ã¯ãªãããšã«æ°ä»ããŸãããåã³ã¡ã³ãã¯åãšããã¯ã§7åã«è¡šç€ºããïŒèšèªã¯ç°ãªããŸããïŒãã¢ãã«ã¯ããã«é©åããããŸããããã§ãçµåããããŒã¿ã»ããã7ã€ã®éå±€åãããéšåã«åå²ããŸãããåéšåã«ã¯ãã³ã¡ã³ãã1åã ãå«ãŸããŠããŸãã次ã«ãæãç³ã¿ããšã«ããã©ã³ã¹ãã©ãŒããŒã3段éã§åŸ®èª¿æŽããŸããã
\nã¹ããã1ïŒ2ã€ã®ãšããã¯ã§7ã€ã®èšèªãã¹ãŠã«åŸ®èª¿æŽãã
\nã¹ããã2ïŒtrãitãesã®ã¿ãå«ãå®å
šãªvalid.csvã«ã®ã¿åŸ®èª¿æŽããŸã
\nã¹ããã3ïŒæå¹ãªåèšèªãžã®3åã®åŸ®èª¿æŽã«ããã3ã€ã®ã¢ãã«ãäœæãããŸã
\n次ã«ãruãäºæž¬ããããã®step1ã¢ãã«ãptãšfrãäºæž¬ããããã®step2ã¢ãã«ãããã³trãitãesã®ããããã®step3ã¢ãã«ã䜿çšããŸãã ptãšfrã«step2ã¢ãã«ã䜿çšãããšãæéã®ãããã®ã«step1ã¢ãã«ã䜿çšããå Žåã«æ¯ã¹ãŠå€§å¹
ã«åäžããŸããããããããesãfrãptã®éã®èšèªã®é¡äŒŒæ§ãåå ã§ãã
åŸé 环ç©ã䜿çšãã32ã®ããããµã€ãºãšãAdamWãªããã£ãã€ã¶ãŒã«ããç·åœ¢æžè¡°ã䌎ã3e-6ã®åŠç¿çãäž»ã«äœ¿çšããŸãããèšåãã䟡å€ã®ããããšã®1ã€ã¯ãããŒã¿ãé·ãã§ãœãŒãããåãé·ãã®ã³ã¡ã³ãã®ããããã©ã³ãã ã«æäŸããäžèšã®ããŒã¿ããŒããŒã®ããã«ãæ倧ã·ãŒã±ã³ã¹é·512ã§ãã¬ãŒãã³ã°ããããšã§ããæ倧512ã®é·ãã䜿çšããŠäœæãããçãã³ã¡ã³ãã®éããã£ã³ã°ãšçµã¿åãããå€§å¹ ãªã¹ããŒãã¢ããã®ã¿ã劥åœã§ãã
\nãããªãã¯ã«ãŒãã«äžã«æ§ç¯ããã1ã€ã®ã¢ãã«ãé€ããŠããã¹ãŠã®ã¢ãã«ã¯pytorchãšGPUã䜿çšããŠãã¬ãŒãã³ã°ãããŸããã
\nEnsembling
\nã¢ã³ãµã³ãã«ã«ã€ããŠã¯ãããŒã ã¡ã³ããŒã®åå¥ã¢ã³ãµã³ãã«ãè¡ããã©ã³ã¯ããŒã»ã³ã¿ã€ã«ã®å éåã§ããŒã ã¡ã³ããŒãšãããªãã¯ã«ãŒãã«ã®ã¢ãã«ãçµã¿åãããŸããã
Post-processing
\næçµæ¥ã«å¹žéã«ãèŠã€ãããã1ã€ã®éèŠãªèŠçŽ ã¯ãæçµäºæž¬ã®åŸåŠçã§ããèšèªã¯åã
ã®ã¢ãã«ãã掟çããŠããŸããã競åææšã¯ãã¹ãŠã®äºæž¬ã®ã°ããŒãã«ã©ã³ã¯ã«ææã§ãããèšèªåºæã§ã¯ãããŸãããããã§ãåèšèªã®äºæž¬ãåå¥ã«ã·ããããããšã§ãçžäºã«æ£ããé¢ä¿ãæã€èšèªãåŠçããŸãããããšãã°ãæçµçãªæåºã§ã¯ã次ã®èŠçŽ ã䜿çšããŸãã
test_df.loc [test_df [\"lang\"] == \"es\"ã \"toxic\"] * = 1.06
\ntest_df.loc [test_df [\"lang\"] == \"fr\"ã \"toxic\"] * = 1.04
\ntest_df.loc [test_df [\"lang\"] == \"it\"ã \"toxic\"] * = 0.97
\ntest_df.loc [test_df [\"lang\"] == \"pt\"ã \"toxic\"] * = 0.96
\ntest_df.loc [test_df [\"lang\"] == \"tr\"ã \"toxic\"] * = 0.98
\nãã¹ãäºæž¬å¹³åãåèšèªã®å
¬éLBå¹³åãšåå¥ã«ç
§åããããšã«ãããå åãå°ãåºããŸããã 1ã€ã®èšèªã1ã«èšå®ããæ®ãã0ã«èšå®ããŠããã¹ãã»ããã調æ»ããããã«ããã§ã«6ã€ã®æåºãè¡ããŸããã次ã«ãåæ§ã®å®éšçãªAUCãåŸãããããã«å¹³åã調æŽãããŠããããšã確èªããŸããããã®posprocessingã¯ç§éãå
Œ
±LBã®14äœãã4äœã«åãããŸããïŒ
ãã®åŸåŠçã¯ä»ã®ããŒã ã«ã圹ç«ã€å¯èœæ§ããããŸããã6ã€ã®èšèªãäºæž¬ããããã«5ã€ã®ã¢ãã«ãæã€ããšã«ãã£ãŠå°å ¥ãããã°ããŒãã«äºæž¬ååžã®çžéãå ·äœçã«ä¿®æ£ããããŒã ãåäžã®ã¢ãã«ã¢ãããŒãã䜿çšããå Žåã¯ããŸã圹ã«ç«ããªãå¯èœæ§ããããŸãã
\nThanks for reading. Questions welcome.
", "votes": null, "replies": [] }, { "id": 898189, "author_name": "andrilko", "author_url": "", "post_date": "06/23/2020 11:16:22", "content": "Post Proc ... Genius! Thank you for your exp!
", "votes": null, "replies": [] }, { "id": 898201, "author_name": "yassinealouini", "author_url": "", "post_date": "06/23/2020 11:21:02", "content": "Question: so, how was the architecture graph done? ð \nMore seriously: what hardware have you used, i.e. your own or cloud? Also, have you tried TPU (I guess not since you only mention V100)? \nGreat achievement and a lot to learn from!
", "votes": null, "replies": [ { "id": 898388, "author_name": "cpmpml", "author_url": "", "post_date": "06/23/2020 13:50:51", "content": "Ahmet used TPU, Dieter and I used GPU (V100). The former used Keras/TF while the latter used Pytorch. There may be a correlation there ;)
", "votes": null, "replies": [] }, { "id": 898586, "author_name": "yassinealouini", "author_url": "", "post_date": "06/23/2020 15:52:59", "content": "Makes sense. :)
", "votes": null, "replies": [] } ] }, { "id": 898221, "author_name": "sai11fkaneko", "author_url": "", "post_date": "06/23/2020 11:32:46", "content": "Thanks for sharing your great approach!\nI used the normal 1 model to predict 6 languages approach with xlmr-large models. I got +0.008 improvement with your post processing.
\n| | public score| private score |\n| --- | --- | --- |\n|xlmr-large my blending | 0.9487 |0.9471 |\n| xlmr-large my blending + your post processing | 0.9494|0.9479|
Amazing work...I never thought your team would boost from 9500 to 9522 in the last dayð . Can you share some details about how your team found this post-processing trick?
", "votes": null, "replies": [ { "id": 898233, "author_name": "christofhenkel", "author_url": "", "post_date": "06/23/2020 11:46:23", "content": "To me it made sense that global ranking matters, and especially the 'ru' prediction should be off as the only thing the step1 model has seen were translations from en, which I think are weaker in terms of toxicity as original russian comments. Hence we expected those predictions to be in average too low. Then we checked shifting the predictions on the validation set, and it gave a significant improvement in score. So we worked on a method to shift test predictions too...
", "votes": null, "replies": [] }, { "id": 898248, "author_name": "tonyxu", "author_url": "", "post_date": "06/23/2020 11:55:52", "content": "Thank you for your explanation!!!\nHope to learn more from you
", "votes": null, "replies": [] } ] }, { "id": 898247, "author_name": "mzr2017", "author_url": "", "post_date": "06/23/2020 11:54:54", "content": "Congrats for one week journey and anther gold medal!\nWe also find each language need calibration in the last day, but we have got no idea about the coef in only 5 submissions, we just try some lucky coef and get a small boost on post processing.\nAmazing work and thanks for detailed writeup!
", "votes": null, "replies": [] }, { "id": 898250, "author_name": "medrau", "author_url": "", "post_date": "06/23/2020 11:56:01", "content": "The post-processing is real magic! My model got +0.006 and +0.011 improvement when I just multiple Spanish by 1.2 .
", "votes": null, "replies": [] }, { "id": 898275, "author_name": "sheriytm", "author_url": "", "post_date": "06/23/2020 12:20:12", "content": "Congrats @christofhenkel, @cpmpml, @aerdem4 and thanks for sharing a very good solution writeup.
", "votes": null, "replies": [ { "id": 898287, "author_name": "cpmpml", "author_url": "", "post_date": "06/23/2020 12:25:04", "content": "You forget @aerdem4 ;)
", "votes": null, "replies": [] }, { "id": 898290, "author_name": "sheriytm", "author_url": "", "post_date": "06/23/2020 12:27:53", "content": "@cpmpml, amended :-)
", "votes": null, "replies": [] }, { "id": 898292, "author_name": "sheriytm", "author_url": "", "post_date": "06/23/2020 12:30:07", "content": "@cpmpml, it must be the Tweet and Jigsaw exhaustion. @aerdem4 may agree :-)
", "votes": null, "replies": [] } ] }, { "id": 898285, "author_name": "sebastienm", "author_url": "", "post_date": "06/23/2020 12:23:45", "content": "So many brilliant ideas! Thanks for sharing.
\n\nOn the ensembling, is the weighted sum of rank percentile the method of choice to try first, from experience, or just one from a toolbox ?
You shoudl try many methods and see their effect. But when roc-auc is the metric, only rank matter, hence using rank makes sense.
", "votes": null, "replies": [] } ] }, { "id": 898373, "author_name": "drpatrickchan", "author_url": "", "post_date": "06/23/2020 13:41:24", "content": "Dieter and team, very discipline and intelligent move indeed. Watching you and your team come after us from behind is akin watching a horror movie. What a great chase. Thanks very much for sharing the insights. Dr.
", "votes": null, "replies": [] }, { "id": 898529, "author_name": "cdeotte", "author_url": "", "post_date": "06/23/2020 15:14:41", "content": "Great model, congrats on building such a great model in a short time.
\n\nWow, nice post post processing trick. I just applied it to my final model. It increased both public and private LB by 0.008 and increased my private rank from 61st to 24th. Wow.
", "votes": null, "replies": [ { "id": 898597, "author_name": "philippsinger", "author_url": "", "post_date": "06/23/2020 16:02:47", "content": "I am sure if you would optimize it further on public part you could get even higher. The optimal scaling factors can differ quite a bit depending on how your predictions look like.
", "votes": null, "replies": [] }, { "id": 899656, "author_name": "cpmpml", "author_url": "", "post_date": "06/24/2020 11:08:09", "content": "My takeway from recent deep learning competitions is that postprocessing is the best way to improve score, more than NN architecture. I'm serious.
", "votes": null, "replies": [] }, { "id": 899668, "author_name": "shahules", "author_url": "", "post_date": "06/24/2020 11:20:26", "content": "@cpmpml happened for both jigsaw and twitter. And suprisingly both were expected to have big shakeups and ended up having nothing.
", "votes": null, "replies": [] }, { "id": 899676, "author_name": "leecming", "author_url": "", "post_date": "06/24/2020 11:33:56", "content": "When everyone is working off the same set of pretrained models and knowledge of common architectural tweaks, postprocessing is one of the few ways to distinguish your models.
\n\nThe main downside with PP is that it's easy to do it at the last minute so people can see their leads evaporate literally overnight.
", "votes": null, "replies": [] }, { "id": 899680, "author_name": "cpmpml", "author_url": "", "post_date": "06/24/2020 11:38:00", "content": "@shahules it also happened in Bengali
\n\n@leecming sometimes the pp is tricky. In Tweet sentiment I didn't found it nor did @philippsinger for instance.
", "votes": null, "replies": [] }, { "id": 899704, "author_name": "philippsinger", "author_url": "", "post_date": "06/24/2020 11:58:00", "content": "Yeah, I found/profited from it here, but did not in Tweets.
", "votes": null, "replies": [] }, { "id": 899713, "author_name": "christofhenkel", "author_url": "", "post_date": "06/24/2020 12:03:48", "content": "And to be fair, if we would have had more time, I would have tried to incorporate fixing the disturbed individual language distributions in the modelling, instead of fixing it via pp. Often post-processing fixes the symptoms but does not solve the root cause.
", "votes": null, "replies": [] }, { "id": 899741, "author_name": "philippsinger", "author_url": "", "post_date": "06/24/2020 12:24:07", "content": "You still need to adjust for test distributions. So in both ways you are doing some form of test pp, even if it is from a different perspective.
", "votes": null, "replies": [] }, { "id": 899747, "author_name": "aerdem4", "author_url": "", "post_date": "06/24/2020 12:28:51", "content": "We wouldn't postprocess without LB probing. If Kaggle splits the public/private not random, it could be much different. In the past, Kaggle had more tricky splits.
", "votes": null, "replies": [] }, { "id": 899884, "author_name": "philippsinger", "author_url": "", "post_date": "06/24/2020 13:56:34", "content": "Yes, I wanted to make a longer post about it but pub LB should probably not have included all languages, and private test should have been completely hidden. I still dont understand the technical argument of why TPUs need to have the full test data available.
", "votes": null, "replies": [] } ] }, { "id": 899129, "author_name": "mahdhiashraf", "author_url": "", "post_date": "06/24/2020 02:29:35", "content": "Great Post-processing Guys !
", "votes": null, "replies": [] }, { "id": 899200, "author_name": "shahules", "author_url": "", "post_date": "06/24/2020 04:44:14", "content": "Congratulations sir.
", "votes": null, "replies": [] }, { "id": 899414, "author_name": "xj609210970", "author_url": "", "post_date": "06/24/2020 08:18:06", "content": "Congrats and thanks for sharing! All of the methods you've shared are so cool and effective, which have opened my eyes! I also tried to apply your post-processing to my final model. It increased my private rank from 23rd to 13thïŒI'd never thought of such a fast but effective trick to improve score before. That's really awesome! \nThere's so much more I need to learn.\nI got 0.9499 on private LB by just applying the PP trick to my (not chosen) best model ( from 0.9478 to 0.9499 )
\n\n
yeah, its a good pp trick :D
", "votes": null, "replies": [] } ] }, { "id": 910784, "author_name": "", "author_url": "", "post_date": "07/01/2020 11:22:15", "content": "Congratulation! Super solid solution and really loved it, may I ask you how you implemented the padding by batch trick in detail? It would be great if you can share any links for refs, sort of new to NLP and BERT so need some help. By the way around the last 2 days, we were at 14th and then you guys jumped up over us, it makes losing a gold medal not that suffering if it was you guys who outplayed our team LOL, thanks for sharing your cool ideas.
", "votes": null, "replies": [] }, { "id": 916408, "author_name": "karlyukang", "author_url": "", "post_date": "07/05/2020 15:50:36", "content": "Congrats! \nAnd sorry for late asking.\nI'm still confused about the way from 6 single language submissions to pp factors, could you sharing more details?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "898155": "Thanks to kaggle for hosting such an interesting and challenging competition! Multilingual NLP is something very interesting yet difficult. In hindsight I (Dieter) wish Iâd spend less time on the tweet sentiment extraction competition and more on this one. \n\n# Brief Summary\nOur solution is a simple blend of several transformer models (mainly xlm-roberta-large) paired with some post-processing. The architecture was the same as in public available kernels with the classification head taking either max+mean pooling of hidden states or the hidden state of the CLS token. We used a 3-step approach for training our models, where starting from 7 languages we fine-tuned to 3 languages and finished at a single language. As this 3-step approach results in distortion of global predictions we shifted languages individually by a factor as a post-processing step. \n\n# Detailed Summary\nWe are still astonished by the great result in such a short time. While @aerdem4 worked a bit longer on this competition and already had a good understanding of specific challenges @cpmpml and @christofhenkel joined right after the tweet sentiment competition a few hours before team merger deadline. Two days ago we were still at a 100+ spot and were hoping for a silver medal at best. But with persistence and the right amount of intuition what ideas to proceed with, and of course some luck we managed to climb right to the upper gold position.\n\n## Preprocessing\nNone. That's NLP in 2020 :D\n\n## Speeding up training\nWe did two things. One was to downsample negative samples to get a more balanced dataset and a smaller dataset at the same time. If you only use as many negative as positive then dataset is reduced 5x roughly, same for time to run one epoch. Some of our models were trained that way.\n\nAnother speedup came from padding by batch. The main idea is to limit the amount of padding to what is necessary for the current batch. Inputs are padded to the length of the longest input in the batch rather than a fixed length, say 512. This is now a well known technique, and it has been used in previous competitions. It accelerates training significantly. \nWe refined the idea by sorting the samples by their length so that the samples in a given batch have similar length. This reduces even further the need for padding. If all inputs in the batch have the same length then there is no padding at all. \n\nGiven samples are sorted, we cannot shuffle them in training mode. We rather shuffled batches. This yields an extra 2x speedup compared to the original padding by batch. Training one epoch for xlm-roberta-large on the first train set takes about 17 minutes on a V100 GPU.\n\n## Cross-validation\nAs time was short and hence we did not have many submissions to spare for getting LB feedback we worked on getting a reliable cross-validation. We used a mix of Group-3fold per language and simple 3fold of the validation data to represent the unknown languages ru,pt,fr as well as the known languages it, es and tr. which results in a 6fold scheme. So do for example fold1 represent what if es would not be in valid.csv which should behave the same as pt is not in valid. Taking the mean AUC of all folds was a very good proxy for Public and Private LB.\n\n## Architectures\nAs most teams we used xlm-roberta large as the main backbone with a simple classification head that either uses max+mean pooling or the hidden states of the CLS token. In total we used the following backbones with their approximate weighting in our final ensemble in brackets.\n \n- 5x xlm-roberta-large (85%)\n- 1x xlm-roberta-base (5%)\n- 1x mBart-large (10%)\n\n## Training strategy\nOne main ingredient for our good result is a stepwise finetuning towards single language models for tr, it and es. Let us illustrate the whole approach before explaining:\n\n\n\nAs input data we used the english jigsaw-toxic-comment-train.csv and its six translations available as public datasets which combined are roughly 1.4M comments. I realized that training a model directly on the combined dataset is not optimal, as each comment appears 7x in each epoch (although in different languages) and the model overfits on that. So I divided the combined dataset into 7 stratified parts, where each part contains a comment only once. For each fold we then finetuned a transformer in a 3step manner:\n\n\n- Step 1: finetune to all 7 languages for 2 epochs\n- Step 2: finetune only to the full valid.csv which only has tr, it and es\n- Step 3: 3x finetune to each language in valid resulting in 3 models\n\nWe then use the step1 model for predicting ru, the step2 model for predicting pt and fr and the respective step3 models for tr, it and es. Using the step2 model for pt and fr gave a significant boost compared to using step1 model for those due. Most likely due to the language similarity between it, es, fr and pt.\n\nWe used mainly a batchsize of 32 using gradient accumulation and a learning rate of 3e-6 with linear decay with AdamW optimizer. One thing worth mentioning is that we train on a max sequence length of 512 due to the dataloader mentioned above which sorts the data by length and then randomly serves batches of same length comments. Only the huge speed-up paired with the non padding of shorter comments made using a 512 max length reasonable. \n\nApart from one model which was built on top of a public kernel, all models were trained using pytorch and GPU.\n\n## Ensembling\nAs for ensembling we did team member individual ensembling and then combined the models of the team members and public kernels by weighted sum of rank percentile \n\n## Post-processing\nAnother key ingredient, which we luckily found on the last day is post processing of the final predictions. Although languages are derived from individual models, the competition metric is sensitive to the global rank of all predictions and not language specific. So we took care of the languages having a correct relation to each other, by shifting the predictions of each language individually. In our final submission for example we used the following factors \n\n```\ntest_df.loc[test_df[\"lang\"] == \"es\", \"toxic\"] *= 1.06\ntest_df.loc[test_df[\"lang\"] == \"fr\", \"toxic\"] *= 1.04\ntest_df.loc[test_df[\"lang\"] == \"it\", \"toxic\"] *= 0.97\ntest_df.loc[test_df[\"lang\"] == \"pt\", \"toxic\"] *= 0.96\ntest_df.loc[test_df[\"lang\"] == \"tr\", \"toxic\"] *= 0.98\n```\n\n\nWe derived the factors by matching the test prediction mean with the public LB mean for each language individually. We had already made 6 submissions for probing the test set by setting one language to 1 and the rest to 0. Then we made sure that means are aligned to give similar experimental AUC. That posprocessing moved us from 14th place to 4th place on public LB! \n\nWhile this postprocessing might also help other teams, we think it specifically fixes the divergence of global prediction distributions introduced by having 5 models to predict 6 languages, and might not help much if a team used a single model approach.\n\nThanks for reading. Questions welcome.", "898189": "Post Proc ... Genius! Thank you for your exp!", "898201": "Question: so, how was the architecture graph done? ð \nMore seriously: what hardware have you used, i.e. your own or cloud? Also, have you tried TPU (I guess not since you only mention V100)? \nGreat achievement and a lot to learn from!", "898221": "Thanks for sharing your great approach!\nI used the normal 1 model to predict 6 languages approach with xlmr-large models. I got +0.008 improvement with your post processing. \n| | public score| private score |\n| --- | --- | --- |\n|xlmr-large my blending | 0.9487 |0.9471 |\n| xlmr-large my blending + your post processing | 0.9494|0.9479|", "898225": "Amazing work...I never thought your team would boost from 9500 to 9522 in the last dayð . Can you share some details about how your team found this post-processing trick?", "898233": "To me it made sense that global ranking matters, and especially the 'ru' prediction *should* be off as the only thing the step1 model has seen were translations from en, which I think are weaker in terms of toxicity as original russian comments. Hence we expected those predictions to be in average too low. Then we checked shifting the predictions on the validation set, and it gave a significant improvement in score. So we worked on a method to shift test predictions too...", "898247": "Congrats for one week journey and anther gold medal!\nWe also find each language need calibration in the last day, but we have got no idea about the coef in only 5 submissions, we just try some lucky coef and get a small boost on post processing.\nAmazing work and thanks for detailed writeup!", "898248": "Thank you for your explanation!!!\nHope to learn more from you", "898250": "The post-processing is real magic! My model got +0.006 and +0.011 improvement when I just multiple Spanish by 1.2 .", "898275": "Congrats @christofhenkel, @cpmpml, @aerdem4 and thanks for sharing a very good solution writeup.", "898285": "So many brilliant ideas! Thanks for sharing.\n\nOn the ensembling, is the `weighted sum of rank percentile` the method of choice to try first, from experience, or just one from a toolbox ?", "898287": "You forget @aerdem4 ;)", "898290": "cpmpml, amended :-)", "898292": "cpmpml, it must be the Tweet and Jigsaw exhaustion. @aerdem4 may agree :-)", "898373": "Dieter and team, very discipline and intelligent move indeed. Watching you and your team come after us from behind is akin watching a horror movie. What a great chase. Thanks very much for sharing the insights. Dr.", "898388": "Ahmet used TPU, Dieter and I used GPU (V100). The former used Keras/TF while the latter used Pytorch. There may be a correlation there ;)", "898529": "Great model, congrats on building such a great model in a short time. \n\nWow, nice post post processing trick. I just applied it to my final model. It increased both public and private LB by 0.008 and increased my private rank from 61st to 24th. Wow.", "898586": "Makes sense. :)", "898597": "I am sure if you would optimize it further on public part you could get even higher. The optimal scaling factors can differ quite a bit depending on how your predictions look like.", "899129": "Great Post-processing Guys !", "899200": "Congratulations sir.", "899414": "Congrats and thanks for sharing! All of the methods you've shared are so cool and effective, which have opened my eyes! I also tried to apply your post-processing to my final model. It increased my private rank from 23rd to 13thïŒI'd never thought of such a fast but effective trick to improve score before. That's really awesome! \n There's so much more I need to learn.", "899656": "My takeway from recent deep learning competitions is that postprocessing is the best way to improve score, more than NN architecture. I'm serious.", "899668": "cpmpml happened for both jigsaw and twitter. And suprisingly both were expected to have big shakeups and ended up having nothing.", "899676": "When everyone is working off the same set of pretrained models and knowledge of common architectural tweaks, postprocessing is one of the few ways to distinguish your models. \n\nThe main downside with PP is that it's easy to do it at the last minute so people can see their leads evaporate literally overnight.", "899680": "shahules it also happened in Bengali\n\n@leecming sometimes the pp is tricky. In Tweet sentiment I didn't found it nor did @philippsinger for instance.", "899682": "You shoudl try many methods and see their effect. But when roc-auc is the metric, only rank matter, hence using rank makes sense.", "899704": "Yeah, I found/profited from it here, but did not in Tweets.", "899713": "And to be fair, if we would have had more time, I would have tried to incorporate fixing the disturbed individual language distributions in the modelling, instead of fixing it via pp. Often post-processing fixes the symptoms but does not solve the root cause.", "899741": "You still need to adjust for test distributions. So in both ways you are doing some form of test pp, even if it is from a different perspective.", "899747": "We wouldn't postprocess without LB probing. If Kaggle splits the public/private not random, it could be much different. In the past, Kaggle had more tricky splits.", "899884": "Yes, I wanted to make a longer post about it but pub LB should probably not have included all languages, and private test should have been completely hidden. I still dont understand the technical argument of why TPUs need to have the full test data available.", "904323": "I got 0.9499 on private LB by just applying the PP trick to my (not chosen) best model ( from 0.9478 to 0.9499 )\n\n", "904747": "yeah, its a good pp trick :D", "910784": "Congratulation! Super solid solution and really loved it, may I ask you how you implemented the padding by batch trick in detail? It would be great if you can share any links for refs, sort of new to NLP and BERT so need some help. By the way around the last 2 days, we were at 14th and then you guys jumped up over us, it makes losing a gold medal not that suffering if it was you guys who outplayed our team LOL, thanks for sharing your cool ideas.", "916408": "Congrats! \nAnd sorry for late asking.\nI'm still confused about the way from 6 single language submissions to pp factors, could you sharing more details?", "1495839": "Thanks for sharing!! very helpful", "1574214": "Is your BERT model able to detect negation and sarcasm?", "1590621": "æ¥æ¬èªèš³ã§ã\nBrief Summary\nç§ãã¡ã®ãœãªã¥ãŒã·ã§ã³ã¯ãããã€ãã®åŸåŠçãšçµã¿åãããããã€ãã®ãã©ã³ã¹ã¢ãã«ïŒäž»ã«xlm-roberta-largeïŒã®åçŽãªãã¬ã³ãã§ãã ã¢ãŒããã¯ãã£ã¯å ¬éãããŠããã«ãŒãã«ãšåãã§ãåé¡ãããã¯é衚瀺ç¶æ ã®æ倧+å¹³åããŒãªã³ã°ãŸãã¯CLSããŒã¯ã³ã®é衚瀺ç¶æ ã®ãããããåããŸãã ã¢ãã«ã®ãã¬ãŒãã³ã°ã«ã¯3段éã®ã¢ãããŒãã䜿çšããŸãããããã§ã¯ã7ã€ã®èšèªããå§ããŠã3ã€ã®èšèªã«åŸ®èª¿æŽãã1ã€ã®èšèªã§çµäºããŸããã ãã®3ã¹ãããã®ã¢ãããŒãã§ã¯ã°ããŒãã«äºæž¬ãæªããããåŸåŠçã¹ããããšããŠèšèªã1ã€ã®èŠå ã§åå¥ã«ã·ããããŸããã\n\nDetailed Summary\nãããªã«çãæéã§çŽ æŽãããçµæãåŸãããããšã«ãç§ãã¡ã¯ä»ã§ãé©ããŠããŸãã @ aerdem4ã¯ãã®ã³ã³ãã¹ãã§ããå°ãé·ãåããç¹å®ã®èª²é¡ããã§ã«ããç解ããŠããŸãããã@ cpmpmlãš@christofhenkelã¯ãããŒã ã®å䜵æéã®æ°æéåã®ãã€ãŒãææ ã³ã³ãã¹ãã®çŽåŸã«åå ããŸããã 2æ¥åãç§ãã¡ã¯ãŸã 100以äžã®å Žæã«ããŠãããããéã¡ãã«ãæãã§ããŸããã ããããç²ã匷ããšé©åãªéã®çŽæã§ãã©ã®ã¢ã€ãã¢ãé²ãããããããŠãã¡ããéãè¯ããã°ãç§ãã¡ã¯éã®äžäœã«ç»ãããšãã§ããŸããã\n\nPreprocessing\nNone. That's NLP in 2020 :D\n\nSpeeding up training\nç§ãã¡ã¯2ã€ã®ããšãããŸããã 1ã€ã¯ããã¬ãã£ããµã³ãã«ãããŠã³ãµã³ããªã³ã°ããŠããããã©ã³ã¹ã®åããããŒã¿ã»ãããšããå°ããªããŒã¿ã»ãããåæã«ååŸããããšã§ããã ããžãã£ããšãã¬ãã£ãã®æ°ã ãã䜿çšããå ŽåãããŒã¿ã»ããã¯çŽ5åã®1ã«åæžãããŸããããã¯ã1ã€ã®ãšããã¯ãå®è¡ããæéãšåãã§ãã ç§ãã¡ã®ã¢ãã«ã®ããã€ãã¯ãã®ããã«èšç·ŽãããŸããã\n\nãã1ã€ã®ã¹ããŒãã¢ããã¯ããããã«ããããã£ã³ã°ã«ãããã®ã§ãã äž»ãªã¢ã€ãã¢ã¯ãããã£ã³ã°ã®éãçŸåšã®ãããã«å¿ èŠãªéã«å¶éããããšã§ãã å ¥åã¯ãåºå®é·ïŒ512ãªã©ïŒã§ã¯ãªãããããå ã®æé·å ¥åã®é·ãã«ããã£ã³ã°ãããŸããããã¯çŸåšããç¥ãããŠããææ³ã§ããã以åã®ã³ã³ãã¹ãã§äœ¿çšãããŠããŸãã ãã¬ãŒãã³ã°ãå€§å¹ ã«å éããŸãã\nç¹å®ã®ãããã®ãµã³ãã«ã®é·ããåãã«ãªãããã«ããµã³ãã«ãé·ãã§äžŠã¹æ¿ããããšã§ã¢ã€ãã¢ãæŽç·ŽããŸããã ããã«ãããããã£ã³ã°ã®å¿ èŠæ§ãããã«æžå°ããŸãã ãããå ã®ãã¹ãŠã®å ¥åã®é·ããåãã§ããå Žåãããã£ã³ã°ã¯ãŸã£ãããããŸããã\n\nãµã³ãã«ããœãŒããããŠããå Žåããã¬ãŒãã³ã°ã¢ãŒãã§ããããã·ã£ããã«ããããšã¯ã§ããŸããã ãããããããã·ã£ããã«ããŸããã ããã«ããããããã«ããå ã®ããã£ã³ã°ãšæ¯èŒããŠãããã«2åã®ã¹ããŒãã¢ãããåŸãããŸãã æåã®ãã¬ã€ã³ã»ããã§xlm-roberta-largeã®1ã€ã®ãšããã¯ããã¬ãŒãã³ã°ããã«ã¯ãV100GPUã§çŽ17åããããŸãã\n\nCross-validation\næéãçããLBãã£ãŒãããã¯ãååŸããããã®æåºç©ãããŸããªãã£ããããä¿¡é Œæ§ã®é«ãçžäºæ€èšŒã®ååŸã«åãçµã¿ãŸããã æªç¥ã®èšèªruãptãfrãããã³æ¢ç¥ã®èšèªitãesãtrãè¡šãããã«ãèšèªããšã«3åã®ã°ã«ãŒããšæ€èšŒããŒã¿ã®åçŽãª3åã®çµã¿åããã䜿çšããŸããã ãã®çµæã6åã®ã¹ããŒã ã«ãªããŸãã ãããã£ãŠãããšãã°fold1ã¯ãesãvalid.csvã«ãªãå Žåãptãç¡å¹ã§ããã®ãšåãããã«åäœããã¯ãã§ãã ãã¹ãŠã®ãã©ãŒã«ãã®å¹³åAUCãåãããšã¯ããããªãã¯ããã³ãã©ã€ããŒãLBã®éåžžã«åªãããããã·ã§ããã\n\nArchitectures\nã»ãšãã©ã®ããŒã ãšããŠãmax + meanããŒãªã³ã°ãŸãã¯CLSããŒã¯ã³ã®é衚瀺ç¶æ ã®ããããã䜿çšããåçŽãªåé¡ããããåããã¡ã€ã³ããã¯ããŒã³ãšããŠxlm-robertalargeã䜿çšããŸãããåèšã§ã次ã®ããã¯ããŒã³ã䜿çšããæ¬åŒ§å ã®æçµçãªã¢ã³ãµã³ãã«ã§ããããã®éã¿ãä»ããŸããã\n\n5x xlm-roberta-largeïŒ85ïŒ ïŒ\n1x xlm-roberta-baseïŒ5ïŒ ïŒ\n1x mBart-largeïŒ10ïŒ ïŒ\nãã¬ãŒãã³ã°æŠç¥\nç§ãã¡ã®è¯ãçµæã®äž»ãªèŠçŽ ã®1ã€ã¯ãtrãitãããã³esã®åäžèšèªã¢ãã«ã«åãã段éçãªåŸ®èª¿æŽã§ãã説æããåã«ãã¢ãããŒãå šäœã説æããŸãããã\n\n\n\nå ¥åããŒã¿ãšããŠãè±èªã®jigsaw-toxic-comment-train.csvã䜿çšããå ¬éããŒã¿ã»ãããšããŠå©çšå¯èœãª6ã€ã®ç¿»èš³ãçµã¿åããããšãçŽ140äžä»¶ã®ã³ã¡ã³ãã«ãªããŸããçµåãããããŒã¿ã»ããã§ã¢ãã«ãçŽæ¥ãã¬ãŒãã³ã°ããããšã¯æé©ã§ã¯ãªãããšã«æ°ä»ããŸãããåã³ã¡ã³ãã¯åãšããã¯ã§7åã«è¡šç€ºããïŒèšèªã¯ç°ãªããŸããïŒãã¢ãã«ã¯ããã«é©åããããŸããããã§ãçµåããããŒã¿ã»ããã7ã€ã®éå±€åãããéšåã«åå²ããŸãããåéšåã«ã¯ãã³ã¡ã³ãã1åã ãå«ãŸããŠããŸãã次ã«ãæãç³ã¿ããšã«ããã©ã³ã¹ãã©ãŒããŒã3段éã§åŸ®èª¿æŽããŸããã\n\nã¹ããã1ïŒ2ã€ã®ãšããã¯ã§7ã€ã®èšèªãã¹ãŠã«åŸ®èª¿æŽãã\nã¹ããã2ïŒtrãitãesã®ã¿ãå«ãå®å šãªvalid.csvã«ã®ã¿åŸ®èª¿æŽããŸã\nã¹ããã3ïŒæå¹ãªåèšèªãžã®3åã®åŸ®èª¿æŽã«ããã3ã€ã®ã¢ãã«ãäœæãããŸã\n次ã«ãruãäºæž¬ããããã®step1ã¢ãã«ãptãšfrãäºæž¬ããããã®step2ã¢ãã«ãããã³trãitãesã®ããããã®step3ã¢ãã«ã䜿çšããŸãã ptãšfrã«step2ã¢ãã«ã䜿çšãããšãæéã®ãããã®ã«step1ã¢ãã«ã䜿çšããå Žåã«æ¯ã¹ãŠå€§å¹ ã«åäžããŸããããããããesãfrãptã®éã®èšèªã®é¡äŒŒæ§ãåå ã§ãã\n\nåŸé 环ç©ã䜿çšãã32ã®ããããµã€ãºãšãAdamWãªããã£ãã€ã¶ãŒã«ããç·åœ¢æžè¡°ã䌎ã3e-6ã®åŠç¿çãäž»ã«äœ¿çšããŸãããèšåãã䟡å€ã®ããããšã®1ã€ã¯ãããŒã¿ãé·ãã§ãœãŒãããåãé·ãã®ã³ã¡ã³ãã®ããããã©ã³ãã ã«æäŸããäžèšã®ããŒã¿ããŒããŒã®ããã«ãæ倧ã·ãŒã±ã³ã¹é·512ã§ãã¬ãŒãã³ã°ããããšã§ããæ倧512ã®é·ãã䜿çšããŠäœæãããçãã³ã¡ã³ãã®éããã£ã³ã°ãšçµã¿åãããå€§å¹ ãªã¹ããŒãã¢ããã®ã¿ã劥åœã§ãã\n\nãããªãã¯ã«ãŒãã«äžã«æ§ç¯ããã1ã€ã®ã¢ãã«ãé€ããŠããã¹ãŠã®ã¢ãã«ã¯pytorchãšGPUã䜿çšããŠãã¬ãŒãã³ã°ãããŸããã\n\nEnsembling\nã¢ã³ãµã³ãã«ã«ã€ããŠã¯ãããŒã ã¡ã³ããŒã®åå¥ã¢ã³ãµã³ãã«ãè¡ããã©ã³ã¯ããŒã»ã³ã¿ã€ã«ã®å éåã§ããŒã ã¡ã³ããŒãšãããªãã¯ã«ãŒãã«ã®ã¢ãã«ãçµã¿åãããŸããã\n\nPost-processing\næçµæ¥ã«å¹žéã«ãèŠã€ãããã1ã€ã®éèŠãªèŠçŽ ã¯ãæçµäºæž¬ã®åŸåŠçã§ããèšèªã¯åã ã®ã¢ãã«ãã掟çããŠããŸããã競åææšã¯ãã¹ãŠã®äºæž¬ã®ã°ããŒãã«ã©ã³ã¯ã«ææã§ãããèšèªåºæã§ã¯ãããŸãããããã§ãåèšèªã®äºæž¬ãåå¥ã«ã·ããããããšã§ãçžäºã«æ£ããé¢ä¿ãæã€èšèªãåŠçããŸãããããšãã°ãæçµçãªæåºã§ã¯ã次ã®èŠçŽ ã䜿çšããŸãã\n\ntest_df.loc [test_df [\"lang\"] == \"es\"ã \"toxic\"] * = 1.06\ntest_df.loc [test_df [\"lang\"] == \"fr\"ã \"toxic\"] * = 1.04\ntest_df.loc [test_df [\"lang\"] == \"it\"ã \"toxic\"] * = 0.97\ntest_df.loc [test_df [\"lang\"] == \"pt\"ã \"toxic\"] * = 0.96\ntest_df.loc [test_df [\"lang\"] == \"tr\"ã \"toxic\"] * = 0.98\nãã¹ãäºæž¬å¹³åãåèšèªã®å ¬éLBå¹³åãšåå¥ã«ç §åããããšã«ãããå åãå°ãåºããŸããã 1ã€ã®èšèªã1ã«èšå®ããæ®ãã0ã«èšå®ããŠããã¹ãã»ããã調æ»ããããã«ããã§ã«6ã€ã®æåºãè¡ããŸããã次ã«ãåæ§ã®å®éšçãªAUCãåŸãããããã«å¹³åã調æŽãããŠããããšã確èªããŸããããã®posprocessingã¯ç§éãå ¬å ±LBã®14äœãã4äœã«åãããŸããïŒ\n\nãã®åŸåŠçã¯ä»ã®ããŒã ã«ã圹ç«ã€å¯èœæ§ããããŸããã6ã€ã®èšèªãäºæž¬ããããã«5ã€ã®ã¢ãã«ãæã€ããšã«ãã£ãŠå°å ¥ãããã°ããŒãã«äºæž¬ååžã®çžéãå ·äœçã«ä¿®æ£ããããŒã ãåäžã®ã¢ãã«ã¢ãããŒãã䜿çšããå Žåã¯ããŸã圹ã«ç«ããªãå¯èœæ§ããããŸãã\n\nThanks for reading. Questions welcome." }, "source": "meta" }