I would request people at the top to please share some insights of their approaches to motivate others to be in the competition. I have studied Alex's kernels and tried to apply most of the things that he has taught but I am stuck at 0.9464-0.9463 . I see no improvements with anything else\nHere is what all I have tried \n- Use translated dataset for training and validation\n- Balanced the dataset\n- Different Type of Pre-processing (I am not sure if it works )\n- Different architectures on top of XLM-Roberta
\n\nI Have the following questions and what has worked differently for the table toppers\n- Will training on more data will help? If so how many data points? I am currently training on 200,000 examples as limited by memory usage \n- Does changing hyperparams like max_len and batch size lead to iimprovement in model ? I have tried different max_len and bs but didn't improve anything\n- Will it be a good idea to train different classifiers for different languages\n- I know a lot of people have been doing ensembling but using ensembling as brute force can be fatal rigght? What would be a good number of models to use to ensemble?
", "messages": [ { "id": 849499, "postDate": "2020-05-15T20:06:25.460Z", "content": "You got some questions, that's a good starting point. Next step is designing some experiments to answer those questions.
\n\nIn my personal experience, the most valuable things I learn from a kaggle competition usually come from:\n- The ideas (both successful and failed ones) I came up with and implemented myself during the competition. \n- The interesting/unique ideas in winning solutions that I integrated with my solution and verified their effectiveness after the competition ends.
\n\nThe specific ideas are important to win a competition, but the more valuable thing is how we come up with such ideas and how to verify their performances.
", "rawMarkdown": "You got some questions, that's a good starting point. Next step is designing some experiments to answer those questions. \n\nIn my personal experience, the most valuable things I learn from a kaggle competition usually come from:\n- The ideas (both successful and failed ones) I came up with and implemented myself during the competition. \n- The interesting/unique ideas in winning solutions that I integrated with my solution and verified their effectiveness after the competition ends. \n\nThe specific ideas are important to win a competition, but the more valuable thing is how we come up with such ideas and how to verify their performances. ", "votes": 17, "replies": [ { "id": 849665, "postDate": "2020-05-16T00:41:51.463Z", "content": "Wise words @naivelamb , for my experience, for the last year I basically just get into competitions that I don't previously know how to solve, this way I can learn a lot, and in some way, I always win 😄
", "rawMarkdown": "Wise words @naivelamb , for my experience, for the last year I basically just get into competitions that I don't previously know how to solve, this way I can learn a lot, and in some way, I always win 😄 ", "votes": 4 }, { "id": 849770, "postDate": "2020-05-16T03:55:16.313Z", "content": "Thanks a lot for the lovely advices GM @naivelamb\nI will keep these in mind always
", "rawMarkdown": "Thanks a lot for the lovely advices GM @naivelamb\nI will keep these in mind always", "votes": 1 } ] }, { "id": 848890, "postDate": "2020-05-15T10:22:11.433Z", "content": "In my opinion, this is not in the spirit of competitions to beg for high ranked participants to share their solutions (even hints), and I will even encourage them not to \"overshare\". Especially the #1 team.
\n\nSome people currently have an idea (or more) that can win them the competition, why give it away ?\nSharing these ideas would really not benefit anyone's ranking since they will be made public, it will only result in everybody using it and the leaderboard getting even more crowded. People who worked hard to reach high scores may suffer from this, as an approach better than theirs was made available to everybody, regardless of the time invested on the competition.
\n\nThe good time for sharing winning ideas is when the competition ends.
", "rawMarkdown": "In my opinion, this is not in the spirit of competitions to beg for high ranked participants to share their solutions (even hints), and I will even encourage them not to \"overshare\". Especially the #1 team.\n\nSome people currently have an idea (or more) that can win them the competition, why give it away ?\nSharing these ideas would really not benefit anyone's ranking since they will be made public, it will only result in everybody using it and the leaderboard getting even more crowded. People who worked hard to reach high scores may suffer from this, as an approach better than theirs was made available to everybody, regardless of the time invested on the competition.\n\nThe good time for sharing winning ideas is when the competition ends.", "votes": 14, "replies": [ { "id": 848895, "postDate": "2020-05-15T10:32:24.527Z", "content": "I agree that the winning solutions should only be shared only after the competition and I get your point ,However , I am not asking to share their winning Idea or Unique Idea , only a hint that too on the questions I have asked . I am sorry if I have conveyed the wrong Idea I also don't want the lb to be polluted .
", "rawMarkdown": "I agree that the winning solutions should only be shared only after the competition and I get your point ,However , I am not asking to share their winning Idea or Unique Idea , only a hint that too on the questions I have asked . I am sorry if I have conveyed the wrong Idea I also don't want the lb to be polluted .", "votes": -3 }, { "id": 848901, "postDate": "2020-05-15T10:37:18.617Z", "content": "I totally agree with this
\n\nOf course I was ironic (smiley) with my message about the 1st place team, as I know they won't, fortunately, share their approach before the end
\n\nGeneral Ideas may still be shared but spoon feeding is definitely not fair for all those who are hardworking so far to get at the top.
", "rawMarkdown": "I totally agree with this\n\nOf course I was ironic (smiley) with my message about the 1st place team, as I know they won't, fortunately, share their approach before the end \n\nGeneral Ideas may still be shared but spoon feeding is definitely not fair for all those who are hardworking so far to get at the top. ", "votes": 2 } ] }, { "id": 847291, "postDate": "2020-05-14T09:24:52.693Z", "content": "I would request people at the top to please share some insights of their approaches to motivate others to be in the competition. I have studied Alex's kernels and tried to apply most of the things that he has taught but I am stuck at 0.9464-0.9463 . I see no improvements with anything else\nHere is what all I have tried \n- Use translated dataset for training and validation\n- Balanced the dataset\n- Different Type of Pre-processing (I am not sure if it works )\n- Different architectures on top of XLM-Roberta
\n\nI Have the following questions and what has worked differently for the table toppers\n- Will training on more data will help? If so how many data points? I am currently training on 200,000 examples as limited by memory usage \n- Does changing hyperparams like max_len and batch size lead to iimprovement in model ? I have tried different max_len and bs but didn't improve anything\n- Will it be a good idea to train different classifiers for different languages\n- I know a lot of people have been doing ensembling but using ensembling as brute force can be fatal rigght? What would be a good number of models to use to ensemble?
", "rawMarkdown": "I would request people at the top to please share some insights of their approaches to motivate others to be in the competition. I have studied Alex's kernels and tried to apply most of the things that he has taught but I am stuck at 0.9464-0.9463 . I see no improvements with anything else\nHere is what all I have tried \n- Use translated dataset for training and validation\n- Balanced the dataset\n- Different Type of Pre-processing (I am not sure if it works )\n- Different architectures on top of XLM-Roberta\n\nI Have the following questions and what has worked differently for the table toppers\n- Will training on more data will help? If so how many data points? I am currently training on 200,000 examples as limited by memory usage \n- Does changing hyperparams like max_len and batch size lead to iimprovement in model ? I have tried different max_len and bs but didn't improve anything\n- Will it be a good idea to train different classifiers for different languages\n- I know a lot of people have been doing ensembling but using ensembling as brute force can be fatal rigght? What would be a good number of models to use to ensemble?", "votes": 10 }, { "id": 850286, "postDate": "2020-05-16T13:33:58.923Z", "content": "Top kernel have 256 entries that shows that they are trying lot of things and very few of them works, so I think solution is try as many things as you can and see if anything works.
", "rawMarkdown": "Top kernel have 256 entries that shows that they are trying lot of things and very few of them works, so I think solution is try as many things as you can and see if anything works.", "votes": 3, "replies": [ { "id": 850307, "postDate": "2020-05-16T14:00:27.763Z", "content": "Good Point , I am also trying a lot of stuffs and their are still many days left , I hope to do better
", "rawMarkdown": "Good Point , I am also trying a lot of stuffs and their are still many days left , I hope to do better" } ] }, { "id": 847610, "postDate": "2020-05-14T14:17:11.513Z", "content": "I'm also curious to know what the 1st place team is doing ;)
\nThe margin is huge .
But one of them is the co-winner of the first Jigsaw competition.
", "rawMarkdown": "I'm also curious to know what the 1st place team is doing ;) \nThe margin is huge . \n\nBut one of them is the co-winner of the first Jigsaw competition. \n\n", "votes": 4, "replies": [ { "id": 847654, "postDate": "2020-05-14T14:40:02.317Z", "content": "Meanwhile it would be helpful if you share some insights of your approach . Best single model score?\nHow many models have used for ensemble? And if you can answer my above questions I will be glad
", "rawMarkdown": "Meanwhile it would be helpful if you share some insights of your approach . Best single model score?\nHow many models have used for ensemble? And if you can answer my above questions I will be glad" }, { "id": 847768, "postDate": "2020-05-14T15:44:18.803Z", "content": "I've reported my best single model score 0.9465 in this topic
\n\nI've tried almost all the appoaches you listed here without really meaningful success.
\n\nThe dataset is huge...That gives the opportunity to chose carefully the subset for training. And I use about 150 000- 250 000 datapoints in my best model.
\n\nBeware of overfitting too, given the difference in languages between the train and test set (Even if you use data translation and some languages like French and Russian have very sophisticated grammar, so difficult to translate correctly by these API ).
\n\nGood training strategy is the key here, othewise your model will quickly overfit the training set.
\n\nAnd of course ensemble WORKS. I see no reason why it wouldn't work.
", "rawMarkdown": "I've reported my best single model score 0.9465 in [this topic](https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/discussion/141825)\n\nI've tried almost all the appoaches you listed here without really meaningful success. \n\nThe dataset is huge...That gives the opportunity to chose carefully the subset for training. And I use about 150 000- 250 000 datapoints in my best model. \n\nBeware of overfitting too, given the difference in languages between the train and test set (Even if you use data translation and some languages like French and Russian have very sophisticated grammar, so difficult to translate correctly by these API ).\n\n Good training strategy is the key here, othewise your model will quickly overfit the training set. \n\n\nAnd of course ensemble WORKS. I see no reason why it wouldn't work.\n", "votes": 8 }, { "id": 847904, "postDate": "2020-05-14T16:47:12.667Z", "content": "Thanks for sharing\n@rftexas You need to see this
", "rawMarkdown": "Thanks for sharing\n@rftexas You need to see this", "votes": 1 } ] }, { "id": 857688, "postDate": "2020-05-22T20:30:31.277Z", "content": "@tanulsingh077 you can try make pseudo-labeling for test set such as here, I believe it can give boost
\n\nhttps://storage.googleapis.com/kaggle-forum-message-attachments/302467/8860/pl_capture.PNG
", "rawMarkdown": "@tanulsingh077 you can try make pseudo-labeling for test set such as [here](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/52557), I believe it can give boost \n\nhttps://storage.googleapis.com/kaggle-forum-message-attachments/302467/8860/pl_capture.PNG", "votes": 1, "replies": [ { "id": 858239, "postDate": "2020-05-23T10:19:27.717Z", "content": "Thank you @shonenkov I will surely try it out
", "rawMarkdown": "Thank you @shonenkov I will surely try it out", "votes": 1 }, { "id": 876615, "postDate": "2020-06-06T20:37:27.153Z", "content": "@shonenkov , Pseudo labelling is leaky and it will only gives huge CV boost. ref here. I tried it on Tweet Sentiment and my CV improved from ~0.71 to ~0.75 but no improvement on the LB.
", "rawMarkdown": "@shonenkov , Pseudo labelling is leaky and it will only gives huge CV boost. [ref here](https://www.kaggle.com/c/tweet-sentiment-extraction/discussion/156556#876422). I tried it on Tweet Sentiment and my CV improved from ~0.71 to ~0.75 but no improvement on the LB. " } ] }, { "id": 847737, "postDate": "2020-05-14T15:20:49.783Z", "content": "I think that those on top are very experienced persons . That's all.
", "rawMarkdown": "I think that those on top are very experienced persons . That's all.", "votes": 1 }, { "id": 848863, "postDate": "2020-05-15T09:50:45.410Z", "content": "Maybe they manually improve their predictions :) Just need some language skills and the time to go through some thousand sentences.
", "rawMarkdown": "Maybe they manually improve their predictions :) Just need some language skills and the time to go through some thousand sentences.", "votes": -6, "replies": [ { "id": 848891, "postDate": "2020-05-15T10:25:52.733Z", "content": "Hand labelling is against the competition rules . I think all top teams are aware of that
\n\nMay be if you downplay hardworking and ethics to compete fairly, you may still try it and see if you will be at the top on both public and private LB ^^
", "rawMarkdown": "\nHand labelling is against the competition rules . I think all top teams are aware of that\n\n\nMay be if you downplay hardworking and ethics to compete fairly, you may still try it and see if you will be at the top on both public and private LB ^^", "votes": 2 }, { "id": 872390, "postDate": "2020-06-03T06:30:45.270Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 849904, "postDate": "2020-05-16T06:56:51.870Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 853708, "postDate": "2020-05-19T12:27:23.870Z", "content": "I don't think that's a good idea.The model can easily learn the comments with curse/swear words as toxic.But there can be false positives like\nExample:“Oh, I feel like such an asshole now.Sorry, bud.”\nI think the model should actually learn to correctly classify this sort of comments.
exactly, as we're in the world of multi-directional language models which take into account each word together with other word in a given sentence, e.g. in the attention mechanism.
", "rawMarkdown": "exactly, as we're in the world of multi-directional language models which take into account each word together with other word in a given sentence, e.g. in the attention mechanism." } ] }, { "id": 847903, "postDate": "2020-05-14T16:47:01.357Z", "rawMarkdown": "", "votes": -1, "isDeleted": true } ], "comments": [ { "id": 849499, "author_name": "Xuan Cao", "author_url": "", "post_date": "2020-05-15T20:06:25.460000", "content": "You got some questions, that's a good starting point. Next step is designing some experiments to answer those questions.
\n\nIn my personal experience, the most valuable things I learn from a kaggle competition usually come from:\n- The ideas (both successful and failed ones) I came up with and implemented myself during the competition. \n- The interesting/unique ideas in winning solutions that I integrated with my solution and verified their effectiveness after the competition ends.
\n\nThe specific ideas are important to win a competition, but the more valuable thing is how we come up with such ideas and how to verify their performances.
", "votes": 17, "replies": [ { "id": 849665, "author_name": "DimitreOliveira", "author_url": "", "post_date": "2020-05-16T00:41:51.463000", "content": "Wise words @naivelamb , for my experience, for the last year I basically just get into competitions that I don't previously know how to solve, this way I can learn a lot, and in some way, I always win 😄
", "votes": 4, "replies": [] }, { "id": 849770, "author_name": "Mr_KnowNothing", "author_url": "", "post_date": "2020-05-16T03:55:16.313000", "content": "Thanks a lot for the lovely advices GM @naivelamb\nI will keep these in mind always
", "votes": 1, "replies": [] } ] }, { "id": 848890, "author_name": "Theo Viel", "author_url": "", "post_date": "2020-05-15T10:22:11.433000", "content": "In my opinion, this is not in the spirit of competitions to beg for high ranked participants to share their solutions (even hints), and I will even encourage them not to \"overshare\". Especially the #1 team.
\n\nSome people currently have an idea (or more) that can win them the competition, why give it away ?\nSharing these ideas would really not benefit anyone's ranking since they will be made public, it will only result in everybody using it and the leaderboard getting even more crowded. People who worked hard to reach high scores may suffer from this, as an approach better than theirs was made available to everybody, regardless of the time invested on the competition.
\n\nThe good time for sharing winning ideas is when the competition ends.
", "votes": 14, "replies": [ { "id": 848895, "author_name": "Mr_KnowNothing", "author_url": "", "post_date": "2020-05-15T10:32:24.527000", "content": "I agree that the winning solutions should only be shared only after the competition and I get your point ,However , I am not asking to share their winning Idea or Unique Idea , only a hint that too on the questions I have asked . I am sorry if I have conveyed the wrong Idea I also don't want the lb to be polluted .
", "votes": -3, "replies": [] }, { "id": 848901, "author_name": "Serigne ", "author_url": "", "post_date": "2020-05-15T10:37:18.617000", "content": "I totally agree with this
\n\nOf course I was ironic (smiley) with my message about the 1st place team, as I know they won't, fortunately, share their approach before the end
\n\nGeneral Ideas may still be shared but spoon feeding is definitely not fair for all those who are hardworking so far to get at the top.
", "votes": 2, "replies": [] } ] }, { "id": 850286, "author_name": "Maunish dave", "author_url": "", "post_date": "2020-05-16T13:33:58.923000", "content": "Top kernel have 256 entries that shows that they are trying lot of things and very few of them works, so I think solution is try as many things as you can and see if anything works.
", "votes": 3, "replies": [ { "id": 850307, "author_name": "Mr_KnowNothing", "author_url": "", "post_date": "2020-05-16T14:00:27.763000", "content": "Good Point , I am also trying a lot of stuffs and their are still many days left , I hope to do better
", "votes": 0, "replies": [] } ] }, { "id": 847610, "author_name": "Serigne ", "author_url": "", "post_date": "2020-05-14T14:17:11.513000", "content": "I'm also curious to know what the 1st place team is doing ;)
\nThe margin is huge .
But one of them is the co-winner of the first Jigsaw competition.
", "votes": 4, "replies": [ { "id": 847654, "author_name": "Mr_KnowNothing", "author_url": "", "post_date": "2020-05-14T14:40:02.317000", "content": "Meanwhile it would be helpful if you share some insights of your approach . Best single model score?\nHow many models have used for ensemble? And if you can answer my above questions I will be glad
", "votes": 0, "replies": [] }, { "id": 847768, "author_name": "Serigne ", "author_url": "", "post_date": "2020-05-14T15:44:18.803000", "content": "I've reported my best single model score 0.9465 in this topic
\n\nI've tried almost all the appoaches you listed here without really meaningful success.
\n\nThe dataset is huge...That gives the opportunity to chose carefully the subset for training. And I use about 150 000- 250 000 datapoints in my best model.
\n\nBeware of overfitting too, given the difference in languages between the train and test set (Even if you use data translation and some languages like French and Russian have very sophisticated grammar, so difficult to translate correctly by these API ).
\n\nGood training strategy is the key here, othewise your model will quickly overfit the training set.
\n\nAnd of course ensemble WORKS. I see no reason why it wouldn't work.
", "votes": 8, "replies": [] }, { "id": 847904, "author_name": "Mr_KnowNothing", "author_url": "", "post_date": "2020-05-14T16:47:12.667000", "content": "Thanks for sharing\n@rftexas You need to see this
", "votes": 1, "replies": [] } ] }, { "id": 857688, "author_name": "Alex Shonenkov", "author_url": "", "post_date": "2020-05-22T20:30:31.277000", "content": "@tanulsingh077 you can try make pseudo-labeling for test set such as here, I believe it can give boost
\n\nhttps://storage.googleapis.com/kaggle-forum-message-attachments/302467/8860/pl_capture.PNG
", "votes": 1, "replies": [ { "id": 858239, "author_name": "Mr_KnowNothing", "author_url": "", "post_date": "2020-05-23T10:19:27.717000", "content": "Thank you @shonenkov I will surely try it out
", "votes": 1, "replies": [] }, { "id": 876615, "author_name": "Tushar", "author_url": "", "post_date": "2020-06-06T20:37:27.153000", "content": "@shonenkov , Pseudo labelling is leaky and it will only gives huge CV boost. ref here. I tried it on Tweet Sentiment and my CV improved from ~0.71 to ~0.75 but no improvement on the LB.
", "votes": 0, "replies": [] } ] }, { "id": 847737, "author_name": "", "author_url": "", "post_date": "2020-05-14T15:20:49.783000", "content": "I think that those on top are very experienced persons . That's all.
", "votes": 1, "replies": [] }, { "id": 848863, "author_name": "Damian Berger", "author_url": "", "post_date": "2020-05-15T09:50:45.410000", "content": "Maybe they manually improve their predictions :) Just need some language skills and the time to go through some thousand sentences.
", "votes": -6, "replies": [ { "id": 848891, "author_name": "Serigne ", "author_url": "", "post_date": "2020-05-15T10:25:52.733000", "content": "Hand labelling is against the competition rules . I think all top teams are aware of that
\n\nMay be if you downplay hardworking and ethics to compete fairly, you may still try it and see if you will be at the top on both public and private LB ^^
", "votes": 2, "replies": [] }, { "id": 872390, "author_name": "", "author_url": "", "post_date": "2020-06-03T06:30:45.270000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 849904, "author_name": "", "author_url": "", "post_date": "2020-05-16T06:56:51.870000", "content": "", "votes": 0, "replies": [ { "id": 853708, "author_name": "Shahules", "author_url": "", "post_date": "2020-05-19T12:27:23.870000", "content": "I don't think that's a good idea.The model can easily learn the comments with curse/swear words as toxic.But there can be false positives like\nExample:“Oh, I feel like such an asshole now.Sorry, bud.”\nI think the model should actually learn to correctly classify this sort of comments.
exactly, as we're in the world of multi-directional language models which take into account each word together with other word in a given sentence, e.g. in the attention mechanism.
", "votes": 0, "replies": [] } ] }, { "id": 847903, "author_name": "", "author_url": "", "post_date": "2020-05-14T16:47:01.357000", "content": "", "votes": -1, "replies": [] } ], "raw_markdown_by_id": { "849499": "You got some questions, that's a good starting point. Next step is designing some experiments to answer those questions. \n\nIn my personal experience, the most valuable things I learn from a kaggle competition usually come from:\n- The ideas (both successful and failed ones) I came up with and implemented myself during the competition. \n- The interesting/unique ideas in winning solutions that I integrated with my solution and verified their effectiveness after the competition ends. \n\nThe specific ideas are important to win a competition, but the more valuable thing is how we come up with such ideas and how to verify their performances. ", "848890": "In my opinion, this is not in the spirit of competitions to beg for high ranked participants to share their solutions (even hints), and I will even encourage them not to \"overshare\". Especially the #1 team.\n\nSome people currently have an idea (or more) that can win them the competition, why give it away ?\nSharing these ideas would really not benefit anyone's ranking since they will be made public, it will only result in everybody using it and the leaderboard getting even more crowded. People who worked hard to reach high scores may suffer from this, as an approach better than theirs was made available to everybody, regardless of the time invested on the competition.\n\nThe good time for sharing winning ideas is when the competition ends.", "847291": "I would request people at the top to please share some insights of their approaches to motivate others to be in the competition. I have studied Alex's kernels and tried to apply most of the things that he has taught but I am stuck at 0.9464-0.9463 . I see no improvements with anything else\nHere is what all I have tried \n- Use translated dataset for training and validation\n- Balanced the dataset\n- Different Type of Pre-processing (I am not sure if it works )\n- Different architectures on top of XLM-Roberta\n\nI Have the following questions and what has worked differently for the table toppers\n- Will training on more data will help? If so how many data points? I am currently training on 200,000 examples as limited by memory usage \n- Does changing hyperparams like max_len and batch size lead to iimprovement in model ? I have tried different max_len and bs but didn't improve anything\n- Will it be a good idea to train different classifiers for different languages\n- I know a lot of people have been doing ensembling but using ensembling as brute force can be fatal rigght? What would be a good number of models to use to ensemble?", "850286": "Top kernel have 256 entries that shows that they are trying lot of things and very few of them works, so I think solution is try as many things as you can and see if anything works.", "847610": "I'm also curious to know what the 1st place team is doing ;) \nThe margin is huge . \n\nBut one of them is the co-winner of the first Jigsaw competition. \n\n", "857688": "@tanulsingh077 you can try make pseudo-labeling for test set such as [here](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/52557), I believe it can give boost \n\nhttps://storage.googleapis.com/kaggle-forum-message-attachments/302467/8860/pl_capture.PNG", "847737": "I think that those on top are very experienced persons . That's all.", "848863": "Maybe they manually improve their predictions :) Just need some language skills and the time to go through some thousand sentences.", "849904": "", "847903": "" } }![https://storage.googleapis.com/kaggle-forum-message-attachments/302467/8860/pl_capture.PNG](\"https://storage.googleapis.com/kaggle-forum-message-attachments/302467/8860/pl_capture.PNG\"){kind=link}