Hey, I just joined this competition.
\n\nI wanted to ask if anyone found out a better cross validation scheme than simple Kfold
", "messages": [ { "id": "887571", "postDate": "06/15/2020 18:37:31", "content": "Hey, I just joined this competition.
\n\nI wanted to ask if anyone found out a better cross validation scheme than simple Kfold
", "rawMarkdown": "Hey, I just joined this competition.\n\nI wanted to ask if anyone found out a better cross validation scheme than simple Kfold", "votes": null }, { "id": "887912", "postDate": "06/16/2020 01:55:08", "content": "Valid 8k 5fold CV is good enough for me
", "rawMarkdown": "Valid 8k 5fold CV is good enough for me", "votes": null }, { "id": "888820", "postDate": "06/16/2020 15:24:06", "content": "One thing to note is that the test set contains 6 languages (tr, pt, ru, fr, it, es) while the validation set contains only 3 languages (tr, es, it), see e.g. this EDA notebook https://www.kaggle.com/ipythonx/jigsaw-multilingual-quick-eda-tpu-modeling
", "rawMarkdown": "One thing to note is that the test set contains 6 languages (tr, pt, ru, fr, it, es) while the validation set contains only 3 languages (tr, es, it), see e.g. this EDA notebook https://www.kaggle.com/ipythonx/jigsaw-multilingual-quick-eda-tpu-modeling", "votes": null }, { "id": "889475", "postDate": "06/17/2020 01:13:57", "content": "Hi bro, are you a uestc-er?
", "rawMarkdown": "Hi bro, are you a uestc-er?", "votes": null }, { "id": "889600", "postDate": "06/17/2020 03:15:48", "content": "Wow! Are you going to take another gold within a week?
", "rawMarkdown": "Wow! Are you going to take another gold within a week?", "votes": null }, { "id": "889643", "postDate": "06/17/2020 03:45:36", "content": "I wish you were right ;)
", "rawMarkdown": "I wish you were right ;)", "votes": null }, { "id": "889827", "postDate": "06/17/2020 07:10:32", "content": "There is also spanish lang in test set. Hard to find really good set to make good CV, maybe some external datasets
", "rawMarkdown": "There is also spanish lang in test set. Hard to find really good set to make good CV, maybe some external datasets", "votes": null }, { "id": "889841", "postDate": "06/17/2020 07:22:17", "content": "what's uestc-er? :)
", "rawMarkdown": "what's uestc-er? :)", "votes": null }, { "id": "889922", "postDate": "06/17/2020 08:03:53", "content": "Public LB :)
", "rawMarkdown": "Public LB :)", "votes": null }, { "id": "889932", "postDate": "06/17/2020 08:09:45", "content": "Fixed it, thanks! Indeed, CV consistently way above public LB, wondering if it is due to (pt, ru, fr).
", "rawMarkdown": "Fixed it, thanks! Indeed, CV consistently way above public LB, wondering if it is due to (pt, ru, fr).", "votes": null }, { "id": "890202", "postDate": "06/17/2020 11:35:43", "content": "Stratified K-fold
", "rawMarkdown": "Stratified K-fold", "votes": null }, { "id": "890236", "postDate": "06/17/2020 12:00:56", "content": "with only 5 subs per day, that puts us in a difficult position :D
", "rawMarkdown": "with only 5 subs per day, that puts us in a difficult position :D", "votes": null }, { "id": "890250", "postDate": "06/17/2020 12:06:42", "content": "Haha, a university in chengdu.
", "rawMarkdown": "Haha, a university in chengdu.", "votes": null }, { "id": "891039", "postDate": "06/17/2020 21:10:07", "content": "Or in a good position if I think about it 😆
", "rawMarkdown": "Or in a good position if I think about it 😆", "votes": null }, { "id": "891160", "postDate": "06/18/2020 00:56:07", "content": "I personally do not think there's a robust CV strategy. My CV score(.97+) is much higher than my LB score. Data augmentation is probably the key to this competition. Good luck.
", "rawMarkdown": "I personally do not think there's a robust CV strategy. My CV score(.97+) is much higher than my LB score. Data augmentation is probably the key to this competition. Good luck.", "votes": null }, { "id": "891175", "postDate": "06/18/2020 01:24:01", "content": "You have a CV of 0.97+? Wow
", "rawMarkdown": "You have a CV of 0.97+? Wow", "votes": null }, { "id": "891484", "postDate": "06/18/2020 07:53:48", "content": "@mcggood so you're not splitting the training dataset? Only adding 4/5 th of valid data to it before training.
", "rawMarkdown": "mcggood so you're not splitting the training dataset? Only adding 4/5 th of valid data to it before training.", "votes": null }, { "id": "893021", "postDate": "06/19/2020 10:23:34", "content": "This will be one of the greatest shake-ups ever. Hope it turns out I'm wrong :)
", "rawMarkdown": "This will be one of the greatest shake-ups ever. Hope it turns out I'm wrong :)", "votes": null }, { "id": "893046", "postDate": "06/19/2020 10:39:58", "content": "Greatest? Probably not. \nShakeup? Yes
\n\nI think all competitions that don't allow for proper CV are prone for shakeups.
", "rawMarkdown": "Greatest? Probably not. \nShakeup? Yes\n\nI think all competitions that don't allow for proper CV are prone for shakeups.", "votes": null }, { "id": "893165", "postDate": "06/19/2020 12:36:28", "content": "Agreed. Considering that there's no hidden test set with new languages, maybe the shake-up will not be as bad as in MS malware or ELO. But still kinda lottery.
", "rawMarkdown": "Agreed. Considering that there's no hidden test set with new languages, maybe the shake-up will not be as bad as in MS malware or ELO. But still kinda lottery.", "votes": null }, { "id": "893178", "postDate": "06/19/2020 12:45:43", "content": "We were saying tweet would be a lottery, and in the end shakeup was rather limited, at least near top of LB.
", "rawMarkdown": "We were saying tweet would be a lottery, and in the end shakeup was rather limited, at least near top of LB.", "votes": null }, { "id": "893184", "postDate": "06/19/2020 12:46:53", "content": "@cpmpml yes, apart for some team shakeup was limited in tweets competition.Are you here to get the gold you just missed in tweets?😂 😂
", "rawMarkdown": "cpmpml yes, apart for some team shakeup was limited in tweets competition.Are you here to get the gold you just missed in tweets?😂 😂", "votes": null }, { "id": "893274", "postDate": "06/19/2020 14:06:24", "content": "@shahules sorry, to be honest. I don't have a good CV plan. I'm overfitting personaly
", "rawMarkdown": "shahules sorry, to be honest. I don't have a good CV plan. I'm overfitting personaly", "votes": null }, { "id": "893739", "postDate": "06/19/2020 20:58:04", "content": "Personally I don't think there is a robust cross-validation setup: validation set is too small and only 3 languages, public set is large but risky.
", "rawMarkdown": "Personally I don't think there is a robust cross-validation setup: validation set is too small and only 3 languages, public set is large but risky.", "votes": null }, { "id": "893904", "postDate": "06/20/2020 04:33:11", "content": "I shaked down a few hundreds by not selecting the right submission in tweets 😅
", "rawMarkdown": "I shaked down a few hundreds by not selecting the right submission in tweets 😅", "votes": null }, { "id": "894002", "postDate": "06/20/2020 05:32:19", "content": "Looks like I found something working (only made 3 submissions so far):
\n\ncv 0.9287 LB 0.9282\ncv 0.9384 LB 0.9377\ncv 0.9400 LB 0.9410
", "rawMarkdown": "Looks like I found something working (only made 3 submissions so far):\n\ncv 0.9287 LB 0.9282\ncv 0.9384 LB 0.9377\ncv 0.9400 LB 0.9410", "votes": null }, { "id": "894464", "postDate": "06/20/2020 12:58:00", "content": "Averaging many runs (by varying seeds) and using KFold? Haven't yet checked if it works but that's what I am implementing.
", "rawMarkdown": "Averaging many runs (by varying seeds) and using KFold? Haven't yet checked if it works but that's what I am implementing.", "votes": null }, { "id": "894859", "postDate": "06/20/2020 20:44:15", "content": "@cpmpml, in the case of tweet it was more like jump up for a lot of teams than the usual shakeup we see. I was even tempted to run @jtrotman's shakeup script on that competition if not because I need to devote time to Jigsaw.
", "rawMarkdown": "cpmpml, in the case of tweet it was more like jump up for a lot of teams than the usual shakeup we see. I was even tempted to run @jtrotman's shakeup script on that competition if not because I need to devote time to Jigsaw.", "votes": null }, { "id": "895688", "postDate": "06/21/2020 14:46:40", "content": "@shahules I am not solo here, even if we get a gold then it won't make for a solo gold. But I'd be very happy to get a team gold still! We are working hard on it, but time is very limited.
", "rawMarkdown": "shahules I am not solo here, even if we get a gold then it won't make for a solo gold. But I'd be very happy to get a team gold still! We are working hard on it, but time is very limited.", "votes": null }, { "id": "895690", "postDate": "06/21/2020 14:47:15", "content": "@sheriytm what is @jtrotman shakeup script?
", "rawMarkdown": "sheriytm what is @jtrotman shakeup script?", "votes": null }, { "id": "895872", "postDate": "06/21/2020 17:01:23", "content": "@cpmpml I have not used them but here they are:-
\n\n", "rawMarkdown": "cpmpml I have not used them but here they are:-\n\n1. [Meta Kaggle: Scatter Plot Competition Shake-up](https://www.kaggle.com/jtrotman/meta-kaggle-scatter-plot-competition-shake-up)\n\n2. [Meta Kaggle: Competition Shake-up](https://www.kaggle.com/jtrotman/meta-kaggle-competition-shake-up)", "votes": null }, { "id": "896946", "postDate": "06/22/2020 14:34:05", "content": "since the competition is almost over , can you brief up about the data augmentation you did ?
", "rawMarkdown": "since the competition is almost over , can you brief up about the data augmentation you did ?", "votes": null }, { "id": "897539", "postDate": "06/23/2020 00:22:50", "content": "I didn't think it was possible, and yet here we are. Very lucky last 2 days.
", "rawMarkdown": "I didn't think it was possible, and yet here we are. Very lucky last 2 days.", "votes": null }, { "id": "897563", "postDate": "06/23/2020 00:48:53", "content": "Congrats @cpmpml and @christofhenkel. You guys inspired me to keep up in the last couple of days but I still missed the gold. However, I am left energized. See you in the next one.
", "rawMarkdown": "Congrats @cpmpml and @christofhenkel. You guys inspired me to keep up in the last couple of days but I still missed the gold. However, I am left energized. See you in the next one.", "votes": null }, { "id": "897570", "postDate": "06/23/2020 00:53:38", "content": "Yes, congrats, that's a super-human performance! \nIs there any track record of better than 4th place in a week ?
", "rawMarkdown": "Yes, congrats, that's a super-human performance! \nIs there any track record of better than 4th place in a week ?", "votes": null }, { "id": "897877", "postDate": "06/23/2020 06:53:45", "content": "\n\n\nYury Kashnitsky wrote:
\n \nThis will be one of the greatest shake-ups ever. Hope it turns out I'm wrong :)
\n
Narrator: Yury was wrong.
\n\nYury: Wow! Surprised with such robust results for many teams. Will be good to study their validations schemes even if I didn’t actively participate.
", "rawMarkdown": "> **Yury Kashnitsky wrote:**\n> \n> This will be one of the greatest shake-ups ever. Hope it turns out I'm wrong :)\n\nNarrator: Yury was wrong. \n\nYury: Wow! Surprised with such robust results for many teams. Will be good to study their validations schemes even if I didn’t actively participate.", "votes": null }, { "id": "897880", "postDate": "06/23/2020 06:57:12", "content": "We calculated with more shakeup and did not select our best Public LB submission which included the best Public kernel, because we thought its heavily overfit. Turns out we were wrong :D
", "rawMarkdown": "We calculated with more shakeup and did not select our best Public LB submission which included the best Public kernel, because we thought its heavily overfit. Turns out we were wrong :D", "votes": null }, { "id": "897885", "postDate": "06/23/2020 06:58:28", "content": "@kashnitsky LB was the validation scheme.
", "rawMarkdown": "kashnitsky LB was the validation scheme.", "votes": null }, { "id": "897914", "postDate": "06/23/2020 07:13:28", "content": "OMG!\nYou guys achieved something almost impossible!\nCongratulation!
", "rawMarkdown": "OMG!\nYou guys achieved something almost impossible!\nCongratulation!", "votes": null }, { "id": "897926", "postDate": "06/23/2020 07:21:32", "content": "I was concerned about a shake-up as well but it ended up like 2018. If you look at the private/public LB for that competition, very stable as well.
\n\nWonder if there's an explanation for the lack of overfitting given that most of us were making hundreds of subs to fit to public LB.
", "rawMarkdown": "I was concerned about a shake-up as well but it ended up like 2018. If you look at the private/public LB for that competition, very stable as well.\n\nWonder if there's an explanation for the lack of overfitting given that most of us were making hundreds of subs to fit to public LB.", "votes": null }, { "id": "897928", "postDate": "06/23/2020 07:22:45", "content": "\n\n\nWe were saying tweet would be a lottery, and in the end shakeup was rather limited, at least near top of LB.
\n
Same applies here.
\n\nThe fact that public notebooks did not overfit much is a sign of the increase of quality in Kaggle community in general. This is both good and bad. Good, because better public notebooks benefit all. Bad, because it is harder and harder to make a difference ;)
", "rawMarkdown": "> We were saying tweet would be a lottery, and in the end shakeup was rather limited, at least near top of LB.\n\nSame applies here. \n\nThe fact that public notebooks did not overfit much is a sign of the increase of quality in Kaggle community in general. This is both good and bad. Good, because better public notebooks benefit all. Bad, because it is harder and harder to make a difference ;)", "votes": null }, { "id": "897936", "postDate": "06/23/2020 07:25:45", "content": "\n\n\nWonder if there's an explanation for the lack of overfitting given that most of us were making hundreds of subs to fit to public LB.
\n
public/private split was random, and test data size is large enough.
", "rawMarkdown": "> Wonder if there's an explanation for the lack of overfitting given that most of us were making hundreds of subs to fit to public LB. \n\npublic/private split was random, and test data size is large enough.", "votes": null }, { "id": "897943", "postDate": "06/23/2020 07:30:23", "content": "Is there a good rule-of-thumb for determining a sufficient test data size for public LB a priori?
", "rawMarkdown": "Is there a good rule-of-thumb for determining a sufficient test data size for public LB a priori?", "votes": null }, { "id": "897972", "postDate": "06/23/2020 07:50:22", "content": "What other way than optimizing on public LB do you have in a competition like that? Only public included all six languages.
", "rawMarkdown": "What other way than optimizing on public LB do you have in a competition like that? Only public included all six languages.", "votes": null } ], "comments": [ { "id": 887912, "author_name": "mcggood", "author_url": "", "post_date": "06/16/2020 01:55:08", "content": "Valid 8k 5fold CV is good enough for me
", "votes": null, "replies": [ { "id": 889475, "author_name": "kannelliu", "author_url": "", "post_date": "06/17/2020 01:13:57", "content": "Hi bro, are you a uestc-er?
", "votes": null, "replies": [] }, { "id": 889841, "author_name": "mcggood", "author_url": "", "post_date": "06/17/2020 07:22:17", "content": "what's uestc-er? :)
", "votes": null, "replies": [] }, { "id": 890250, "author_name": "kannelliu", "author_url": "", "post_date": "06/17/2020 12:06:42", "content": "Haha, a university in chengdu.
", "votes": null, "replies": [] }, { "id": 891484, "author_name": "shahules", "author_url": "", "post_date": "06/18/2020 07:53:48", "content": "@mcggood so you're not splitting the training dataset? Only adding 4/5 th of valid data to it before training.
", "votes": null, "replies": [] }, { "id": 893274, "author_name": "mcggood", "author_url": "", "post_date": "06/19/2020 14:06:24", "content": "@shahules sorry, to be honest. I don't have a good CV plan. I'm overfitting personaly
", "votes": null, "replies": [] } ] }, { "id": 888820, "author_name": "sebastienm", "author_url": "", "post_date": "06/16/2020 15:24:06", "content": "One thing to note is that the test set contains 6 languages (tr, pt, ru, fr, it, es) while the validation set contains only 3 languages (tr, es, it), see e.g. this EDA notebook https://www.kaggle.com/ipythonx/jigsaw-multilingual-quick-eda-tpu-modeling
", "votes": null, "replies": [ { "id": 889827, "author_name": "aybatov", "author_url": "", "post_date": "06/17/2020 07:10:32", "content": "There is also spanish lang in test set. Hard to find really good set to make good CV, maybe some external datasets
", "votes": null, "replies": [] }, { "id": 889932, "author_name": "sebastienm", "author_url": "", "post_date": "06/17/2020 08:09:45", "content": "Fixed it, thanks! Indeed, CV consistently way above public LB, wondering if it is due to (pt, ru, fr).
", "votes": null, "replies": [] } ] }, { "id": 889600, "author_name": "haqishen", "author_url": "", "post_date": "06/17/2020 03:15:48", "content": "Wow! Are you going to take another gold within a week?
", "votes": null, "replies": [ { "id": 889643, "author_name": "cpmpml", "author_url": "", "post_date": "06/17/2020 03:45:36", "content": "I wish you were right ;)
", "votes": null, "replies": [] }, { "id": 897539, "author_name": "christofhenkel", "author_url": "", "post_date": "06/23/2020 00:22:50", "content": "I didn't think it was possible, and yet here we are. Very lucky last 2 days.
", "votes": null, "replies": [] }, { "id": 897563, "author_name": "sheriytm", "author_url": "", "post_date": "06/23/2020 00:48:53", "content": "Congrats @cpmpml and @christofhenkel. You guys inspired me to keep up in the last couple of days but I still missed the gold. However, I am left energized. See you in the next one.
", "votes": null, "replies": [] }, { "id": 897570, "author_name": "sebastienm", "author_url": "", "post_date": "06/23/2020 00:53:38", "content": "Yes, congrats, that's a super-human performance! \nIs there any track record of better than 4th place in a week ?
", "votes": null, "replies": [] }, { "id": 897914, "author_name": "haqishen", "author_url": "", "post_date": "06/23/2020 07:13:28", "content": "OMG!\nYou guys achieved something almost impossible!\nCongratulation!
", "votes": null, "replies": [] } ] }, { "id": 889922, "author_name": "philippsinger", "author_url": "", "post_date": "06/17/2020 08:03:53", "content": "Public LB :)
", "votes": null, "replies": [ { "id": 890236, "author_name": "christofhenkel", "author_url": "", "post_date": "06/17/2020 12:00:56", "content": "with only 5 subs per day, that puts us in a difficult position :D
", "votes": null, "replies": [] }, { "id": 891039, "author_name": "christofhenkel", "author_url": "", "post_date": "06/17/2020 21:10:07", "content": "Or in a good position if I think about it 😆
", "votes": null, "replies": [] } ] }, { "id": 890202, "author_name": "greatcodes", "author_url": "", "post_date": "06/17/2020 11:35:43", "content": "Stratified K-fold
", "votes": null, "replies": [] }, { "id": 891160, "author_name": "underwearfitting", "author_url": "", "post_date": "06/18/2020 00:56:07", "content": "I personally do not think there's a robust CV strategy. My CV score(.97+) is much higher than my LB score. Data augmentation is probably the key to this competition. Good luck.
", "votes": null, "replies": [ { "id": 891175, "author_name": "veryrobustperson", "author_url": "", "post_date": "06/18/2020 01:24:01", "content": "You have a CV of 0.97+? Wow
", "votes": null, "replies": [] }, { "id": 896946, "author_name": "yash612", "author_url": "", "post_date": "06/22/2020 14:34:05", "content": "since the competition is almost over , can you brief up about the data augmentation you did ?
", "votes": null, "replies": [] } ] }, { "id": 893021, "author_name": "kashnitsky", "author_url": "", "post_date": "06/19/2020 10:23:34", "content": "This will be one of the greatest shake-ups ever. Hope it turns out I'm wrong :)
", "votes": null, "replies": [ { "id": 893046, "author_name": "philippsinger", "author_url": "", "post_date": "06/19/2020 10:39:58", "content": "Greatest? Probably not. \nShakeup? Yes
\n\nI think all competitions that don't allow for proper CV are prone for shakeups.
", "votes": null, "replies": [] }, { "id": 893165, "author_name": "kashnitsky", "author_url": "", "post_date": "06/19/2020 12:36:28", "content": "Agreed. Considering that there's no hidden test set with new languages, maybe the shake-up will not be as bad as in MS malware or ELO. But still kinda lottery.
", "votes": null, "replies": [] }, { "id": 893178, "author_name": "cpmpml", "author_url": "", "post_date": "06/19/2020 12:45:43", "content": "We were saying tweet would be a lottery, and in the end shakeup was rather limited, at least near top of LB.
", "votes": null, "replies": [] }, { "id": 893184, "author_name": "shahules", "author_url": "", "post_date": "06/19/2020 12:46:53", "content": "@cpmpml yes, apart for some team shakeup was limited in tweets competition.Are you here to get the gold you just missed in tweets?😂 😂
", "votes": null, "replies": [] }, { "id": 893904, "author_name": "jay0606", "author_url": "", "post_date": "06/20/2020 04:33:11", "content": "I shaked down a few hundreds by not selecting the right submission in tweets 😅
", "votes": null, "replies": [] }, { "id": 894859, "author_name": "sheriytm", "author_url": "", "post_date": "06/20/2020 20:44:15", "content": "@cpmpml, in the case of tweet it was more like jump up for a lot of teams than the usual shakeup we see. I was even tempted to run @jtrotman's shakeup script on that competition if not because I need to devote time to Jigsaw.
", "votes": null, "replies": [] }, { "id": 895688, "author_name": "cpmpml", "author_url": "", "post_date": "06/21/2020 14:46:40", "content": "@shahules I am not solo here, even if we get a gold then it won't make for a solo gold. But I'd be very happy to get a team gold still! We are working hard on it, but time is very limited.
", "votes": null, "replies": [] }, { "id": 895690, "author_name": "cpmpml", "author_url": "", "post_date": "06/21/2020 14:47:15", "content": "@sheriytm what is @jtrotman shakeup script?
", "votes": null, "replies": [] }, { "id": 895872, "author_name": "sheriytm", "author_url": "", "post_date": "06/21/2020 17:01:23", "content": "@cpmpml I have not used them but here they are:-
\n\n", "votes": null, "replies": [] }, { "id": 897877, "author_name": "kashnitsky", "author_url": "", "post_date": "06/23/2020 06:53:45", "content": "\n\n\nYury Kashnitsky wrote:
\n \nThis will be one of the greatest shake-ups ever. Hope it turns out I'm wrong :)
\n
Narrator: Yury was wrong.
\n\nYury: Wow! Surprised with such robust results for many teams. Will be good to study their validations schemes even if I didn’t actively participate.
", "votes": null, "replies": [] }, { "id": 897880, "author_name": "christofhenkel", "author_url": "", "post_date": "06/23/2020 06:57:12", "content": "We calculated with more shakeup and did not select our best Public LB submission which included the best Public kernel, because we thought its heavily overfit. Turns out we were wrong :D
", "votes": null, "replies": [] }, { "id": 897885, "author_name": "shahules", "author_url": "", "post_date": "06/23/2020 06:58:28", "content": "@kashnitsky LB was the validation scheme.
", "votes": null, "replies": [] }, { "id": 897926, "author_name": "leecming", "author_url": "", "post_date": "06/23/2020 07:21:32", "content": "I was concerned about a shake-up as well but it ended up like 2018. If you look at the private/public LB for that competition, very stable as well.
\n\nWonder if there's an explanation for the lack of overfitting given that most of us were making hundreds of subs to fit to public LB.
", "votes": null, "replies": [] }, { "id": 897928, "author_name": "cpmpml", "author_url": "", "post_date": "06/23/2020 07:22:45", "content": "\n\n\nWe were saying tweet would be a lottery, and in the end shakeup was rather limited, at least near top of LB.
\n
Same applies here.
\n\nThe fact that public notebooks did not overfit much is a sign of the increase of quality in Kaggle community in general. This is both good and bad. Good, because better public notebooks benefit all. Bad, because it is harder and harder to make a difference ;)
", "votes": null, "replies": [] }, { "id": 897936, "author_name": "cpmpml", "author_url": "", "post_date": "06/23/2020 07:25:45", "content": "\n\n\nWonder if there's an explanation for the lack of overfitting given that most of us were making hundreds of subs to fit to public LB.
\n
public/private split was random, and test data size is large enough.
", "votes": null, "replies": [] }, { "id": 897943, "author_name": "leecming", "author_url": "", "post_date": "06/23/2020 07:30:23", "content": "Is there a good rule-of-thumb for determining a sufficient test data size for public LB a priori?
", "votes": null, "replies": [] }, { "id": 897972, "author_name": "philippsinger", "author_url": "", "post_date": "06/23/2020 07:50:22", "content": "What other way than optimizing on public LB do you have in a competition like that? Only public included all six languages.
", "votes": null, "replies": [] } ] }, { "id": 893739, "author_name": "naivelamb", "author_url": "", "post_date": "06/19/2020 20:58:04", "content": "Personally I don't think there is a robust cross-validation setup: validation set is too small and only 3 languages, public set is large but risky.
", "votes": null, "replies": [] }, { "id": 894002, "author_name": "christofhenkel", "author_url": "", "post_date": "06/20/2020 05:32:19", "content": "Looks like I found something working (only made 3 submissions so far):
\n\ncv 0.9287 LB 0.9282\ncv 0.9384 LB 0.9377\ncv 0.9400 LB 0.9410
", "votes": null, "replies": [] }, { "id": 894464, "author_name": "yassinealouini", "author_url": "", "post_date": "06/20/2020 12:58:00", "content": "Averaging many runs (by varying seeds) and using KFold? Haven't yet checked if it works but that's what I am implementing.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "887571": "Hey, I just joined this competition.\n\nI wanted to ask if anyone found out a better cross validation scheme than simple Kfold", "887912": "Valid 8k 5fold CV is good enough for me", "888820": "One thing to note is that the test set contains 6 languages (tr, pt, ru, fr, it, es) while the validation set contains only 3 languages (tr, es, it), see e.g. this EDA notebook https://www.kaggle.com/ipythonx/jigsaw-multilingual-quick-eda-tpu-modeling", "889475": "Hi bro, are you a uestc-er?", "889600": "Wow! Are you going to take another gold within a week?", "889643": "I wish you were right ;)", "889827": "There is also spanish lang in test set. Hard to find really good set to make good CV, maybe some external datasets", "889841": "what's uestc-er? :)", "889922": "Public LB :)", "889932": "Fixed it, thanks! Indeed, CV consistently way above public LB, wondering if it is due to (pt, ru, fr).", "890202": "Stratified K-fold", "890236": "with only 5 subs per day, that puts us in a difficult position :D", "890250": "Haha, a university in chengdu.", "891039": "Or in a good position if I think about it 😆", "891160": "I personally do not think there's a robust CV strategy. My CV score(.97+) is much higher than my LB score. Data augmentation is probably the key to this competition. Good luck.", "891175": "You have a CV of 0.97+? Wow", "891484": "mcggood so you're not splitting the training dataset? Only adding 4/5 th of valid data to it before training.", "893021": "This will be one of the greatest shake-ups ever. Hope it turns out I'm wrong :)", "893046": "Greatest? Probably not. \nShakeup? Yes\n\nI think all competitions that don't allow for proper CV are prone for shakeups.", "893165": "Agreed. Considering that there's no hidden test set with new languages, maybe the shake-up will not be as bad as in MS malware or ELO. But still kinda lottery.", "893178": "We were saying tweet would be a lottery, and in the end shakeup was rather limited, at least near top of LB.", "893184": "cpmpml yes, apart for some team shakeup was limited in tweets competition.Are you here to get the gold you just missed in tweets?😂 😂", "893274": "shahules sorry, to be honest. I don't have a good CV plan. I'm overfitting personaly", "893739": "Personally I don't think there is a robust cross-validation setup: validation set is too small and only 3 languages, public set is large but risky.", "893904": "I shaked down a few hundreds by not selecting the right submission in tweets 😅", "894002": "Looks like I found something working (only made 3 submissions so far):\n\ncv 0.9287 LB 0.9282\ncv 0.9384 LB 0.9377\ncv 0.9400 LB 0.9410", "894464": "Averaging many runs (by varying seeds) and using KFold? Haven't yet checked if it works but that's what I am implementing.", "894859": "cpmpml, in the case of tweet it was more like jump up for a lot of teams than the usual shakeup we see. I was even tempted to run @jtrotman's shakeup script on that competition if not because I need to devote time to Jigsaw.", "895688": "shahules I am not solo here, even if we get a gold then it won't make for a solo gold. But I'd be very happy to get a team gold still! We are working hard on it, but time is very limited.", "895690": "sheriytm what is @jtrotman shakeup script?", "895872": "cpmpml I have not used them but here they are:-\n\n1. [Meta Kaggle: Scatter Plot Competition Shake-up](https://www.kaggle.com/jtrotman/meta-kaggle-scatter-plot-competition-shake-up)\n\n2. [Meta Kaggle: Competition Shake-up](https://www.kaggle.com/jtrotman/meta-kaggle-competition-shake-up)", "896946": "since the competition is almost over , can you brief up about the data augmentation you did ?", "897539": "I didn't think it was possible, and yet here we are. Very lucky last 2 days.", "897563": "Congrats @cpmpml and @christofhenkel. You guys inspired me to keep up in the last couple of days but I still missed the gold. However, I am left energized. See you in the next one.", "897570": "Yes, congrats, that's a super-human performance! \nIs there any track record of better than 4th place in a week ?", "897877": "> **Yury Kashnitsky wrote:**\n> \n> This will be one of the greatest shake-ups ever. Hope it turns out I'm wrong :)\n\nNarrator: Yury was wrong. \n\nYury: Wow! Surprised with such robust results for many teams. Will be good to study their validations schemes even if I didn’t actively participate.", "897880": "We calculated with more shakeup and did not select our best Public LB submission which included the best Public kernel, because we thought its heavily overfit. Turns out we were wrong :D", "897885": "kashnitsky LB was the validation scheme.", "897914": "OMG!\nYou guys achieved something almost impossible!\nCongratulation!", "897926": "I was concerned about a shake-up as well but it ended up like 2018. If you look at the private/public LB for that competition, very stable as well.\n\nWonder if there's an explanation for the lack of overfitting given that most of us were making hundreds of subs to fit to public LB.", "897928": "> We were saying tweet would be a lottery, and in the end shakeup was rather limited, at least near top of LB.\n\nSame applies here. \n\nThe fact that public notebooks did not overfit much is a sign of the increase of quality in Kaggle community in general. This is both good and bad. Good, because better public notebooks benefit all. Bad, because it is harder and harder to make a difference ;)", "897936": "> Wonder if there's an explanation for the lack of overfitting given that most of us were making hundreds of subs to fit to public LB. \n\npublic/private split was random, and test data size is large enough.", "897943": "Is there a good rule-of-thumb for determining a sufficient test data size for public LB a priori?", "897972": "What other way than optimizing on public LB do you have in a competition like that? Only public included all six languages." }, "source": "meta" }