Hello everyone, the competition will almost end and it is time to start thinking about adding diversity and ensembling our base models them efficiently.
\n\nI am doing that by maximizing the OOF to determine what are the best weights. Also, my validation strategy is Triple Stratified KFold shared here with K == 3.
\n\nI want to have an idea about your best CV scores with some details like (blending strategy, validation strategy) since the leaderboard isn't reliable at all in this competition.
\n\nFor me my final ensemble gives me 0.947 CV with weighted averaging => 0.951 LB. I am working at improving this score.
", "messages": [ { "id": 962789, "postDate": "2020-08-08T12:49:13.913Z", "content": "Hello everyone, the competition will almost end and it is time to start thinking about adding diversity and ensembling our base models them efficiently.
\n\nI am doing that by maximizing the OOF to determine what are the best weights. Also, my validation strategy is Triple Stratified KFold shared here with K == 3.
\n\nI want to have an idea about your best CV scores with some details like (blending strategy, validation strategy) since the leaderboard isn't reliable at all in this competition.
\n\nFor me my final ensemble gives me 0.947 CV with weighted averaging => 0.951 LB. I am working at improving this score.
", "rawMarkdown": "Hello everyone, the competition will almost end and it is time to start thinking about adding diversity and ensembling our base models them efficiently.\n\nI am doing that by maximizing the OOF to determine what are the best weights. Also, my validation strategy is Triple Stratified KFold shared [here](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords/output) with K == 3. \n\nI want to have an idea about your best CV scores with some details like (blending strategy, validation strategy) since the leaderboard isn't reliable at all in this competition. \n\nFor me my final ensemble gives me 0.947 CV with weighted averaging => 0.951 LB. I am working at improving this score.", "votes": 13 }, { "id": 968485, "postDate": "2020-08-13T04:08:10.670Z", "content": "It's fascinating that many posts here have similar CV 0.950 but the LBs vary. I wonder where everyone will end on private LB. My ensemble CV is over 0.950 but my ensemble LB struggles to beat public LB ensembles. I'm curious to see if optimized CV ensembles beat public notebook LB ensembles.
", "rawMarkdown": "It's fascinating that many posts here have similar CV 0.950 but the LBs vary. I wonder where everyone will end on private LB. My ensemble CV is over 0.950 but my ensemble LB struggles to beat public LB ensembles. I'm curious to see if optimized CV ensembles beat public notebook LB ensembles.", "votes": 3, "replies": [ { "id": 968585, "postDate": "2020-08-13T05:57:28.993Z", "content": "@cdeotte thank you! I have been learning from your source! it helps me a lot!
\nMy cv is just based on ISIC 2020.
\nMy best cv is 9614. but this cv is not proportional to lb (LB : 9501), so I decided to abandon it
\nMy second cv choice is around 95.4 and proportional to LB with 0.1 meta submission added (LB : 95.6)
\nWhen second cv goes up, LB also goes up
\nBut the resultant LB is not enough to be in top 10% ( I am afraid that CV is not strong to get a good score at PRIVATE LB and far from the current PUBLIC LEADERBOARD)
What do you think CV of most people converges to around 0.950?
\nand what strategy is recommended to sample three submissions(we have three submissions)
\n(CV best, public LB best)
(My current LB is i guess try and error based overfitted LB)
", "rawMarkdown": "@cdeotte thank you! I have been learning from your source! it helps me a lot!\n\nMy cv is just based on ISIC 2020.\nMy best cv is 9614. but this cv is not proportional to lb (LB : 9501), so I decided to abandon it\nMy second cv choice is around 95.4 and proportional to LB with 0.1 meta submission added (LB : 95.6)\nWhen second cv goes up, LB also goes up\nBut the resultant LB is not enough to be in top 10% ( I am afraid that CV is not strong to get a good score at PRIVATE LB and far from the current PUBLIC LEADERBOARD) \n\nWhat do you think CV of most people converges to around 0.950?\nand what strategy is recommended to sample three submissions(we have three submissions)\n(CV best, public LB best)\n\n(My current LB is i guess try and error based overfitted LB)\n", "votes": 1 }, { "id": 968588, "postDate": "2020-08-13T06:02:51.140Z", "content": "Our CV ensemble struggles to beat your image only public model :D
", "rawMarkdown": "Our CV ensemble struggles to beat your image only public model :D", "votes": 3 }, { "id": 968622, "postDate": "2020-08-13T06:29:16.353Z", "content": "\n\nOur CV ensemble struggles to beat your image only public model :D
\n
That's weird. I wonder if it is a PyTorch versus TensorFlow thing (like Ion Comp π). All my offline TF models beat my public notebook in both CV and LB.
\nBut in this competition, I have no idea who will win. Perhaps PyTorch will beat TensorFlow on private LB
", "rawMarkdown": "> Our CV ensemble struggles to beat your image only public model :D\n\nThat's weird. I wonder if it is a PyTorch versus TensorFlow thing (like Ion Comp π). All my offline TF models beat my public notebook in both CV and LB.\n\nBut in this competition, I have no idea who will win. Perhaps PyTorch will beat TensorFlow on private LB", "votes": 1 }, { "id": 968641, "postDate": "2020-08-13T06:44:38.520Z", "content": "\n\nwhat strategy is recommended to sample three submissions(we have three submissions)
\n
Great question Statking ( @deepkim ). I think this comp is all about picking wisely our final three submissions. I wouldn't be surprised if someone wins a gold medal from ensembling public notebooks in a smart way.
", "rawMarkdown": "> what strategy is recommended to sample three submissions(we have three submissions)\n\nGreat question Statking ( @deepkim ). I think this comp is all about picking wisely our final three submissions. I wouldn't be surprised if someone wins a gold medal from ensembling public notebooks in a smart way.", "votes": 2 }, { "id": 968727, "postDate": "2020-08-13T07:53:21.063Z", "content": "@cdeotte
\nI hope it is not the case, but it seems somehow like that.
\nTo be fair your public NB does not have the highest CV, I think it is 0.910, right?
@philippsinger Hi, Psi. I used chris' kernel before, and modified something. It can got a cv around 930(LB 94+) or 92+(LB 94+ or 95+)
", "rawMarkdown": "@philippsinger Hi, Psi. I used chris' kernel before, and modified something. It can got a cv around 930(LB 94+) or 92+(LB 94+ or 95+)" }, { "id": 968785, "postDate": "2020-08-13T08:32:19.140Z", "content": "anyone tried blending tf and pytorch models together? how is it performing?
\ni think more diversity and models from 2 different framework together can do well in private lb but pytorch is not doing great job for us!!
My best stacking :
\nOOF CV SCORE : 0.95075
\nAVG CV AUC : 0.9522
\nLB -> 0.9439
It's a mystery, I feel like I failed the most basic part of a Kaggle competition : getting a reliable CV. (I'm using triple stratified no duplicates strategy, similar from @cdeotte) I got better results with single model that had CV around 0.92β¦
\nNow I'm starting to think (and secretly hope?) that the shake up is going to be real bad. On the other hand I really hope to learn something from the top solutions as I feel like I exhausted most of my ideas. (all in all, except from @cdeotte who shared an amazing amount of work, I feel like discussions failed to make ideas emerged for this competition).
\nIt's a bit pathetic but seeing @philippsinger struggling too kind of cheers me up! π
", "rawMarkdown": "My best stacking :\nOOF CV SCORE : 0.95075\nAVG CV AUC : 0.9522\nLB -> 0.9439\n\nIt's a mystery, I feel like I failed the most basic part of a Kaggle competition : getting a reliable CV. (I'm using triple stratified no duplicates strategy, similar from @cdeotte) I got better results with single model that had CV around 0.92...\n\nNow I'm starting to think (and secretly hope?) that the shake up is going to be real bad. On the other hand I really hope to learn something from the top solutions as I feel like I exhausted most of my ideas. (all in all, except from @cdeotte who shared an amazing amount of work, I feel like discussions failed to make ideas emerged for this competition).\n\nIt's a bit pathetic but seeing @philippsinger struggling too kind of cheers me up! π" }, { "id": 968816, "postDate": "2020-08-13T09:02:32.283Z", "content": "@cdeotte thank you for the sincere reply!
", "rawMarkdown": "@cdeotte thank you for the sincere reply!", "votes": 1 }, { "id": 968847, "postDate": "2020-08-13T09:24:53.800Z", "content": "@garybios thats what I mean, CV LB gap is crazy
\n@optimo first time I cant beat public baseline, we will see if that is good or bad :)
", "rawMarkdown": "@garybios thats what I mean, CV LB gap is crazy\n\n@optimo first time I cant beat public baseline, we will see if that is good or bad :)", "votes": 2 }, { "id": 969010, "postDate": "2020-08-13T12:02:43.960Z", "content": "@mobassir my current score is a blend of Pytorch and TF models (something like 70 % Pytorch and 30% TF)
\n@yuval6967 reported much earlier a single Pytorch model scoring 0.960 on LB
", "rawMarkdown": "@mobassir my current score is a blend of Pytorch and TF models (something like 70 % Pytorch and 30% TF)\n\n@yuval6967 reported much earlier a single Pytorch model scoring 0.960 on LB \n\n", "votes": 1 }, { "id": 969202, "postDate": "2020-08-13T14:42:28.750Z", "content": "@serigne sir are you using shonenkov's kernel? sorry if you do not want to disclose early, but thanks for the info,,i think your approach is much trustable for private lb
", "rawMarkdown": "@serigne sir are you using shonenkov's kernel? sorry if you do not want to disclose early, but thanks for the info,,i think your approach is much trustable for private lb ", "votes": 1, "replies": [ { "id": 969270, "postDate": "2020-08-13T15:22:58.680Z", "content": "I was using it at the begninning but ended up building my own. I couldn't make his kernel running \" as is\" on Colab.
\nMaking Pytorch TPU up and running is really tricky.
\nThanks but I don't trust that much my current best score ;)
\nI think we are many overfitting here :)
", "rawMarkdown": "I was using it at the begninning but ended up building my own. I couldn't make his kernel running \" as is\" on Colab. \n\nMaking Pytorch TPU up and running is really tricky. \n\n\nThanks but I don't trust that much my current best score ;) \n\nI think we are many overfitting here :)\n\n", "votes": 1 } ] }, { "id": 971014, "postDate": "2020-08-15T05:08:15.707Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 968773, "postDate": "2020-08-13T08:24:57.337Z", "content": "ensemble cv 0.946~0.953, lb 0.950~0.955, We analyzing about 250 images of the top probabilities, and looking for the strategy for how to submit them.
\nTrust CV or Trust LB?
", "rawMarkdown": "ensemble cv 0.946~0.953, lb 0.950~0.955, We analyzing about 250 images of the top probabilities, and looking for the strategy for how to submit them. \n\nTrust CV or Trust LB?", "votes": 1, "replies": [ { "id": 968783, "postDate": "2020-08-13T08:28:19.407Z", "content": "@yeonmin put trust on both
\nwe have 3 final submissions to chose for this competition
@yeonmin keep in mind that you are not allowed to do any kind of manual interaction with the test predictions (e.g., manual labeling)
", "rawMarkdown": "@yeonmin keep in mind that you are not allowed to do any kind of manual interaction with the test predictions (e.g., manual labeling)", "votes": 2 }, { "id": 969090, "postDate": "2020-08-13T12:59:05.217Z", "content": "Yes. We don't even do hand labeling for already known leaks. Does the test data leak that everyone knows carry out Hand Labeling?
", "rawMarkdown": "Yes. We don't even do hand labeling for already known leaks. Does the test data leak that everyone knows carry out Hand Labeling?" }, { "id": 969091, "postDate": "2020-08-13T13:00:05.637Z", "content": "You mean those 6 images?
", "rawMarkdown": "You mean those 6 images?", "votes": 1 }, { "id": 969115, "postDate": "2020-08-13T13:15:44.977Z", "content": "Is it okay to label using an algorithm that finds duplicate data in existing data without hand labeling??
\nlabel1 = ['ISIC9353360', 'ISIC9207777', 'ISIC5224960', 'ISIC6457527', 'ISIC8372206']
\nlabel0 = ['ISIC3689290', 'ISIC3584949', 'ISIC8347588']
Yes this is OK.
", "rawMarkdown": "Yes this is OK." }, { "id": 971207, "postDate": "2020-08-15T09:31:14.217Z", "content": "You can also include these images in your train set. Your model might be able to remember them when you make your test predictions
", "rawMarkdown": "You can also include these images in your train set. Your model might be able to remember them when you make your test predictions" } ] }, { "id": 967719, "postDate": "2020-08-12T13:04:59.780Z", "content": "Mine is CV 0.9477 LB 0.9530 validate only on isic2020
", "rawMarkdown": "Mine is CV 0.9477 LB 0.9530 validate only on isic2020", "votes": 1, "replies": [ { "id": 967953, "postDate": "2020-08-12T15:46:19.733Z", "content": "Keep the good work : )
", "rawMarkdown": "Keep the good work : )" } ] }, { "id": 965892, "postDate": "2020-08-11T00:18:39.503Z", "content": "Nice one !! Keep sharing
", "rawMarkdown": "Nice one !! Keep sharing", "votes": 1 }, { "id": 967891, "postDate": "2020-08-12T15:07:09.320Z", "content": "CV .950, LB .945. No external data, no post processing. I ensemble a few EfficientNet and DenseNet models.
", "rawMarkdown": "CV .950, LB .945. No external data, no post processing. I ensemble a few EfficientNet and DenseNet models.", "votes": 2, "replies": [ { "id": 967952, "postDate": "2020-08-12T15:46:04.187Z", "content": "Sounds interesting, I will be glad to see your solution when the competition ends
", "rawMarkdown": "Sounds interesting, I will be glad to see your solution when the competition ends" } ] }, { "id": 965769, "postDate": "2020-08-10T20:54:12.780Z", "content": "Ensemble oof cv of 0.9517 only validating on 2020 data
\nLB 0.9549
Sounds promising, are you using post processing techniques ?
", "rawMarkdown": "Sounds promising, are you using post processing techniques ? ", "votes": 1 }, { "id": 965774, "postDate": "2020-08-10T21:00:16.020Z", "content": "Using a simple optimization to find the best weights for each base model that maximize the oof roc auc.
", "rawMarkdown": "Using a simple optimization to find the best weights for each base model that maximize the oof roc auc.", "votes": 1 }, { "id": 968486, "postDate": "2020-08-13T04:09:39.263Z", "content": "Great job beating CV 0.950 @ragnar123
", "rawMarkdown": "Great job beating CV 0.950 @ragnar123 ", "votes": 1 } ] }, { "id": 962841, "postDate": "2020-08-08T13:30:58.723Z", "content": "our ensemble strategy is also very similar :)\nwe have got CV 0.950461638257698 \nand LB 0.9505 (for minmax)\nsince the public LB of this competition is of no use,we are still unsure about the upper and lower bound threshold for final submission
\n\nanother submission where we took simple average had : \nCV = 0.9296710190571207\nLb = 0.9511\none question,if you do not mind - \"is your current best LB submission a blend of blends or nested blend? :)\"
", "rawMarkdown": "our ensemble strategy is also very similar :)\nwe have got CV 0.950461638257698 \nand LB 0.9505 (for minmax)\nsince the public LB of this competition is of no use,we are still unsure about the upper and lower bound threshold for final submission\n\nanother submission where we took simple average had : \nCV = 0.9296710190571207\nLb = 0.9511\none question,if you do not mind - \"is your current best LB submission a blend of blends or nested blend? :)\"", "votes": 2, "replies": [ { "id": 962866, "postDate": "2020-08-08T13:56:39.297Z", "content": "My current best LB submission is a blend of blends, \nSince we have 3 submissions, I will rely on luck on one submission. I don't like doing that but many will do it.
", "rawMarkdown": "My current best LB submission is a blend of blends, \nSince we have 3 submissions, I will rely on luck on one submission. I don't like doing that but many will do it.", "votes": 1 }, { "id": 962867, "postDate": "2020-08-08T13:57:03.430Z", "content": "Btw 0.950 is a good CV score. Continue the good work.
", "rawMarkdown": "Btw 0.950 is a good CV score. Continue the good work.", "votes": 1 }, { "id": 962874, "postDate": "2020-08-08T14:08:17.190Z", "content": "i did few blends of blend and i realized that it is not ML so just stopped doing it,,,planning to submit 3 good blends, if my team fails then we will not mind,,there is a possibility that 3 blends of blends submission can win lottery,,doesn't matter,, we will trust on CV and happy if we fail with good CV :)\ni am curious how people going pass 0.97 \nthe best single model thread is kind of frozen,,reminds me of \"tweet sentiment competition\" :)
", "rawMarkdown": "i did few blends of blend and i realized that it is not ML so just stopped doing it,,,planning to submit 3 good blends, if my team fails then we will not mind,,there is a possibility that 3 blends of blends submission can win lottery,,doesn't matter,, we will trust on CV and happy if we fail with good CV :)\ni am curious how people going pass 0.97 \nthe best single model thread is kind of frozen,,reminds me of \"tweet sentiment competition\" :)", "votes": 2 }, { "id": 962900, "postDate": "2020-08-08T14:27:29.290Z", "content": "Selecting good models and not relying on luck is indeed a very very important skill.\nI don't like having 3 subs in this competition.
", "rawMarkdown": "Selecting good models and not relying on luck is indeed a very very important skill.\nI don't like having 3 subs in this competition.", "votes": 1 }, { "id": 964107, "postDate": "2020-08-09T15:49:35.320Z", "content": "Thank you for sharing.If i didnt get your point wrong, your CV scores is 0.950 for MinMax, 0.929 for simple average?
", "rawMarkdown": "Thank you for sharing.If i didnt get your point wrong, your CV scores is 0.950 for MinMax, 0.929 for simple average?", "votes": 1 }, { "id": 964130, "postDate": "2020-08-09T16:10:53.567Z", "content": "@changewow yes
", "rawMarkdown": "@changewow yes" }, { "id": 965451, "postDate": "2020-08-10T16:40:39.797Z", "content": "What is MinMax?
", "rawMarkdown": "What is MinMax?" }, { "id": 965453, "postDate": "2020-08-10T16:43:39.370Z", "content": "@philippsinger sir sorry,i mean MinMax ensemble,please check this : https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/167465
", "rawMarkdown": "@philippsinger sir sorry,i mean MinMax ensemble,please check this : https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/167465" }, { "id": 965524, "postDate": "2020-08-10T17:37:43.970Z", "content": "weird stuff, this also works on CV? it seems very overfitty
", "rawMarkdown": "weird stuff, this also works on CV? it seems very overfitty", "votes": 1 }, { "id": 965531, "postDate": "2020-08-10T17:42:05.003Z", "content": "@philippsinger sorry\nthere is a mistake\ncorrection : \nSimple Average Cv : 0.9507\nwith MinMax Cv : 0.94818
", "rawMarkdown": "@philippsinger sorry\nthere is a mistake\ncorrection : \nSimple Average Cv : 0.9507\nwith MinMax Cv : 0.94818" }, { "id": 965536, "postDate": "2020-08-10T17:44:31.677Z", "content": "What is CV and LB now for simple vs minmax?
", "rawMarkdown": "What is CV and LB now for simple vs minmax?" }, { "id": 965556, "postDate": "2020-08-10T17:54:01.467Z", "content": "@philippsinger \nLB 0.9346 for minmax
\n\nLb = 0.9511 for simple
", "rawMarkdown": "@philippsinger \nLB 0.9346 for minmax\n\nLb = 0.9511 for simple" }, { "id": 965683, "postDate": "2020-08-10T19:02:39.803Z", "content": "So to summarize:
\n\nMinmax CV 0.9507 LB 0.9346\nSimple CV: 0.94818 LB 0.9511
\n\n?
", "rawMarkdown": "So to summarize:\n\nMinmax CV 0.9507 LB 0.9346\nSimple CV: 0.94818 LB 0.9511\n\n?" }, { "id": 965691, "postDate": "2020-08-10T19:10:50.470Z", "content": "@philippsinger \nSimple Average Cv : 0.9507 LB 0.9511\nwith MinMax Cv : 0.94818 LB 0.9346
\n\nsimple average maybe better
", "rawMarkdown": "@philippsinger \nSimple Average Cv : 0.9507 LB 0.9511\nwith MinMax Cv : 0.94818 LB 0.9346\n\nsimple average maybe better" }, { "id": 965697, "postDate": "2020-08-10T19:23:51.527Z", "content": "@philippsinger another simple average result : \ncv 0.951118782197574\nlb 0.9502
", "rawMarkdown": "@philippsinger another simple average result : \ncv 0.951118782197574\nlb 0.9502" }, { "id": 968489, "postDate": "2020-08-13T04:10:57.853Z", "content": "Great job beating CV 0.950 @mobassir
", "rawMarkdown": "Great job beating CV 0.950 @mobassir ", "votes": 1 }, { "id": 968946, "postDate": "2020-08-13T11:09:42.320Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 962795, "postDate": "2020-08-08T12:52:52.847Z", "content": "Do you use external data with your ensemble 0.947 CV?
", "rawMarkdown": "Do you use external data with your ensemble 0.947 CV?", "votes": 2, "replies": [ { "id": 962818, "postDate": "2020-08-08T13:04:25.493Z", "content": "Yes, I am using external data provided here and I only use images of this competition during validation.
", "rawMarkdown": "Yes, I am using external data provided [here](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/164910) and I only use images of this competition during validation.", "votes": 1 } ] }, { "id": 970763, "postDate": "2020-08-14T18:47:55.513Z", "content": "My best oof cv is 9669 using the methods in this notebook(https://www.kaggle.com/steubk/simple-oof-ensembling-methods-for-classification), with the lb score being 9453 so im not sure if i should trust it. My lb score currently is an ensemble of oof scores 91-94 with most of them being on the lower end
", "rawMarkdown": "My best oof cv is 9669 using the methods in this notebook(https://www.kaggle.com/steubk/simple-oof-ensembling-methods-for-classification), with the lb score being 9453 so im not sure if i should trust it. My lb score currently is an ensemble of oof scores 91-94 with most of them being on the lower end" }, { "id": 965264, "postDate": "2020-08-10T13:59:37.650Z", "content": "I have one question to all of you, is your CV resistant/robust to seed change? I mean have you guys tried keeping the same setup but just changing the seed and what is ur CV after that?
", "rawMarkdown": "I have one question to all of you, is your CV resistant/robust to seed change? I mean have you guys tried keeping the same setup but just changing the seed and what is ur CV after that?", "replies": [ { "id": 965269, "postDate": "2020-08-10T14:06:19.377Z", "content": "If you change the seed, results will change a bit. Fortunately with ensembling we can reduce this variance.
", "rawMarkdown": "If you change the seed, results will change a bit. Fortunately with ensembling we can reduce this variance. ", "replies": [ { "id": 965275, "postDate": "2020-08-10T14:13:11.723Z", "content": "Ya, I understand that. But can u roughly say what is the difference between ur best CV and the seed changed CV?
", "rawMarkdown": "Ya, I understand that. But can u roughly say what is the difference between ur best CV and the seed changed CV?", "votes": 1 }, { "id": 967590, "postDate": "2020-08-12T11:12:31.630Z", "content": "I tried to retrain some base models with a different seed and the CV score of each model varies between +- 0.002. This difference must be less when I ensemble but I will need to retrain everything : ) No time for that
", "rawMarkdown": "I tried to retrain some base models with a different seed and the CV score of each model varies between +- 0.002. This difference must be less when I ensemble but I will need to retrain everything : ) No time for that " }, { "id": 967611, "postDate": "2020-08-12T11:32:02.513Z", "content": "Given that ur cv fluctuation is +-0.002, I think your models are quite robust with respect to others' models I am reading in the discussion. Nice work!
", "rawMarkdown": "Given that ur cv fluctuation is +-0.002, I think your models are quite robust with respect to others' models I am reading in the discussion. Nice work! " } ] } ] }, { "id": 963572, "postDate": "2020-08-09T06:13:09.607Z", "content": "My Best CV 9614, LB : 9510\nSecond CV 9530 LB 9557\nWhich one should I trust? I have no idea!
", "rawMarkdown": "My Best CV 9614, LB : 9510\nSecond CV 9530 LB 9557\nWhich one should I trust? I have no idea!\n", "replies": [ { "id": 963663, "postDate": "2020-08-09T07:44:08.047Z", "content": "The one with the highest cv if you believe your validation strategy is the right one
", "rawMarkdown": "The one with the highest cv if you believe your validation strategy is the right one", "votes": 1 }, { "id": 964067, "postDate": "2020-08-09T15:07:23.773Z", "content": "What I am afraid of is that the highest cv is also overfitted to training sets since I guess the training sets and the test sets have different distributions each other\nI have been using chris's stratified data
", "rawMarkdown": "What I am afraid of is that the highest cv is also overfitted to training sets since I guess the training sets and the test sets have different distributions each other\nI have been using chris's stratified data" }, { "id": 965444, "postDate": "2020-08-10T16:31:17.300Z", "content": "Be careful when you add external data :)
", "rawMarkdown": "Be careful when you add external data :) ", "votes": 1 }, { "id": 968487, "postDate": "2020-08-13T04:10:17.913Z", "content": "Great job @deepkim , CV over 0.960 is fantastic!
", "rawMarkdown": "Great job @deepkim , CV over 0.960 is fantastic!", "votes": 1 } ] }, { "id": 968915, "postDate": "2020-08-13T10:41:57.073Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 968998, "postDate": "2020-08-13T11:53:05.363Z", "content": "Why donβt you look at your CV using metadata?
", "rawMarkdown": "Why donβt you look at your CV using metadata?" } ] } ], "comments": [ { "id": 968485, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-13T04:08:10.670000", "content": "It's fascinating that many posts here have similar CV 0.950 but the LBs vary. I wonder where everyone will end on private LB. My ensemble CV is over 0.950 but my ensemble LB struggles to beat public LB ensembles. I'm curious to see if optimized CV ensembles beat public notebook LB ensembles.
", "votes": 3, "replies": [ { "id": 968585, "author_name": "kaggler", "author_url": "", "post_date": "2020-08-13T05:57:28.993000", "content": "@cdeotte thank you! I have been learning from your source! it helps me a lot!
\nMy cv is just based on ISIC 2020.
\nMy best cv is 9614. but this cv is not proportional to lb (LB : 9501), so I decided to abandon it
\nMy second cv choice is around 95.4 and proportional to LB with 0.1 meta submission added (LB : 95.6)
\nWhen second cv goes up, LB also goes up
\nBut the resultant LB is not enough to be in top 10% ( I am afraid that CV is not strong to get a good score at PRIVATE LB and far from the current PUBLIC LEADERBOARD)
What do you think CV of most people converges to around 0.950?
\nand what strategy is recommended to sample three submissions(we have three submissions)
\n(CV best, public LB best)
(My current LB is i guess try and error based overfitted LB)
", "votes": 1, "replies": [] }, { "id": 968588, "author_name": "Psi", "author_url": "", "post_date": "2020-08-13T06:02:51.140000", "content": "Our CV ensemble struggles to beat your image only public model :D
", "votes": 3, "replies": [] }, { "id": 968622, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-13T06:29:16.353000", "content": "\n\nOur CV ensemble struggles to beat your image only public model :D
\n
That's weird. I wonder if it is a PyTorch versus TensorFlow thing (like Ion Comp π). All my offline TF models beat my public notebook in both CV and LB.
\nBut in this competition, I have no idea who will win. Perhaps PyTorch will beat TensorFlow on private LB
", "votes": 1, "replies": [] }, { "id": 968641, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-13T06:44:38.520000", "content": "\n\nwhat strategy is recommended to sample three submissions(we have three submissions)
\n
Great question Statking ( @deepkim ). I think this comp is all about picking wisely our final three submissions. I wouldn't be surprised if someone wins a gold medal from ensembling public notebooks in a smart way.
", "votes": 2, "replies": [] }, { "id": 968727, "author_name": "Psi", "author_url": "", "post_date": "2020-08-13T07:53:21.063000", "content": "@cdeotte
\nI hope it is not the case, but it seems somehow like that.
\nTo be fair your public NB does not have the highest CV, I think it is 0.910, right?
@philippsinger Hi, Psi. I used chris' kernel before, and modified something. It can got a cv around 930(LB 94+) or 92+(LB 94+ or 95+)
", "votes": 0, "replies": [] }, { "id": 968785, "author_name": "Mobassir", "author_url": "", "post_date": "2020-08-13T08:32:19.140000", "content": "anyone tried blending tf and pytorch models together? how is it performing?
\ni think more diversity and models from 2 different framework together can do well in private lb but pytorch is not doing great job for us!!
My best stacking :
\nOOF CV SCORE : 0.95075
\nAVG CV AUC : 0.9522
\nLB -> 0.9439
It's a mystery, I feel like I failed the most basic part of a Kaggle competition : getting a reliable CV. (I'm using triple stratified no duplicates strategy, similar from @cdeotte) I got better results with single model that had CV around 0.92β¦
\nNow I'm starting to think (and secretly hope?) that the shake up is going to be real bad. On the other hand I really hope to learn something from the top solutions as I feel like I exhausted most of my ideas. (all in all, except from @cdeotte who shared an amazing amount of work, I feel like discussions failed to make ideas emerged for this competition).
\nIt's a bit pathetic but seeing @philippsinger struggling too kind of cheers me up! π
", "votes": 0, "replies": [] }, { "id": 968816, "author_name": "kaggler", "author_url": "", "post_date": "2020-08-13T09:02:32.283000", "content": "@cdeotte thank you for the sincere reply!
", "votes": 1, "replies": [] }, { "id": 968847, "author_name": "Psi", "author_url": "", "post_date": "2020-08-13T09:24:53.800000", "content": "@garybios thats what I mean, CV LB gap is crazy
\n@optimo first time I cant beat public baseline, we will see if that is good or bad :)
", "votes": 2, "replies": [] }, { "id": 969010, "author_name": "Serigne ", "author_url": "", "post_date": "2020-08-13T12:02:43.960000", "content": "@mobassir my current score is a blend of Pytorch and TF models (something like 70 % Pytorch and 30% TF)
\n@yuval6967 reported much earlier a single Pytorch model scoring 0.960 on LB
", "votes": 1, "replies": [] }, { "id": 969202, "author_name": "Mobassir", "author_url": "", "post_date": "2020-08-13T14:42:28.750000", "content": "@serigne sir are you using shonenkov's kernel? sorry if you do not want to disclose early, but thanks for the info,,i think your approach is much trustable for private lb
", "votes": 1, "replies": [ { "id": 969270, "author_name": "Serigne ", "author_url": "", "post_date": "2020-08-13T15:22:58.680000", "content": "I was using it at the begninning but ended up building my own. I couldn't make his kernel running \" as is\" on Colab.
\nMaking Pytorch TPU up and running is really tricky.
\nThanks but I don't trust that much my current best score ;)
\nI think we are many overfitting here :)
", "votes": 1, "replies": [] } ] }, { "id": 971014, "author_name": "", "author_url": "", "post_date": "2020-08-15T05:08:15.707000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 968773, "author_name": "yeonmin", "author_url": "", "post_date": "2020-08-13T08:24:57.337000", "content": "ensemble cv 0.946~0.953, lb 0.950~0.955, We analyzing about 250 images of the top probabilities, and looking for the strategy for how to submit them.
\nTrust CV or Trust LB?
", "votes": 1, "replies": [ { "id": 968783, "author_name": "Mobassir", "author_url": "", "post_date": "2020-08-13T08:28:19.407000", "content": "@yeonmin put trust on both
\nwe have 3 final submissions to chose for this competition
@yeonmin keep in mind that you are not allowed to do any kind of manual interaction with the test predictions (e.g., manual labeling)
", "votes": 2, "replies": [] }, { "id": 969090, "author_name": "yeonmin", "author_url": "", "post_date": "2020-08-13T12:59:05.217000", "content": "Yes. We don't even do hand labeling for already known leaks. Does the test data leak that everyone knows carry out Hand Labeling?
", "votes": 0, "replies": [] }, { "id": 969091, "author_name": "Psi", "author_url": "", "post_date": "2020-08-13T13:00:05.637000", "content": "You mean those 6 images?
", "votes": 1, "replies": [] }, { "id": 969115, "author_name": "yeonmin", "author_url": "", "post_date": "2020-08-13T13:15:44.977000", "content": "Is it okay to label using an algorithm that finds duplicate data in existing data without hand labeling??
\nlabel1 = ['ISIC9353360', 'ISIC9207777', 'ISIC5224960', 'ISIC6457527', 'ISIC8372206']
\nlabel0 = ['ISIC3689290', 'ISIC3584949', 'ISIC8347588']
Yes this is OK.
", "votes": 0, "replies": [] }, { "id": 971207, "author_name": "datasaurus", "author_url": "", "post_date": "2020-08-15T09:31:14.217000", "content": "You can also include these images in your train set. Your model might be able to remember them when you make your test predictions
", "votes": 0, "replies": [] } ] }, { "id": 967719, "author_name": "toto", "author_url": "", "post_date": "2020-08-12T13:04:59.780000", "content": "Mine is CV 0.9477 LB 0.9530 validate only on isic2020
", "votes": 1, "replies": [ { "id": 967953, "author_name": "Seifeddine Fezzani", "author_url": "", "post_date": "2020-08-12T15:46:19.733000", "content": "Keep the good work : )
", "votes": 0, "replies": [] } ] }, { "id": 965892, "author_name": "__n1kshaN__", "author_url": "", "post_date": "2020-08-11T00:18:39.503000", "content": "Nice one !! Keep sharing
", "votes": 1, "replies": [] }, { "id": 967891, "author_name": "Dave Lorenz", "author_url": "", "post_date": "2020-08-12T15:07:09.320000", "content": "CV .950, LB .945. No external data, no post processing. I ensemble a few EfficientNet and DenseNet models.
", "votes": 2, "replies": [ { "id": 967952, "author_name": "Seifeddine Fezzani", "author_url": "", "post_date": "2020-08-12T15:46:04.187000", "content": "Sounds interesting, I will be glad to see your solution when the competition ends
", "votes": 0, "replies": [] } ] }, { "id": 965769, "author_name": "Martin Kovacevic Buvinic", "author_url": "", "post_date": "2020-08-10T20:54:12.780000", "content": "Ensemble oof cv of 0.9517 only validating on 2020 data
\nLB 0.9549
Sounds promising, are you using post processing techniques ?
", "votes": 1, "replies": [] }, { "id": 965774, "author_name": "Martin Kovacevic Buvinic", "author_url": "", "post_date": "2020-08-10T21:00:16.020000", "content": "Using a simple optimization to find the best weights for each base model that maximize the oof roc auc.
", "votes": 1, "replies": [] }, { "id": 968486, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-13T04:09:39.263000", "content": "Great job beating CV 0.950 @ragnar123
", "votes": 1, "replies": [] } ] }, { "id": 962841, "author_name": "Mobassir", "author_url": "", "post_date": "2020-08-08T13:30:58.723000", "content": "our ensemble strategy is also very similar :)\nwe have got CV 0.950461638257698 \nand LB 0.9505 (for minmax)\nsince the public LB of this competition is of no use,we are still unsure about the upper and lower bound threshold for final submission
\n\nanother submission where we took simple average had : \nCV = 0.9296710190571207\nLb = 0.9511\none question,if you do not mind - \"is your current best LB submission a blend of blends or nested blend? :)\"
", "votes": 2, "replies": [ { "id": 962866, "author_name": "Seifeddine Fezzani", "author_url": "", "post_date": "2020-08-08T13:56:39.297000", "content": "My current best LB submission is a blend of blends, \nSince we have 3 submissions, I will rely on luck on one submission. I don't like doing that but many will do it.
", "votes": 1, "replies": [] }, { "id": 962867, "author_name": "Seifeddine Fezzani", "author_url": "", "post_date": "2020-08-08T13:57:03.430000", "content": "Btw 0.950 is a good CV score. Continue the good work.
", "votes": 1, "replies": [] }, { "id": 962874, "author_name": "Mobassir", "author_url": "", "post_date": "2020-08-08T14:08:17.190000", "content": "i did few blends of blend and i realized that it is not ML so just stopped doing it,,,planning to submit 3 good blends, if my team fails then we will not mind,,there is a possibility that 3 blends of blends submission can win lottery,,doesn't matter,, we will trust on CV and happy if we fail with good CV :)\ni am curious how people going pass 0.97 \nthe best single model thread is kind of frozen,,reminds me of \"tweet sentiment competition\" :)
", "votes": 2, "replies": [] }, { "id": 962900, "author_name": "Seifeddine Fezzani", "author_url": "", "post_date": "2020-08-08T14:27:29.290000", "content": "Selecting good models and not relying on luck is indeed a very very important skill.\nI don't like having 3 subs in this competition.
", "votes": 1, "replies": [] }, { "id": 964107, "author_name": "shiba", "author_url": "", "post_date": "2020-08-09T15:49:35.320000", "content": "Thank you for sharing.If i didnt get your point wrong, your CV scores is 0.950 for MinMax, 0.929 for simple average?
", "votes": 1, "replies": [] }, { "id": 964130, "author_name": "Mobassir", "author_url": "", "post_date": "2020-08-09T16:10:53.567000", "content": "@changewow yes
", "votes": 0, "replies": [] }, { "id": 965451, "author_name": "Psi", "author_url": "", "post_date": "2020-08-10T16:40:39.797000", "content": "What is MinMax?
", "votes": 0, "replies": [] }, { "id": 965453, "author_name": "Mobassir", "author_url": "", "post_date": "2020-08-10T16:43:39.370000", "content": "@philippsinger sir sorry,i mean MinMax ensemble,please check this : https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/167465
", "votes": 0, "replies": [] }, { "id": 965524, "author_name": "Psi", "author_url": "", "post_date": "2020-08-10T17:37:43.970000", "content": "weird stuff, this also works on CV? it seems very overfitty
", "votes": 1, "replies": [] }, { "id": 965531, "author_name": "Mobassir", "author_url": "", "post_date": "2020-08-10T17:42:05.003000", "content": "@philippsinger sorry\nthere is a mistake\ncorrection : \nSimple Average Cv : 0.9507\nwith MinMax Cv : 0.94818
", "votes": 0, "replies": [] }, { "id": 965536, "author_name": "Psi", "author_url": "", "post_date": "2020-08-10T17:44:31.677000", "content": "What is CV and LB now for simple vs minmax?
", "votes": 0, "replies": [] }, { "id": 965556, "author_name": "Mobassir", "author_url": "", "post_date": "2020-08-10T17:54:01.467000", "content": "@philippsinger \nLB 0.9346 for minmax
\n\nLb = 0.9511 for simple
", "votes": 0, "replies": [] }, { "id": 965683, "author_name": "Psi", "author_url": "", "post_date": "2020-08-10T19:02:39.803000", "content": "So to summarize:
\n\nMinmax CV 0.9507 LB 0.9346\nSimple CV: 0.94818 LB 0.9511
\n\n?
", "votes": 0, "replies": [] }, { "id": 965691, "author_name": "Mobassir", "author_url": "", "post_date": "2020-08-10T19:10:50.470000", "content": "@philippsinger \nSimple Average Cv : 0.9507 LB 0.9511\nwith MinMax Cv : 0.94818 LB 0.9346
\n\nsimple average maybe better
", "votes": 0, "replies": [] }, { "id": 965697, "author_name": "Mobassir", "author_url": "", "post_date": "2020-08-10T19:23:51.527000", "content": "@philippsinger another simple average result : \ncv 0.951118782197574\nlb 0.9502
", "votes": 0, "replies": [] }, { "id": 968489, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-13T04:10:57.853000", "content": "Great job beating CV 0.950 @mobassir
", "votes": 1, "replies": [] }, { "id": 968946, "author_name": "", "author_url": "", "post_date": "2020-08-13T11:09:42.320000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 962795, "author_name": "KhanhVD", "author_url": "", "post_date": "2020-08-08T12:52:52.847000", "content": "Do you use external data with your ensemble 0.947 CV?
", "votes": 2, "replies": [ { "id": 962818, "author_name": "Seifeddine Fezzani", "author_url": "", "post_date": "2020-08-08T13:04:25.493000", "content": "Yes, I am using external data provided here and I only use images of this competition during validation.
", "votes": 1, "replies": [] } ] }, { "id": 970763, "author_name": "isaac in", "author_url": "", "post_date": "2020-08-14T18:47:55.513000", "content": "My best oof cv is 9669 using the methods in this notebook(https://www.kaggle.com/steubk/simple-oof-ensembling-methods-for-classification), with the lb score being 9453 so im not sure if i should trust it. My lb score currently is an ensemble of oof scores 91-94 with most of them being on the lower end
", "votes": 0, "replies": [] }, { "id": 965264, "author_name": "Viraj Bagal", "author_url": "", "post_date": "2020-08-10T13:59:37.650000", "content": "I have one question to all of you, is your CV resistant/robust to seed change? I mean have you guys tried keeping the same setup but just changing the seed and what is ur CV after that?
", "votes": 0, "replies": [ { "id": 965269, "author_name": "Seifeddine Fezzani", "author_url": "", "post_date": "2020-08-10T14:06:19.377000", "content": "If you change the seed, results will change a bit. Fortunately with ensembling we can reduce this variance.
", "votes": 0, "replies": [ { "id": 965275, "author_name": "Viraj Bagal", "author_url": "", "post_date": "2020-08-10T14:13:11.723000", "content": "Ya, I understand that. But can u roughly say what is the difference between ur best CV and the seed changed CV?
", "votes": 1, "replies": [] }, { "id": 967590, "author_name": "Seifeddine Fezzani", "author_url": "", "post_date": "2020-08-12T11:12:31.630000", "content": "I tried to retrain some base models with a different seed and the CV score of each model varies between +- 0.002. This difference must be less when I ensemble but I will need to retrain everything : ) No time for that
", "votes": 0, "replies": [] }, { "id": 967611, "author_name": "Viraj Bagal", "author_url": "", "post_date": "2020-08-12T11:32:02.513000", "content": "Given that ur cv fluctuation is +-0.002, I think your models are quite robust with respect to others' models I am reading in the discussion. Nice work!
", "votes": 0, "replies": [] } ] } ] }, { "id": 963572, "author_name": "kaggler", "author_url": "", "post_date": "2020-08-09T06:13:09.607000", "content": "My Best CV 9614, LB : 9510\nSecond CV 9530 LB 9557\nWhich one should I trust? I have no idea!
", "votes": 0, "replies": [ { "id": 963663, "author_name": "Seifeddine Fezzani", "author_url": "", "post_date": "2020-08-09T07:44:08.047000", "content": "The one with the highest cv if you believe your validation strategy is the right one
", "votes": 1, "replies": [] }, { "id": 964067, "author_name": "kaggler", "author_url": "", "post_date": "2020-08-09T15:07:23.773000", "content": "What I am afraid of is that the highest cv is also overfitted to training sets since I guess the training sets and the test sets have different distributions each other\nI have been using chris's stratified data
", "votes": 0, "replies": [] }, { "id": 965444, "author_name": "Seifeddine Fezzani", "author_url": "", "post_date": "2020-08-10T16:31:17.300000", "content": "Be careful when you add external data :)
", "votes": 1, "replies": [] }, { "id": 968487, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-13T04:10:17.913000", "content": "Great job @deepkim , CV over 0.960 is fantastic!
", "votes": 1, "replies": [] } ] }, { "id": 968915, "author_name": "", "author_url": "", "post_date": "2020-08-13T10:41:57.073000", "content": "", "votes": 0, "replies": [ { "id": 968998, "author_name": "Optimo", "author_url": "", "post_date": "2020-08-13T11:53:05.363000", "content": "Why donβt you look at your CV using metadata?
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "962789": "Hello everyone, the competition will almost end and it is time to start thinking about adding diversity and ensembling our base models them efficiently.\n\nI am doing that by maximizing the OOF to determine what are the best weights. Also, my validation strategy is Triple Stratified KFold shared [here](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords/output) with K == 3. \n\nI want to have an idea about your best CV scores with some details like (blending strategy, validation strategy) since the leaderboard isn't reliable at all in this competition. \n\nFor me my final ensemble gives me 0.947 CV with weighted averaging => 0.951 LB. I am working at improving this score.", "968485": "It's fascinating that many posts here have similar CV 0.950 but the LBs vary. I wonder where everyone will end on private LB. My ensemble CV is over 0.950 but my ensemble LB struggles to beat public LB ensembles. I'm curious to see if optimized CV ensembles beat public notebook LB ensembles.", "968773": "ensemble cv 0.946~0.953, lb 0.950~0.955, We analyzing about 250 images of the top probabilities, and looking for the strategy for how to submit them. \n\nTrust CV or Trust LB?", "967719": "Mine is CV 0.9477 LB 0.9530 validate only on isic2020", "965892": "Nice one !! Keep sharing", "967891": "CV .950, LB .945. No external data, no post processing. I ensemble a few EfficientNet and DenseNet models.", "965769": "Ensemble oof cv of 0.9517 only validating on 2020 data\nLB 0.9549", "962841": "our ensemble strategy is also very similar :)\nwe have got CV 0.950461638257698 \nand LB 0.9505 (for minmax)\nsince the public LB of this competition is of no use,we are still unsure about the upper and lower bound threshold for final submission\n\nanother submission where we took simple average had : \nCV = 0.9296710190571207\nLb = 0.9511\none question,if you do not mind - \"is your current best LB submission a blend of blends or nested blend? :)\"", "962795": "Do you use external data with your ensemble 0.947 CV?", "970763": "My best oof cv is 9669 using the methods in this notebook(https://www.kaggle.com/steubk/simple-oof-ensembling-methods-for-classification), with the lb score being 9453 so im not sure if i should trust it. My lb score currently is an ensemble of oof scores 91-94 with most of them being on the lower end", "965264": "I have one question to all of you, is your CV resistant/robust to seed change? I mean have you guys tried keeping the same setup but just changing the seed and what is ur CV after that?", "963572": "My Best CV 9614, LB : 9510\nSecond CV 9530 LB 9557\nWhich one should I trust? I have no idea!\n", "968915": "" } }