After two weeks struggling with training a good single model, I give up at .663\nThere are someone can achieve more than .667, cannot wait seeing their solution after it ends.\nNow it's time to ensemble, personally I have three networks for ensemble, two are still training different folds, till now two networks are >.660, one is .657, first ensemble result in .668.
\n\nHow about networks in your ensemble, or other kinds, like different image sizes?
", "messages": [ { "id": "673725", "postDate": "11/15/2019 12:01:41", "content": "After two weeks struggling with training a good single model, I give up at .663\nThere are someone can achieve more than .667, cannot wait seeing their solution after it ends.\nNow it's time to ensemble, personally I have three networks for ensemble, two are still training different folds, till now two networks are >.660, one is .657, first ensemble result in .668.
\n\nHow about networks in your ensemble, or other kinds, like different image sizes?
", "rawMarkdown": "After two weeks struggling with training a good single model, I give up at .663\nThere are someone can achieve more than .667, cannot wait seeing their solution after it ends.\nNow it's time to ensemble, personally I have three networks for ensemble, two are still training different folds, till now two networks are >.660, one is .657, first ensemble result in .668.\n\nHow about networks in your ensemble, or other kinds, like different image sizes?", "votes": null }, { "id": "673789", "postDate": "11/15/2019 13:39:39", "content": "My single model best only got 0.659... I gave up much earlier.. about one week ago. My current best LB was getting by ensemble 4 models with classifier. 1->0.659, 2 ->0.660, 3->0.664, 4->0.668\nBut it seems to reach to the upper bound, since all of my single model just not that good. π
", "rawMarkdown": "My single model best only got 0.659... I gave up much earlier.. about one week ago. My current best LB was getting by ensemble 4 models with classifier. 1->0.659, 2 ->0.660, 3->0.664, 4->0.668\nBut it seems to reach to the upper bound, since all of my single model just not that good. π", "votes": null }, { "id": "673808", "postDate": "11/15/2019 14:12:47", "content": "try to go beyond fpn, unet, deeplab. there are many new architecture like jpu, pspnet, texture encoding, object context encoding.
\n\ni currently about 8 network of different segmentation head, loss and input size to get LB 0.672. each network is in the range of 0.65 to 0.66
", "rawMarkdown": "try to go beyond fpn, unet, deeplab. there are many new architecture like jpu, pspnet, texture encoding, object context encoding. \n\ni currently about 8 network of different segmentation head, loss and input size to get LB 0.672. each network is in the range of 0.65 to 0.66", "votes": null }, { "id": "673823", "postDate": "11/15/2019 14:26:22", "content": "It seems most of the people will ensemble at least 3 models. Does this means the shake up will be less possible?
", "rawMarkdown": "It seems most of the people will ensemble at least 3 models. Does this means the shake up will be less possible?", "votes": null }, { "id": "673825", "postDate": "11/15/2019 14:30:00", "content": "yes. ensemble always give better results (public and private).
\n\nresnet34 seems sufficient. it takes about 1 to 2 hr to train one segmentation model. \nso i think most people will have more than 3 models.
", "rawMarkdown": "yes. ensemble always give better results (public and private).\n\nresnet34 seems sufficient. it takes about 1 to 2 hr to train one segmentation model. \nso i think most people will have more than 3 models.", "votes": null }, { "id": "673836", "postDate": "11/15/2019 14:41:34", "content": "True, my best single model was trained with seresnet18 backbone.
", "rawMarkdown": "True, my best single model was trained with seresnet18 backbone.", "votes": null }, { "id": "673838", "postDate": "11/15/2019 14:42:37", "content": "Thanks Heng, I'll definitely try.
", "rawMarkdown": "Thanks Heng, I'll definitely try.", "votes": null }, { "id": "673841", "postDate": "11/15/2019 14:44:16", "content": "That's OK, it's wise decision for this competition.
", "rawMarkdown": "That's OK, it's wise decision for this competition.", "votes": null }, { "id": "673871", "postDate": "11/15/2019 15:29:07", "content": "@xiejialun haha, my best is deeplabv3+, which was always my best performance network from previous competitions...
", "rawMarkdown": "xiejialun haha, my best is deeplabv3+, which was always my best performance network from previous competitions...", "votes": null }, { "id": "673981", "postDate": "11/15/2019 18:28:13", "content": "Hey @xiejialun how are you ensembling your models? As for me LB is not consistent with different ensembles
", "rawMarkdown": "Hey @xiejialun how are you ensembling your models? As for me LB is not consistent with different ensembles", "votes": null }, { "id": "674126", "postDate": "11/15/2019 23:09:07", "content": "@ubamba98 I'm using average the raw prediction for ensemble. I did train these model with slightly different process/loss on purpose to make them can help each other's performance. I don't sure this will help or I just get luckyπ. But the number of model is quite consistent with LB.
", "rawMarkdown": "ubamba98 I'm using average the raw prediction for ensemble. I did train these model with slightly different process/loss on purpose to make them can help each other's performance. I don't sure this will help or I just get luckyπ. But the number of model is quite consistent with LB.", "votes": null }, { "id": "674193", "postDate": "11/16/2019 01:48:37", "content": "Thanks I will try that in future.
", "rawMarkdown": "Thanks I will try that in future.", "votes": null }, { "id": "674199", "postDate": "11/16/2019 01:54:17", "content": "I'm also struggling to find a good model for ensemble. π
", "rawMarkdown": "I'm also struggling to find a good model for ensemble. π", "votes": null }, { "id": "674202", "postDate": "11/16/2019 01:59:46", "content": "it is difficult. sometimes we use stacking or learned another level-2 network. \nyou can also see this: https://ai.googleblog.com/2018/10/introducing-adanet-fast-and-flexible.html
\n\n\"Building an ensemble of neural networks has several challenges: What are the best subnetwork architectures to consider? Is it best to reuse the same architectures or encourage diversity? While complex subnetworks with more parameters will tend to perform better on the training set, they may not generalize to unseen data due to their greater complexity. \"
", "rawMarkdown": "it is difficult. sometimes we use stacking or learned another level-2 network. \nyou can also see this: https://ai.googleblog.com/2018/10/introducing-adanet-fast-and-flexible.html\n\n\"Building an ensemble of neural networks has several challenges: What are the best subnetwork architectures to consider? Is it best to reuse the same architectures or encourage diversity? While complex subnetworks with more parameters will tend to perform better on the training set, they may not generalize to unseen data due to their greater complexity. \"", "votes": null }, { "id": "674211", "postDate": "11/16/2019 02:37:09", "content": "Single fpn 6640 + single unet 6612 only got 6647 : (
", "rawMarkdown": "Single fpn 6640 + single unet 6612 only got 6647 : (", "votes": null }, { "id": "674212", "postDate": "11/16/2019 02:37:59", "content": "You're right, it's difficult.\nSo I haven't improved my LB score in days. π
", "rawMarkdown": "You're right, it's difficult.\nSo I haven't improved my LB score in days. π", "votes": null }, { "id": "674224", "postDate": "11/16/2019 03:09:12", "content": "Sometimes itβs fun to see the 1st place make a joke π π
", "rawMarkdown": "Sometimes itβs fun to see the 1st place make a joke π π", "votes": null }, { "id": "674227", "postDate": "11/16/2019 03:34:27", "content": "\"Single fpn 6640 + single unet 6612 only got 6647 : (\"
\n\nmy suggestion is:
\n\nbreakdown the results of single model and ensemble, e.g.\nsingle model false positive =xx, ensemble model false positive =yy
if yy is better than xx, than we know that the effects of ensemble is reduction of fp.
then we can think of ways to further improve this effect or to address problems that are solved by ensemble
\"So I haven't improved my LB score in days\"
\n\nlesson from steel competition is that preventing shakeup is also important
", "rawMarkdown": "\"So I haven't improved my LB score in days\"\n\nlesson from steel competition is that preventing shakeup is also important", "votes": null }, { "id": "674236", "postDate": "11/16/2019 03:59:29", "content": "Yes... It's a sad experience for us.\nAnd when I found that I got a score [private: 0.90828, public: 0.92286], when I just set threshold to [0.7, 0.7, 0.7, 0.7], I cried... πΏ
\n\nSo I will only use high thresholds in this competition...
", "rawMarkdown": "Yes... It's a sad experience for us.\nAnd when I found that I got a score [private: 0.90828, public: 0.92286], when I just set threshold to [0.7, 0.7, 0.7, 0.7], I cried... πΏ \n\nSo I will only use high thresholds in this competition...", "votes": null }, { "id": "674243", "postDate": "11/16/2019 04:16:16", "content": "my experiment shows higher threshold works better but we still have to be cautious.
\n\ni haven't think of a way yet but we should think of a way to use both the public and private data to confirm. (unlike the steel competition, this time we get to see the private data set as well)
\n\nsome possible solutions\n1. visual inspection of test data\n2. measure the similarity of test data \n3. think of a way to verify label consistency in test data. e.g. we make prediction on test. then we use pseudo label only to train and verify on train set. then we apply mixup augmentation pseudo label and see the effects on error of the same train set, etc ...
\n\nsince we are only interested in image level label for fp rejection, we have a lot of options actually. it is a problem of verifying (or robustness, uncertainty estimation) the classification (not segmentation) of test set
", "rawMarkdown": "my experiment shows higher threshold works better but we still have to be cautious.\n\ni haven't think of a way yet but we should think of a way to use both the public and private data to confirm. (unlike the steel competition, this time we get to see the private data set as well)\n\nsome possible solutions\n1. visual inspection of test data\n2. measure the similarity of test data \n3. think of a way to verify label consistency in test data. e.g. we make prediction on test. then we use pseudo label only to train and verify on train set. then we apply mixup augmentation pseudo label and see the effects on error of the same train set, etc ...\n\nsince we are only interested in image level label for fp rejection, we have a lot of options actually. it is a problem of verifying (or robustness, uncertainty estimation) the classification (not segmentation) of test set", "votes": null }, { "id": "674260", "postDate": "11/16/2019 05:33:37", "content": "Thanks for your suggestions.\nHeng, I respect your sharing and I'm always rooting for you.
", "rawMarkdown": "Thanks for your suggestions.\nHeng, I respect your sharing and I'm always rooting for you.", "votes": null }, { "id": "674448", "postDate": "11/16/2019 14:24:32", "content": "single image size, 8x folds Unet, 3 more folds Unet with different encoders. Contribution of the different encoder is negligble I think.
", "rawMarkdown": "single image size, 8x folds Unet, 3 more folds Unet with different encoders. Contribution of the different encoder is negligble I think.", "votes": null }, { "id": "674452", "postDate": "11/16/2019 14:33:40", "content": "I can only gain very small benefit on multi-fold training, but different encoder did improve my score. This is quite weird haha.
\n\nBtw, can't wait to see your solution! single model with 0.667..that's insane!
", "rawMarkdown": "I can only gain very small benefit on multi-fold training, but different encoder did improve my score. This is quite weird haha.\n\nBtw, can't wait to see your solution! single model with 0.667..that's insane!", "votes": null }, { "id": "674518", "postDate": "11/16/2019 16:14:18", "content": "another is diversity in loss, e.g
\n\n```
\n\nclass Net(nn.Module):\n ....\n def forward(self, x):\n feature = self.block(x)\n ....\n .....\n label_probability1 = self.one_way_to_predict( ....)\n ....\n .....\n label_probability2 = self.another_way_to_predict( .... different feature used ....)\n .....\n label_probability3 = self.yet_another_way_to_predict( .... yet different feature used ....)
\n\n```
\n\nfor inference, average the 3 label prediction (since image label is important, we only create multi classifiers).
\n\nthis is like using shared features to learn 3 classifiers. and then average them like ensemble.
\n\nfor back propagation, use:
\n\n```\nloss = a*loss(label_probability1 ) + b*loss(label_probability2 ) + c*loss(label_probability3 ) + ...
\n\na,b,c are random variable with certain range e.g. a+b+c = 1 and a in range 0.3333 +/- noise, etc ....\n```
\n\nwe inject the weighing loss noise to make it more robust
", "rawMarkdown": "another is diversity in loss, e.g\n\n```\n\nclass Net(nn.Module):\n ....\n def forward(self, x):\n feature = self.block(x)\n ....\n .....\n label_probability1 = self.one_way_to_predict( ....)\n ....\n .....\n label_probability2 = self.another_way_to_predict( .... different feature used ....)\n .....\n label_probability3 = self.yet_another_way_to_predict( .... yet different feature used ....)\n\n\n```\n\nfor inference, average the 3 label prediction (since image label is important, we only create multi classifiers).\n\nthis is like using shared features to learn 3 classifiers. and then average them like ensemble.\n\nfor back propagation, use:\n\n```\nloss = a*loss(label_probability1 ) + b*loss(label_probability2 ) + c*loss(label_probability3 ) + ...\n\na,b,c are random variable with certain range e.g. a+b+c = 1 and a in range 0.3333 +/- noise, etc ....\n```\n\nwe inject the weighing loss noise to make it more robust", "votes": null }, { "id": "674780", "postDate": "11/17/2019 04:15:00", "content": "Got .6671 network, now I suspect some kaggler got even higher single model performance...
", "rawMarkdown": "Got .6671 network, now I suspect some kaggler got even higher single model performance...", "votes": null }, { "id": "674790", "postDate": "11/17/2019 04:41:52", "content": "definitely yes.
\n\nthe ensemble results just prove that there is that much discriminative information that can extracted from the train set.
\n\nit is whether we can design good learning method to extract the information for a single model
\n\nif you read through the papers of imagenet, you will find that the ensemble of many vgg-16 is close to the performance of single resnet50
", "rawMarkdown": "definitely yes.\n\nthe ensemble results just prove that there is that much discriminative information that can extracted from the train set.\n\nit is whether we can design good learning method to extract the information for a single model\n\nif you read through the papers of imagenet, you will find that the ensemble of many vgg-16 is close to the performance of single resnet50", "votes": null }, { "id": "674842", "postDate": "11/17/2019 06:24:25", "content": "single fpn with or without classifier?
", "rawMarkdown": "single fpn with or without classifier?", "votes": null }, { "id": "674869", "postDate": "11/17/2019 07:10:03", "content": "@niuddd amazing work! is it K-fold or single fold?
", "rawMarkdown": "niuddd amazing work! is it K-fold or single fold?", "votes": null }, { "id": "675374", "postDate": "11/18/2019 02:19:08", "content": "I got 0.6686 from a single network with 4 folds. However, looks like it doesn't matter; at least for the Public LB. Because while ensembling, the result becomes funnier. For example, the ensemble of lower scored networks giving higher public LB compared to the ensemble with the score of the best-scored network. I am not sure what is causing that.
", "rawMarkdown": "I got 0.6686 from a single network with 4 folds. However, looks like it doesn't matter; at least for the Public LB. Because while ensembling, the result becomes funnier. For example, the ensemble of lower scored networks giving higher public LB compared to the ensemble with the score of the best-scored network. I am not sure what is causing that.", "votes": null }, { "id": "675375", "postDate": "11/18/2019 02:21:57", "content": "@mykttu true I am also experiencing the same effect I don't think private LB is a good generalization.
", "rawMarkdown": "mykttu true I am also experiencing the same effect I don't think private LB is a good generalization.", "votes": null }, { "id": "675415", "postDate": "11/18/2019 03:53:38", "content": "I made a mistake, 6640 model is unet resnet34. So it is a reasonable ensemble score.
", "rawMarkdown": "I made a mistake, 6640 model is unet resnet34. So it is a reasonable ensemble score.", "votes": null }, { "id": "675442", "postDate": "11/18/2019 05:00:35", "content": "True. Going beyond .670, I realized there's sub-competition of tuning ensembles.
", "rawMarkdown": "True. Going beyond .670, I realized there's sub-competition of tuning ensembles.", "votes": null }, { "id": "675446", "postDate": "11/18/2019 05:08:23", "content": "Another question is would you select two best scoring (public LB) submissions, or would you prefer two that have most confidence with? Personally, I would choose at least one to be a big ensemble (of as many models as I have), which statistically has lower variance (though lower LB) to fight against the possible shake up. What about you?
", "rawMarkdown": "Another question is would you select two best scoring (public LB) submissions, or would you prefer two that have most confidence with? Personally, I would choose at least one to be a big ensemble (of as many models as I have), which statistically has lower variance (though lower LB) to fight against the possible shake up. What about you?", "votes": null }, { "id": "675455", "postDate": "11/18/2019 05:14:42", "content": "I think submission should be selected such that you can minimize the false positives. I will select my submission based on it not the lb score.
", "rawMarkdown": "I think submission should be selected such that you can minimize the false positives. I will select my submission based on it not the lb score.", "votes": null }, { "id": "675463", "postDate": "11/18/2019 05:23:41", "content": "Same as you, one with best public LB, one with big ensemble.\nGood luck
", "rawMarkdown": "Same as you, one with best public LB, one with big ensemble.\nGood luck", "votes": null }, { "id": "675482", "postDate": "11/18/2019 05:49:19", "content": "5 different input size model with same head and decoder got me from 0.65 to 0.666
", "rawMarkdown": "5 different input size model with same head and decoder got me from 0.65 to 0.666", "votes": null }, { "id": "675544", "postDate": "11/18/2019 07:53:00", "content": "5 different architectures each 5 fold.\nEnsemble Strategy in 5 folds -> sigmoid(mean(probabilities))\nEnsemble Strategy to merge binary masks -> max voting\nBest single architecture performance -> 0.6658\nLB score -> 0.6674
", "rawMarkdown": "5 different architectures each 5 fold.\nEnsemble Strategy in 5 folds -> sigmoid(mean(probabilities))\nEnsemble Strategy to merge binary masks -> max voting\nBest single architecture performance -> 0.6658\nLB score -> 0.6674", "votes": null }, { "id": "675551", "postDate": "11/18/2019 08:05:17", "content": "What is the voting threshold did you use I am experiencing a huge difference in LB score with different thresh in voting. Is that the case with you too?
", "rawMarkdown": "What is the voting threshold did you use I am experiencing a huge difference in LB score with different thresh in voting. Is that the case with you too?", "votes": null }, { "id": "675587", "postDate": "11/18/2019 09:12:08", "content": "I have experience such situation. Shakeup is around 0.001. Currently I found out higher theshold has better LB score.
", "rawMarkdown": "I have experience such situation. Shakeup is around 0.001. Currently I found out higher theshold has better LB score.", "votes": null }, { "id": "675594", "postDate": "11/18/2019 09:26:45", "content": "Ya same. Its really hard to select final submission.
", "rawMarkdown": "Ya same. Its really hard to select final submission.", "votes": null } ], "comments": [ { "id": 673789, "author_name": "xiejialun", "author_url": "", "post_date": "11/15/2019 13:39:39", "content": "My single model best only got 0.659... I gave up much earlier.. about one week ago. My current best LB was getting by ensemble 4 models with classifier. 1->0.659, 2 ->0.660, 3->0.664, 4->0.668\nBut it seems to reach to the upper bound, since all of my single model just not that good. π
", "votes": null, "replies": [ { "id": 673841, "author_name": "niuddd", "author_url": "", "post_date": "11/15/2019 14:44:16", "content": "That's OK, it's wise decision for this competition.
", "votes": null, "replies": [] }, { "id": 673981, "author_name": "ubamba98", "author_url": "", "post_date": "11/15/2019 18:28:13", "content": "Hey @xiejialun how are you ensembling your models? As for me LB is not consistent with different ensembles
", "votes": null, "replies": [] }, { "id": 674126, "author_name": "xiejialun", "author_url": "", "post_date": "11/15/2019 23:09:07", "content": "@ubamba98 I'm using average the raw prediction for ensemble. I did train these model with slightly different process/loss on purpose to make them can help each other's performance. I don't sure this will help or I just get luckyπ. But the number of model is quite consistent with LB.
", "votes": null, "replies": [] }, { "id": 674193, "author_name": "ubamba98", "author_url": "", "post_date": "11/16/2019 01:48:37", "content": "Thanks I will try that in future.
", "votes": null, "replies": [] } ] }, { "id": 673808, "author_name": "hengck23", "author_url": "", "post_date": "11/15/2019 14:12:47", "content": "try to go beyond fpn, unet, deeplab. there are many new architecture like jpu, pspnet, texture encoding, object context encoding.
\n\ni currently about 8 network of different segmentation head, loss and input size to get LB 0.672. each network is in the range of 0.65 to 0.66
", "votes": null, "replies": [ { "id": 673823, "author_name": "xiejialun", "author_url": "", "post_date": "11/15/2019 14:26:22", "content": "It seems most of the people will ensemble at least 3 models. Does this means the shake up will be less possible?
", "votes": null, "replies": [] }, { "id": 673825, "author_name": "hengck23", "author_url": "", "post_date": "11/15/2019 14:30:00", "content": "yes. ensemble always give better results (public and private).
\n\nresnet34 seems sufficient. it takes about 1 to 2 hr to train one segmentation model. \nso i think most people will have more than 3 models.
", "votes": null, "replies": [] }, { "id": 673836, "author_name": "xiejialun", "author_url": "", "post_date": "11/15/2019 14:41:34", "content": "True, my best single model was trained with seresnet18 backbone.
", "votes": null, "replies": [] }, { "id": 673838, "author_name": "niuddd", "author_url": "", "post_date": "11/15/2019 14:42:37", "content": "Thanks Heng, I'll definitely try.
", "votes": null, "replies": [] }, { "id": 673871, "author_name": "niuddd", "author_url": "", "post_date": "11/15/2019 15:29:07", "content": "@xiejialun haha, my best is deeplabv3+, which was always my best performance network from previous competitions...
", "votes": null, "replies": [] } ] }, { "id": 674199, "author_name": "limerobot", "author_url": "", "post_date": "11/16/2019 01:54:17", "content": "I'm also struggling to find a good model for ensemble. π
", "votes": null, "replies": [ { "id": 674202, "author_name": "hengck23", "author_url": "", "post_date": "11/16/2019 01:59:46", "content": "it is difficult. sometimes we use stacking or learned another level-2 network. \nyou can also see this: https://ai.googleblog.com/2018/10/introducing-adanet-fast-and-flexible.html
\n\n\"Building an ensemble of neural networks has several challenges: What are the best subnetwork architectures to consider? Is it best to reuse the same architectures or encourage diversity? While complex subnetworks with more parameters will tend to perform better on the training set, they may not generalize to unseen data due to their greater complexity. \"
", "votes": null, "replies": [] }, { "id": 674212, "author_name": "limerobot", "author_url": "", "post_date": "11/16/2019 02:37:59", "content": "You're right, it's difficult.\nSo I haven't improved my LB score in days. π
", "votes": null, "replies": [] }, { "id": 674224, "author_name": "ratthachat", "author_url": "", "post_date": "11/16/2019 03:09:12", "content": "Sometimes itβs fun to see the 1st place make a joke π π
", "votes": null, "replies": [] }, { "id": 674228, "author_name": "hengck23", "author_url": "", "post_date": "11/16/2019 03:35:21", "content": "\"So I haven't improved my LB score in days\"
\n\nlesson from steel competition is that preventing shakeup is also important
", "votes": null, "replies": [] }, { "id": 674236, "author_name": "limerobot", "author_url": "", "post_date": "11/16/2019 03:59:29", "content": "Yes... It's a sad experience for us.\nAnd when I found that I got a score [private: 0.90828, public: 0.92286], when I just set threshold to [0.7, 0.7, 0.7, 0.7], I cried... πΏ
\n\nSo I will only use high thresholds in this competition...
", "votes": null, "replies": [] }, { "id": 674243, "author_name": "hengck23", "author_url": "", "post_date": "11/16/2019 04:16:16", "content": "my experiment shows higher threshold works better but we still have to be cautious.
\n\ni haven't think of a way yet but we should think of a way to use both the public and private data to confirm. (unlike the steel competition, this time we get to see the private data set as well)
\n\nsome possible solutions\n1. visual inspection of test data\n2. measure the similarity of test data \n3. think of a way to verify label consistency in test data. e.g. we make prediction on test. then we use pseudo label only to train and verify on train set. then we apply mixup augmentation pseudo label and see the effects on error of the same train set, etc ...
\n\nsince we are only interested in image level label for fp rejection, we have a lot of options actually. it is a problem of verifying (or robustness, uncertainty estimation) the classification (not segmentation) of test set
", "votes": null, "replies": [] }, { "id": 674260, "author_name": "limerobot", "author_url": "", "post_date": "11/16/2019 05:33:37", "content": "Thanks for your suggestions.\nHeng, I respect your sharing and I'm always rooting for you.
", "votes": null, "replies": [] } ] }, { "id": 674211, "author_name": "qiaoshiji", "author_url": "", "post_date": "11/16/2019 02:37:09", "content": "Single fpn 6640 + single unet 6612 only got 6647 : (
", "votes": null, "replies": [ { "id": 674227, "author_name": "hengck23", "author_url": "", "post_date": "11/16/2019 03:34:27", "content": "\"Single fpn 6640 + single unet 6612 only got 6647 : (\"
\n\nmy suggestion is:
\n\nbreakdown the results of single model and ensemble, e.g.\nsingle model false positive =xx, ensemble model false positive =yy
if yy is better than xx, than we know that the effects of ensemble is reduction of fp.
then we can think of ways to further improve this effect or to address problems that are solved by ensemble
single fpn with or without classifier?
", "votes": null, "replies": [] }, { "id": 675415, "author_name": "qiaoshiji", "author_url": "", "post_date": "11/18/2019 03:53:38", "content": "I made a mistake, 6640 model is unet resnet34. So it is a reasonable ensemble score.
", "votes": null, "replies": [] } ] }, { "id": 674448, "author_name": "tugstugi", "author_url": "", "post_date": "11/16/2019 14:24:32", "content": "single image size, 8x folds Unet, 3 more folds Unet with different encoders. Contribution of the different encoder is negligble I think.
", "votes": null, "replies": [ { "id": 674452, "author_name": "xiejialun", "author_url": "", "post_date": "11/16/2019 14:33:40", "content": "I can only gain very small benefit on multi-fold training, but different encoder did improve my score. This is quite weird haha.
\n\nBtw, can't wait to see your solution! single model with 0.667..that's insane!
", "votes": null, "replies": [] }, { "id": 674518, "author_name": "hengck23", "author_url": "", "post_date": "11/16/2019 16:14:18", "content": "another is diversity in loss, e.g
\n\n```
\n\nclass Net(nn.Module):\n ....\n def forward(self, x):\n feature = self.block(x)\n ....\n .....\n label_probability1 = self.one_way_to_predict( ....)\n ....\n .....\n label_probability2 = self.another_way_to_predict( .... different feature used ....)\n .....\n label_probability3 = self.yet_another_way_to_predict( .... yet different feature used ....)
\n\n```
\n\nfor inference, average the 3 label prediction (since image label is important, we only create multi classifiers).
\n\nthis is like using shared features to learn 3 classifiers. and then average them like ensemble.
\n\nfor back propagation, use:
\n\n```\nloss = a*loss(label_probability1 ) + b*loss(label_probability2 ) + c*loss(label_probability3 ) + ...
\n\na,b,c are random variable with certain range e.g. a+b+c = 1 and a in range 0.3333 +/- noise, etc ....\n```
\n\nwe inject the weighing loss noise to make it more robust
", "votes": null, "replies": [] } ] }, { "id": 674780, "author_name": "niuddd", "author_url": "", "post_date": "11/17/2019 04:15:00", "content": "Got .6671 network, now I suspect some kaggler got even higher single model performance...
", "votes": null, "replies": [ { "id": 674790, "author_name": "hengck23", "author_url": "", "post_date": "11/17/2019 04:41:52", "content": "definitely yes.
\n\nthe ensemble results just prove that there is that much discriminative information that can extracted from the train set.
\n\nit is whether we can design good learning method to extract the information for a single model
\n\nif you read through the papers of imagenet, you will find that the ensemble of many vgg-16 is close to the performance of single resnet50
", "votes": null, "replies": [] }, { "id": 674869, "author_name": "adish333", "author_url": "", "post_date": "11/17/2019 07:10:03", "content": "@niuddd amazing work! is it K-fold or single fold?
", "votes": null, "replies": [] }, { "id": 675374, "author_name": "mykttu", "author_url": "", "post_date": "11/18/2019 02:19:08", "content": "I got 0.6686 from a single network with 4 folds. However, looks like it doesn't matter; at least for the Public LB. Because while ensembling, the result becomes funnier. For example, the ensemble of lower scored networks giving higher public LB compared to the ensemble with the score of the best-scored network. I am not sure what is causing that.
", "votes": null, "replies": [] }, { "id": 675375, "author_name": "ubamba98", "author_url": "", "post_date": "11/18/2019 02:21:57", "content": "@mykttu true I am also experiencing the same effect I don't think private LB is a good generalization.
", "votes": null, "replies": [] }, { "id": 675442, "author_name": "niuddd", "author_url": "", "post_date": "11/18/2019 05:00:35", "content": "True. Going beyond .670, I realized there's sub-competition of tuning ensembles.
", "votes": null, "replies": [] } ] }, { "id": 675446, "author_name": "niuddd", "author_url": "", "post_date": "11/18/2019 05:08:23", "content": "Another question is would you select two best scoring (public LB) submissions, or would you prefer two that have most confidence with? Personally, I would choose at least one to be a big ensemble (of as many models as I have), which statistically has lower variance (though lower LB) to fight against the possible shake up. What about you?
", "votes": null, "replies": [ { "id": 675455, "author_name": "ubamba98", "author_url": "", "post_date": "11/18/2019 05:14:42", "content": "I think submission should be selected such that you can minimize the false positives. I will select my submission based on it not the lb score.
", "votes": null, "replies": [] }, { "id": 675463, "author_name": "xiejialun", "author_url": "", "post_date": "11/18/2019 05:23:41", "content": "Same as you, one with best public LB, one with big ensemble.\nGood luck
", "votes": null, "replies": [] } ] }, { "id": 675482, "author_name": "markson14", "author_url": "", "post_date": "11/18/2019 05:49:19", "content": "5 different input size model with same head and decoder got me from 0.65 to 0.666
", "votes": null, "replies": [] }, { "id": 675544, "author_name": "adish333", "author_url": "", "post_date": "11/18/2019 07:53:00", "content": "5 different architectures each 5 fold.\nEnsemble Strategy in 5 folds -> sigmoid(mean(probabilities))\nEnsemble Strategy to merge binary masks -> max voting\nBest single architecture performance -> 0.6658\nLB score -> 0.6674
", "votes": null, "replies": [ { "id": 675551, "author_name": "ubamba98", "author_url": "", "post_date": "11/18/2019 08:05:17", "content": "What is the voting threshold did you use I am experiencing a huge difference in LB score with different thresh in voting. Is that the case with you too?
", "votes": null, "replies": [] }, { "id": 675587, "author_name": "markson14", "author_url": "", "post_date": "11/18/2019 09:12:08", "content": "I have experience such situation. Shakeup is around 0.001. Currently I found out higher theshold has better LB score.
", "votes": null, "replies": [] }, { "id": 675594, "author_name": "ubamba98", "author_url": "", "post_date": "11/18/2019 09:26:45", "content": "Ya same. Its really hard to select final submission.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "673725": "After two weeks struggling with training a good single model, I give up at .663\nThere are someone can achieve more than .667, cannot wait seeing their solution after it ends.\nNow it's time to ensemble, personally I have three networks for ensemble, two are still training different folds, till now two networks are >.660, one is .657, first ensemble result in .668.\n\nHow about networks in your ensemble, or other kinds, like different image sizes?", "673789": "My single model best only got 0.659... I gave up much earlier.. about one week ago. My current best LB was getting by ensemble 4 models with classifier. 1->0.659, 2 ->0.660, 3->0.664, 4->0.668\nBut it seems to reach to the upper bound, since all of my single model just not that good. π", "673808": "try to go beyond fpn, unet, deeplab. there are many new architecture like jpu, pspnet, texture encoding, object context encoding. \n\ni currently about 8 network of different segmentation head, loss and input size to get LB 0.672. each network is in the range of 0.65 to 0.66", "673823": "It seems most of the people will ensemble at least 3 models. Does this means the shake up will be less possible?", "673825": "yes. ensemble always give better results (public and private).\n\nresnet34 seems sufficient. it takes about 1 to 2 hr to train one segmentation model. \nso i think most people will have more than 3 models.", "673836": "True, my best single model was trained with seresnet18 backbone.", "673838": "Thanks Heng, I'll definitely try.", "673841": "That's OK, it's wise decision for this competition.", "673871": "xiejialun haha, my best is deeplabv3+, which was always my best performance network from previous competitions...", "673981": "Hey @xiejialun how are you ensembling your models? As for me LB is not consistent with different ensembles", "674126": "ubamba98 I'm using average the raw prediction for ensemble. I did train these model with slightly different process/loss on purpose to make them can help each other's performance. I don't sure this will help or I just get luckyπ. But the number of model is quite consistent with LB.", "674193": "Thanks I will try that in future.", "674199": "I'm also struggling to find a good model for ensemble. π", "674202": "it is difficult. sometimes we use stacking or learned another level-2 network. \nyou can also see this: https://ai.googleblog.com/2018/10/introducing-adanet-fast-and-flexible.html\n\n\"Building an ensemble of neural networks has several challenges: What are the best subnetwork architectures to consider? Is it best to reuse the same architectures or encourage diversity? While complex subnetworks with more parameters will tend to perform better on the training set, they may not generalize to unseen data due to their greater complexity. \"", "674211": "Single fpn 6640 + single unet 6612 only got 6647 : (", "674212": "You're right, it's difficult.\nSo I haven't improved my LB score in days. π", "674224": "Sometimes itβs fun to see the 1st place make a joke π π", "674227": "\"Single fpn 6640 + single unet 6612 only got 6647 : (\"\n\nmy suggestion is:\n\n1. breakdown the results of single model and ensemble, e.g.\nsingle model false positive =xx, ensemble model false positive =yy\n\n\n2. if yy is better than xx, than we know that the effects of ensemble is reduction of fp.\n\n3. then we can think of ways to further improve this effect or to address problems that are solved by ensemble", "674228": "\"So I haven't improved my LB score in days\"\n\nlesson from steel competition is that preventing shakeup is also important", "674236": "Yes... It's a sad experience for us.\nAnd when I found that I got a score [private: 0.90828, public: 0.92286], when I just set threshold to [0.7, 0.7, 0.7, 0.7], I cried... πΏ \n\nSo I will only use high thresholds in this competition...", "674243": "my experiment shows higher threshold works better but we still have to be cautious.\n\ni haven't think of a way yet but we should think of a way to use both the public and private data to confirm. (unlike the steel competition, this time we get to see the private data set as well)\n\nsome possible solutions\n1. visual inspection of test data\n2. measure the similarity of test data \n3. think of a way to verify label consistency in test data. e.g. we make prediction on test. then we use pseudo label only to train and verify on train set. then we apply mixup augmentation pseudo label and see the effects on error of the same train set, etc ...\n\nsince we are only interested in image level label for fp rejection, we have a lot of options actually. it is a problem of verifying (or robustness, uncertainty estimation) the classification (not segmentation) of test set", "674260": "Thanks for your suggestions.\nHeng, I respect your sharing and I'm always rooting for you.", "674448": "single image size, 8x folds Unet, 3 more folds Unet with different encoders. Contribution of the different encoder is negligble I think.", "674452": "I can only gain very small benefit on multi-fold training, but different encoder did improve my score. This is quite weird haha.\n\nBtw, can't wait to see your solution! single model with 0.667..that's insane!", "674518": "another is diversity in loss, e.g\n\n```\n\nclass Net(nn.Module):\n ....\n def forward(self, x):\n feature = self.block(x)\n ....\n .....\n label_probability1 = self.one_way_to_predict( ....)\n ....\n .....\n label_probability2 = self.another_way_to_predict( .... different feature used ....)\n .....\n label_probability3 = self.yet_another_way_to_predict( .... yet different feature used ....)\n\n\n```\n\nfor inference, average the 3 label prediction (since image label is important, we only create multi classifiers).\n\nthis is like using shared features to learn 3 classifiers. and then average them like ensemble.\n\nfor back propagation, use:\n\n```\nloss = a*loss(label_probability1 ) + b*loss(label_probability2 ) + c*loss(label_probability3 ) + ...\n\na,b,c are random variable with certain range e.g. a+b+c = 1 and a in range 0.3333 +/- noise, etc ....\n```\n\nwe inject the weighing loss noise to make it more robust", "674780": "Got .6671 network, now I suspect some kaggler got even higher single model performance...", "674790": "definitely yes.\n\nthe ensemble results just prove that there is that much discriminative information that can extracted from the train set.\n\nit is whether we can design good learning method to extract the information for a single model\n\nif you read through the papers of imagenet, you will find that the ensemble of many vgg-16 is close to the performance of single resnet50", "674842": "single fpn with or without classifier?", "674869": "niuddd amazing work! is it K-fold or single fold?", "675374": "I got 0.6686 from a single network with 4 folds. However, looks like it doesn't matter; at least for the Public LB. Because while ensembling, the result becomes funnier. For example, the ensemble of lower scored networks giving higher public LB compared to the ensemble with the score of the best-scored network. I am not sure what is causing that.", "675375": "mykttu true I am also experiencing the same effect I don't think private LB is a good generalization.", "675415": "I made a mistake, 6640 model is unet resnet34. So it is a reasonable ensemble score.", "675442": "True. Going beyond .670, I realized there's sub-competition of tuning ensembles.", "675446": "Another question is would you select two best scoring (public LB) submissions, or would you prefer two that have most confidence with? Personally, I would choose at least one to be a big ensemble (of as many models as I have), which statistically has lower variance (though lower LB) to fight against the possible shake up. What about you?", "675455": "I think submission should be selected such that you can minimize the false positives. I will select my submission based on it not the lb score.", "675463": "Same as you, one with best public LB, one with big ensemble.\nGood luck", "675482": "5 different input size model with same head and decoder got me from 0.65 to 0.666", "675544": "5 different architectures each 5 fold.\nEnsemble Strategy in 5 folds -> sigmoid(mean(probabilities))\nEnsemble Strategy to merge binary masks -> max voting\nBest single architecture performance -> 0.6658\nLB score -> 0.6674", "675551": "What is the voting threshold did you use I am experiencing a huge difference in LB score with different thresh in voting. Is that the case with you too?", "675587": "I have experience such situation. Shakeup is around 0.001. Currently I found out higher theshold has better LB score.", "675594": "Ya same. Its really hard to select final submission." }, "source": "meta" }