UPDATE: code is now available here.
\n\nThanks for Max Planck Institute for Meteorology and Kaggle for hosting such an interesting competition. Congrats to all the winners.
\n\nThe key in my solution is training two segmentation models: seg1 trained on all data with BCE loss, and seg2 trained on non-empty images only with soft DICE loss. I think it works because this competition basically has two tasks: 1) detect the empty images; 2) predict accurate masks for the non-empty images. The two segmentation models address these two tasks respectively.
\n\nI started the competition with resnet34-FPN using BCE loss (seg1). This model achieves ~0.608 on LB and the major contribution comes from capturing the empty mask: it captures ~80% empty masks. I tried a lot to improve the non-empty part, like using combo loss of BCE and DICE, but it is hard to improve the neg-dice (dice score for the empty masks) and pos-dice (dice score for the non-empty makes) simultaneously.
\n\nTo predict the non-empty mask accurately, I decided to train 4 individual segmentation models for the non-empty images and then ensemble them together. Since all the train images are non-empty, we can use soft DICE loss directly and the model would focus on predicting accurate masks. I used exactly the same network structure, resnet34-FPN (seg2). Then I simply replace all the non-empty predictions from seg1 model using the predictions from ‘seg2’. Only 1 fold of this 2-stage segmentation pipeline, no TTA, no min-size remover, no classifier, no threshold adjustment (all 0.5) could achieve LB 0.652. After including a resnet34 classifier (0.5 threshold), I got LB 0.655.
\n\nLater on, I managed to train all 4 classes in one model by implementing pos-only soft DICE loss. The code looks like:
\n\npython\ndef dice_only_pos(logits, labels, labels_fc):\n # logits -> pixel level predictions\n # labels -> pixel level labels\n # labels_fc -> image/channel level labels\n pos_idx = (labels_fc > 0.5)\n neg_idx = (labels_fc < 0.5)\n loss = SoftDiceLoss()(logits[pos_idx], labels[pos_idx])\n return loss\n\nThis loss only counts the non-empty channels and ignores all the empty channels.
In summary the pipeline looks like: \n>1. seg1: a multi-label segmentation model trained with BCE loss\n>2. seg2: a multi-label segmentation model trained with pos-only soft DICE loss\n>3. cls: a multi-label classifier trained with BCE loss.
\n\nThe final submission is achieved by the following steps:\n>1. Get predictions using seg1\n>2. Replacing the non-empty masks from seg1 by predictions from seg2\n>3. Removing more empty masks using cls
\n\nBoth pixel-level (segmentation) and image-level (classifier) thresholds are 0.5.
\n\nModel summary:\n>Network: Resnet34-FPN \n>Image size: 384x576\n>Batch size: 16\n>Optimizer: Adam\n>Scheduler: reduceLR for seg1, warmRestart for seg2.\n>Augmentations: H/V flip, ShiftScalerRotate and GridDistortion\n>TTA: raw, Horizontal Flip, Vertical Flip
\n\nResults:\n>1-fold: 0.664 \n>5-fold + TTA3: 0.669\n>5-fold + TTA3 + classifier: 0.670.
\n\nTTA1 means only raw images; TTA3 means raw + H/V flip.
\n\nThe rest of my work is just trying different backbones to find the best one. My final models are:
\n\n>seg1: densenet121-FPN, TTA1\n>seg2: b7-FPN, TTA3\n>cls: b1, TTA1
\n\nResults:\n>1-fold LB: 0.673\n>5-fold LB: 0.6788
\n\nI ensembled multiple seg2 models using major vote. By including 4 models (b5-Unet, InceptionResnetV2-FPN, b7-FPN and b7-Unet), I achieved 0.6792 on LB.
\n\nI selected the pseudo labels based a LB 0.6790 submission with the following rules:\n>1. Empty channels with classifier prediction < 0.3\n>2. Non-empty channels with classifier prediction > 0.7
\n\nAn image is selected when all the 4 channels satisfy one of the conditions. 835 images are selected. I retrained the b7-FPN and b1-classifier including the pseudo labeling samples, and the final models are:\n>seg1: densenet121-FPN, TTA1\n>seg2: b5-Unet + InceptionResnetV2-FPN + b7-Unet + b7-FPN + b7-FPN-PL, TTA3\n>cls: b1-PL, TTA3
\n\nPL means the model is retrained with pseudo labels
\n\nThis model achieves 0.6794 LB.
\n\nOn the last day, I decide to optimize the classifier threshold channel wise to achieve the best local CV, which gives me 0.6805 LB.
\n\nUPDATE: code is now available here.
\n\nThanks for Max Planck Institute for Meteorology and Kaggle for hosting such an interesting competition. Congrats to all the winners.
\n\nThe key in my solution is training two segmentation models: seg1 trained on all data with BCE loss, and seg2 trained on non-empty images only with soft DICE loss. I think it works because this competition basically has two tasks: 1) detect the empty images; 2) predict accurate masks for the non-empty images. The two segmentation models address these two tasks respectively.
\n\nI started the competition with resnet34-FPN using BCE loss (seg1). This model achieves ~0.608 on LB and the major contribution comes from capturing the empty mask: it captures ~80% empty masks. I tried a lot to improve the non-empty part, like using combo loss of BCE and DICE, but it is hard to improve the neg-dice (dice score for the empty masks) and pos-dice (dice score for the non-empty makes) simultaneously.
\n\nTo predict the non-empty mask accurately, I decided to train 4 individual segmentation models for the non-empty images and then ensemble them together. Since all the train images are non-empty, we can use soft DICE loss directly and the model would focus on predicting accurate masks. I used exactly the same network structure, resnet34-FPN (seg2). Then I simply replace all the non-empty predictions from seg1 model using the predictions from ‘seg2’. Only 1 fold of this 2-stage segmentation pipeline, no TTA, no min-size remover, no classifier, no threshold adjustment (all 0.5) could achieve LB 0.652. After including a resnet34 classifier (0.5 threshold), I got LB 0.655.
\n\nLater on, I managed to train all 4 classes in one model by implementing pos-only soft DICE loss. The code looks like:
\n\npython\ndef dice_only_pos(logits, labels, labels_fc):\n # logits -> pixel level predictions\n # labels -> pixel level labels\n # labels_fc -> image/channel level labels\n pos_idx = (labels_fc > 0.5)\n neg_idx = (labels_fc < 0.5)\n loss = SoftDiceLoss()(logits[pos_idx], labels[pos_idx])\n return loss\n\nThis loss only counts the non-empty channels and ignores all the empty channels.
In summary the pipeline looks like: \n>1. seg1: a multi-label segmentation model trained with BCE loss\n>2. seg2: a multi-label segmentation model trained with pos-only soft DICE loss\n>3. cls: a multi-label classifier trained with BCE loss.
\n\nThe final submission is achieved by the following steps:\n>1. Get predictions using seg1\n>2. Replacing the non-empty masks from seg1 by predictions from seg2\n>3. Removing more empty masks using cls
\n\nBoth pixel-level (segmentation) and image-level (classifier) thresholds are 0.5.
\n\nModel summary:\n>Network: Resnet34-FPN \n>Image size: 384x576\n>Batch size: 16\n>Optimizer: Adam\n>Scheduler: reduceLR for seg1, warmRestart for seg2.\n>Augmentations: H/V flip, ShiftScalerRotate and GridDistortion\n>TTA: raw, Horizontal Flip, Vertical Flip
\n\nResults:\n>1-fold: 0.664 \n>5-fold + TTA3: 0.669\n>5-fold + TTA3 + classifier: 0.670.
\n\nTTA1 means only raw images; TTA3 means raw + H/V flip.
\n\nThe rest of my work is just trying different backbones to find the best one. My final models are:
\n\n>seg1: densenet121-FPN, TTA1\n>seg2: b7-FPN, TTA3\n>cls: b1, TTA1
\n\nResults:\n>1-fold LB: 0.673\n>5-fold LB: 0.6788
\n\nI ensembled multiple seg2 models using major vote. By including 4 models (b5-Unet, InceptionResnetV2-FPN, b7-FPN and b7-Unet), I achieved 0.6792 on LB.
\n\nI selected the pseudo labels based a LB 0.6790 submission with the following rules:\n>1. Empty channels with classifier prediction < 0.3\n>2. Non-empty channels with classifier prediction > 0.7
\n\nAn image is selected when all the 4 channels satisfy one of the conditions. 835 images are selected. I retrained the b7-FPN and b1-classifier including the pseudo labeling samples, and the final models are:\n>seg1: densenet121-FPN, TTA1\n>seg2: b5-Unet + InceptionResnetV2-FPN + b7-Unet + b7-FPN + b7-FPN-PL, TTA3\n>cls: b1-PL, TTA3
\n\nPL means the model is retrained with pseudo labels
\n\nThis model achieves 0.6794 LB.
\n\nOn the last day, I decide to optimize the classifier threshold channel wise to achieve the best local CV, which gives me 0.6805 LB.
\n\n\n\n\nFinally GM& 1st time won prize money! And 3rd place solution.
\n
I have 3x congratulations for you!!! Well done and well deserved.
", "rawMarkdown": "> Finally GM& 1st time won prize money! And 3rd place solution.\n\nI have 3x congratulations for you!!! Well done and well deserved.", "votes": 1 }, { "id": 676132, "postDate": "2019-11-19T01:32:28.413Z", "content": "What you did is truly amazing! Congratulations, and thank you for sharing your solution.
\n\nIf I understood correctly, we can say that you used 'seg model with pos-only dice loss' as a base network and used 'seg model with bce loss' and 'cls model' to zero-out some masks to get public score of 0.670. Then you changed backbones of those three models and got 0.6788.
\n\nIt is surprising that just changing backbones leaded to so much improvement of nearly 0.09. Many of us didn't enjoy significant improvement from changing backbone architectures.
\n\nWhat do you think made such an improvement possible?
\n\nOne more, you said you used 'soft dice-loss'. I googled it and found this. (which references https://mediatum.ub.tum.de/doc/1395260/1395260.pdf page 72) It squares masks element-wise.
\n\n\ndef soft_dice_loss(y_true, y_pred, epsilon=1e-6): \n axes = tuple(range(1, len(y_pred.shape)-1)) \n numerator = 2. * np.sum(y_pred * y_true, axes)\n denominator = np.sum(np.square(y_pred) + np.square(y_true), axes)\n return 1 - np.mean(numerator / (denominator + epsilon))\n
Did using it instead of normal dice loss improve your score?
\n\nThanks again for your kind writeup.
", "rawMarkdown": "What you did is truly amazing! Congratulations, and thank you for sharing your solution.\n\nIf I understood correctly, we can say that you used **'seg model with pos-only dice loss'** as a base network and used **'seg model with bce loss'** and **'cls model'** to zero-out some masks to get public score of **0.670**. Then you changed backbones of those three models and got **0.6788**.\n\nIt is surprising that just changing backbones leaded to so much improvement of nearly **0.09**. Many of us didn't enjoy significant improvement from changing backbone architectures.\n\nWhat do you think made such an improvement possible?\n\nOne more, you said you used 'soft dice-loss'. I googled it and found [this](https://gist.github.com/jeremyjordan/9ea3032a32909f71dd2ab35fe3bacc08). (which references https://mediatum.ub.tum.de/doc/1395260/1395260.pdf page 72) It squares masks element-wise.\n\n```\ndef soft_dice_loss(y_true, y_pred, epsilon=1e-6): \n axes = tuple(range(1, len(y_pred.shape)-1)) \n numerator = 2. * np.sum(y_pred * y_true, axes)\n denominator = np.sum(np.square(y_pred) + np.square(y_true), axes)\n return 1 - np.mean(numerator / (denominator + epsilon))\n```\n\nDid using it instead of normal dice loss improve your score?\n\nThanks again for your kind writeup.", "votes": 2, "replies": [ { "id": 676227, "postDate": "2019-11-19T03:19:58.580Z", "content": "\n\n\nWhat do you think made such an improvement possible?
\n
My 2-stage segmentation pipeline. It is much easier to improve empty and non-empty predictions respectively.
\n\nThe soft-dice loss is basically soft-f1 loss at pixel level.
", "rawMarkdown": ">What do you think made such an improvement possible?\n\nMy 2-stage segmentation pipeline. It is much easier to improve empty and non-empty predictions respectively. \n\nThe soft-dice loss is basically soft-f1 loss at pixel level. ", "votes": 1 } ] }, { "id": 676058, "postDate": "2019-11-19T00:09:15.323Z", "content": "Congratulations. Well deserved. That solo gold is a doozy.
\n\nAhh I was so close. I have a variant of your non-empty loss as well. My rationale was that it would focus purely on improving the quality of the mask and less on reducing the false positives. Ended up not moving forward with it because I couldnt get anything out of it initially. Shouldve spent more time on it
", "rawMarkdown": "Congratulations. Well deserved. That solo gold is a doozy. \n\nAhh I was so close. I have a variant of your non-empty loss as well. My rationale was that it would focus purely on improving the quality of the mask and less on reducing the false positives. Ended up not moving forward with it because I couldnt get anything out of it initially. Shouldve spent more time on it", "votes": 2, "replies": [ { "id": 676060, "postDate": "2019-11-19T00:12:31.323Z", "content": "Yeah, you are so close! This technique makes improving models much easier, since you only need to focus on one thing at a time.
", "rawMarkdown": "Yeah, you are so close! This technique makes improving models much easier, since you only need to focus on one thing at a time. ", "votes": 1 } ] }, { "id": 677203, "postDate": "2019-11-19T23:17:56.830Z", "content": "Congrats!
", "rawMarkdown": "Congrats!", "replies": [ { "id": 678692, "postDate": "2019-11-21T18:23:21.023Z", "content": "Thanks.
", "rawMarkdown": "Thanks. " } ] }, { "id": 677149, "postDate": "2019-11-19T21:25:01.337Z", "content": "Congratulations, great work and post !
", "rawMarkdown": "Congratulations, great work and post !", "replies": [ { "id": 678693, "postDate": "2019-11-21T18:23:27.427Z", "content": "Thanks.
", "rawMarkdown": "Thanks. " } ] }, { "id": 677106, "postDate": "2019-11-19T19:59:45.377Z", "content": "Congratz!
", "rawMarkdown": "Congratz! ", "replies": [ { "id": 678695, "postDate": "2019-11-21T18:23:35.620Z", "content": "Thanks.
", "rawMarkdown": "Thanks. " } ] }, { "id": 677008, "postDate": "2019-11-19T17:53:03.477Z", "content": "Congratulations Xuan on solo Gold and becoming Grand Master.
\n\nI like your def dice_only_pos(logits, labels, labels_fc) function. I plan to use that in the future. That is key for Kaggle competitions with their discontinuous Dice metric.
I also noticed how different segmentation models would achieve different empty mask prediction accuracies. I even added a metric to my models:
\n\ndef kaggle_acc(y_true, y_pred0, pix=0.5, area=24000, dim=(384,576)):\n\n# PIXEL THRESHOLD\ny_pred = K.cast( K.greater(y_pred0,pix), K.floatx() )\n\n# MIN AREA THRESHOLD\ns = K.sum(y_pred, axis=(1,2))\ns = K.cast( K.greater(s, area), K.floatx() )\n\n# REMOVE MIN AREA\ns = K.reshape(s,(-1,1))\ns = K.repeat(s,dim[0]*dim[1])\ns = K.reshape(s,(-1,1))\ny_pred = K.permute_dimensions(y_pred,(0,3,1,2))\ny_pred = K.reshape(y_pred,shape=(-1,1))\ny_pred = s*y_pred\ny_pred = K.reshape(y_pred,(-1,y_pred0.shape[3],dim[0],dim[1]))\ny_pred = K.permute_dimensions(y_pred,(0,2,3,1))\n\n# COMPUTE KAGGLE ACC\ntotal_y_true = K.sum(y_true, axis=(1,2))\ntotal_y_true = K.cast( K.greater(total_y_true, 0), K.floatx() )\n\ntotal_y_pred = K.sum(y_pred, axis=(1,2))\ntotal_y_pred = K.cast( K.greater(total_y_pred, 0), K.floatx() )\n\nreturn 1 - K.mean( K.abs( total_y_pred - total_y_true ) )\nAnd then similar to you, I noticed that some models had low LB score but very high accuracy predicting empty masks. While other models had higher LB score but lower accuracy predicting empty masks. I did something similar to you and converted the low LB high ACC models into classifiers to remove false positives from the high LB low ACC models.
", "rawMarkdown": "Congratulations Xuan on solo Gold and becoming Grand Master. \n\nI like your `def dice_only_pos(logits, labels, labels_fc)` function. I plan to use that in the future. That is key for Kaggle competitions with their discontinuous Dice metric. \n\nI also noticed how different segmentation models would achieve different empty mask prediction accuracies. I even added a metric to my models:\n\n def kaggle_acc(y_true, y_pred0, pix=0.5, area=24000, dim=(384,576)):\n \n # PIXEL THRESHOLD\n y_pred = K.cast( K.greater(y_pred0,pix), K.floatx() )\n \n # MIN AREA THRESHOLD\n s = K.sum(y_pred, axis=(1,2))\n s = K.cast( K.greater(s, area), K.floatx() )\n\n # REMOVE MIN AREA\n s = K.reshape(s,(-1,1))\n s = K.repeat(s,dim[0]*dim[1])\n s = K.reshape(s,(-1,1))\n y_pred = K.permute_dimensions(y_pred,(0,3,1,2))\n y_pred = K.reshape(y_pred,shape=(-1,1))\n y_pred = s*y_pred\n y_pred = K.reshape(y_pred,(-1,y_pred0.shape[3],dim[0],dim[1]))\n y_pred = K.permute_dimensions(y_pred,(0,2,3,1))\n\n # COMPUTE KAGGLE ACC\n total_y_true = K.sum(y_true, axis=(1,2))\n total_y_true = K.cast( K.greater(total_y_true, 0), K.floatx() )\n\n total_y_pred = K.sum(y_pred, axis=(1,2))\n total_y_pred = K.cast( K.greater(total_y_pred, 0), K.floatx() )\n\n return 1 - K.mean( K.abs( total_y_pred - total_y_true ) )\n\nAnd then similar to you, I noticed that some models had low LB score but very high accuracy predicting empty masks. While other models had higher LB score but lower accuracy predicting empty masks. I did something similar to you and converted the low LB high ACC models into classifiers to remove false positives from the high LB low ACC models.", "replies": [ { "id": 677348, "postDate": "2019-11-20T04:05:08.257Z", "content": "\n\n\nThat is key for Kaggle competitions with their discontinuous Dice metric.
\n
Yes, I realized it in steel, but was unable to overcome it there. There were too many empty masks in steel, and this approach would make each batch very unstable. I was thinking about training 4 segmentation models but gave up at last considering the inference time limit.
\n\nGood luck to your image competition journal, wish you get your first CV gold medal soon!
", "rawMarkdown": ">That is key for Kaggle competitions with their discontinuous Dice metric.\n\nYes, I realized it in steel, but was unable to overcome it there. There were too many empty masks in steel, and this approach would make each batch very unstable. I was thinking about training 4 segmentation models but gave up at last considering the inference time limit. \n\nGood luck to your image competition journal, wish you get your first CV gold medal soon! \n\n", "votes": 1 } ] }, { "id": 676988, "postDate": "2019-11-19T17:28:52.513Z", "content": "Very nice and congratulations! I really like the write up. It is nice to be in the head of the winners sometimes ;-)
\n\nThe \"things worth mentioning\" I think are probably very important - especially point 1! Always trust your CV
", "rawMarkdown": "Very nice and congratulations! I really like the write up. It is nice to be in the head of the winners sometimes ;-)\n\nThe \"things worth mentioning\" I think are probably very important - especially point 1! Always trust your CV" }, { "id": 676685, "postDate": "2019-11-19T12:23:24.150Z", "content": "Double congrats @naivelamb !
", "rawMarkdown": "Double congrats @naivelamb ! ", "replies": [ { "id": 677066, "postDate": "2019-11-19T18:49:35.027Z", "content": "Thanks Giba!
", "rawMarkdown": "Thanks Giba! " } ] }, { "id": 676660, "postDate": "2019-11-19T11:55:39.980Z", "content": "Congratulations @naivelamb , great work with your solution. Your GM title is well deserved :)
", "rawMarkdown": "Congratulations @naivelamb , great work with your solution. Your GM title is well deserved :)" }, { "id": 676641, "postDate": "2019-11-19T11:39:23.020Z", "content": "Сongratulations and thanks!
\n\nseg1: a multi-label segmentation model trained with BCE loss\nseg2: a multi-label segmentation model trained with pos-only soft DICE loss\ncls: a multi-label classifier trained with BCE loss.
\n\nGood idea!
", "rawMarkdown": "Сongratulations and thanks!\n\nseg1: a multi-label segmentation model trained with BCE loss\nseg2: a multi-label segmentation model trained with pos-only soft DICE loss\ncls: a multi-label classifier trained with BCE loss.\n\nGood idea!" }, { "id": 676612, "postDate": "2019-11-19T10:54:07.650Z", "content": "Great work, thanks for your write up, and congratulations!
\n\nJust out of interest - were you using Kaggle Kernels, or do you have your own GPU?
", "rawMarkdown": "Great work, thanks for your write up, and congratulations!\n\nJust out of interest - were you using Kaggle Kernels, or do you have your own GPU?", "replies": [ { "id": 677065, "postDate": "2019-11-19T18:49:25.317Z", "content": "I used my own GPU.
", "rawMarkdown": "I used my own GPU. ", "votes": 1 } ] }, { "id": 676401, "postDate": "2019-11-19T06:18:57.220Z", "content": "Great write up! Well deserved GM!!
", "rawMarkdown": "Great write up! Well deserved GM!!" }, { "id": 676288, "postDate": "2019-11-19T04:22:22.940Z", "content": "Congratulations, good solution:)
", "rawMarkdown": "Congratulations, good solution:)" }, { "id": 676213, "postDate": "2019-11-19T03:02:06.337Z", "content": "Congratulations！\nI saw you trained several big models(with B5/B7 backbone), can you tell me how did you set the batchsize when you train your biggest model?
", "rawMarkdown": "Congratulations！\nI saw you trained several big models(with B5/B7 backbone), can you tell me how did you set the batchsize when you train your biggest model? ", "replies": [ { "id": 676233, "postDate": "2019-11-19T03:23:41.593Z", "content": "I have mentioned the batch size, 16 for all models.
", "rawMarkdown": "I have mentioned the batch size, 16 for all models. ", "votes": 1 }, { "id": 676242, "postDate": "2019-11-19T03:30:14.200Z", "content": "Oh, sorry, I comment after I read your topic roughtly. Your topic includes so much information which is useful for me, thanks for your sharing!
", "rawMarkdown": "Oh, sorry, I comment after I read your topic roughtly. Your topic includes so much information which is useful for me, thanks for your sharing!" }, { "id": 677124, "postDate": "2019-11-19T20:44:50.183Z", "content": "Congratulations for your performance ! \nDid batch size of 16 fit in your GPU memory even for large architectures ? Did you use fp16 ?
", "rawMarkdown": "Congratulations for your performance ! \nDid batch size of 16 fit in your GPU memory even for large architectures ? Did you use fp16 ?" }, { "id": 677323, "postDate": "2019-11-20T03:35:13.970Z", "content": "apex is your good friend.
", "rawMarkdown": "apex is your good friend. ", "votes": 1 } ] }, { "id": 676206, "postDate": "2019-11-19T02:53:15.597Z", "content": "Congratulations😄 😄
", "rawMarkdown": "Congratulations😄 😄 ", "replies": [ { "id": 676235, "postDate": "2019-11-19T03:24:31.123Z", "content": "Same to you.
", "rawMarkdown": "Same to you. " } ] }, { "id": 676174, "postDate": "2019-11-19T02:18:59.670Z", "content": "Congrats!!!!! Pseudo labeling part is really impressive!!
", "rawMarkdown": "Congrats!!!!! Pseudo labeling part is really impressive!!" }, { "id": 676172, "postDate": "2019-11-19T02:15:46.563Z", "content": "Hi, Congratulations for getting money and GM!!
\n\nI have a question on some part of your solution.
\n\nYou mentioned the below sentences.
\n\n> To predict the non-empty mask accurately, I decided to train 4 individual segmentation models for the non-empty images and then ensemble them together.
\n\nPlease correct my understanding.
\n\nSeg 1.
\n\nMake a 4-class segmentation model. and get sub1.
\n\nSeg 2.\nMake binary segmentation models for 4 each label. (4 segmentation models)\nImages of this competition can have multiple labels, so the same images can be used to train multiple labels.
\n\nAnd, replace the predictions in sub1 with the predictions from 4 segmentation models.
\n\ncls 1.\nMake a 4-class classifier(finding an empty mask). and remove the false-postive masks in sub1.
", "rawMarkdown": "Hi, Congratulations for getting money and GM!!\n\nI have a question on some part of your solution.\n\nYou mentioned the below sentences.\n\n> To predict the non-empty mask accurately, I decided to train 4 individual segmentation models for the non-empty images and then ensemble them together.\n\nPlease correct my understanding. \n\nSeg 1.\n\nMake a 4-class segmentation model. and get sub1.\n\nSeg 2.\nMake binary segmentation models for 4 each label. (4 segmentation models)\nImages of this competition can have multiple labels, so the same images can be used to train multiple labels. \n\nAnd, replace the predictions in sub1 with the predictions from 4 segmentation models.\n\ncls 1.\nMake a 4-class classifier(finding an empty mask). and remove the false-postive masks in sub1.", "replies": [ { "id": 676175, "postDate": "2019-11-19T02:19:19.450Z", "content": "At first, you did that. Later on, you made one seg2 model with special loss (only pos), right..?
", "rawMarkdown": "At first, you did that. Later on, you made one seg2 model with special loss (only pos), right..?" }, { "id": 676232, "postDate": "2019-11-19T03:23:15.030Z", "content": "The very first version of seg2 is 4 binary segmentation models trained on non-empty images only (channel wise, so different channels will have different numbers of train images).
\n\nThe non-empty predictions in sub1 is replaced by those from seg2, since seg2 is better in giving accurate masks.
", "rawMarkdown": "The very first version of seg2 is 4 binary segmentation models trained on non-empty images only (channel wise, so different channels will have different numbers of train images). \n\nThe non-empty predictions in sub1 is replaced by those from seg2, since seg2 is better in giving accurate masks. " } ] }, { "id": 676160, "postDate": "2019-11-19T02:04:17.497Z", "content": "Congrats Cao.
", "rawMarkdown": "Congrats Cao. " }, { "id": 676155, "postDate": "2019-11-19T01:57:26.630Z", "content": "Congrats on winning and thanks for sharing the solution over to us.
", "rawMarkdown": "Congrats on winning and thanks for sharing the solution over to us." }, { "id": 676148, "postDate": "2019-11-19T01:48:41.437Z", "content": "Double congratulations @naivelamb and well done.
\n\nYour solution is interesting and thanks for sharing. Would you be rreleasing your code?
", "rawMarkdown": "Double congratulations @naivelamb and well done. \n\nYour solution is interesting and thanks for sharing. Would you be rreleasing your code?", "replies": [ { "id": 676234, "postDate": "2019-11-19T03:24:20.337Z", "content": "Thanks. I haven't decided for the code, it is messy.
", "rawMarkdown": "Thanks. I haven't decided for the code, it is messy. ", "votes": 1 }, { "id": 677181, "postDate": "2019-11-19T22:23:05.120Z", "content": "@naivelamb, thanks for answering my question. I do not mind a messy code as I would like to reproduce the top results of this comeptition with the aim of beating the 1st place scores. I want to do that with a single model that combines different top ideas in a couple of weeks when I have some free time.
", "rawMarkdown": "@naivelamb, thanks for answering my question. I do not mind a messy code as I would like to reproduce the top results of this comeptition with the aim of beating the 1st place scores. I want to do that with a single model that combines different top ideas in a couple of weeks when I have some free time." } ] }, { "id": 676121, "postDate": "2019-11-19T01:16:22.683Z", "content": "Congratulations :)
", "rawMarkdown": "Congratulations :) " }, { "id": 676061, "postDate": "2019-11-19T00:14:44.130Z", "content": "Thank you for sharing and congratulations!
", "rawMarkdown": "Thank you for sharing and congratulations!" }, { "id": 676056, "postDate": "2019-11-19T00:07:12.440Z", "content": "Congrats! Thanks for sharing, very impressive!
", "rawMarkdown": "Congrats! Thanks for sharing, very impressive!" }, { "id": 676053, "postDate": "2019-11-19T00:05:16.813Z", "content": "Congratulations! And thanks for sharing!!!
", "rawMarkdown": "Congratulations! And thanks for sharing!!!" }, { "id": 676349, "postDate": "2019-11-19T05:22:39.070Z", "rawMarkdown": "", "isDeleted": true }, { "id": 676714, "postDate": "2019-11-19T12:59:36.957Z", "content": "Congrats! Thanks for sharing!
", "rawMarkdown": "Congrats! Thanks for sharing!" }, { "id": 676397, "postDate": "2019-11-19T06:15:05.733Z", "content": "Congrats Xuan! Thanks for sharing!
", "rawMarkdown": "Congrats Xuan! Thanks for sharing!" } ], "comments": [ { "id": 676104, "author_name": "Bibek", "author_url": "", "post_date": "2019-11-19T00:54:02.740000", "content": "\n\n\nFinally GM& 1st time won prize money! And 3rd place solution.
\n
I have 3x congratulations for you!!! Well done and well deserved.
", "votes": 1, "replies": [] }, { "id": 676132, "author_name": "YoonSoo", "author_url": "", "post_date": "2019-11-19T01:32:28.413000", "content": "What you did is truly amazing! Congratulations, and thank you for sharing your solution.
\n\nIf I understood correctly, we can say that you used 'seg model with pos-only dice loss' as a base network and used 'seg model with bce loss' and 'cls model' to zero-out some masks to get public score of 0.670. Then you changed backbones of those three models and got 0.6788.
\n\nIt is surprising that just changing backbones leaded to so much improvement of nearly 0.09. Many of us didn't enjoy significant improvement from changing backbone architectures.
\n\nWhat do you think made such an improvement possible?
\n\nOne more, you said you used 'soft dice-loss'. I googled it and found this. (which references https://mediatum.ub.tum.de/doc/1395260/1395260.pdf page 72) It squares masks element-wise.
\n\n\ndef soft_dice_loss(y_true, y_pred, epsilon=1e-6): \n axes = tuple(range(1, len(y_pred.shape)-1)) \n numerator = 2. * np.sum(y_pred * y_true, axes)\n denominator = np.sum(np.square(y_pred) + np.square(y_true), axes)\n return 1 - np.mean(numerator / (denominator + epsilon))\n
Did using it instead of normal dice loss improve your score?
\n\nThanks again for your kind writeup.
", "votes": 2, "replies": [ { "id": 676227, "author_name": "Xuan Cao", "author_url": "", "post_date": "2019-11-19T03:19:58.580000", "content": "\n\n\nWhat do you think made such an improvement possible?
\n
My 2-stage segmentation pipeline. It is much easier to improve empty and non-empty predictions respectively.
\n\nThe soft-dice loss is basically soft-f1 loss at pixel level.
", "votes": 1, "replies": [] } ] }, { "id": 676058, "author_name": "ryches", "author_url": "", "post_date": "2019-11-19T00:09:15.323000", "content": "Congratulations. Well deserved. That solo gold is a doozy.
\n\nAhh I was so close. I have a variant of your non-empty loss as well. My rationale was that it would focus purely on improving the quality of the mask and less on reducing the false positives. Ended up not moving forward with it because I couldnt get anything out of it initially. Shouldve spent more time on it
", "votes": 2, "replies": [ { "id": 676060, "author_name": "Xuan Cao", "author_url": "", "post_date": "2019-11-19T00:12:31.323000", "content": "Yeah, you are so close! This technique makes improving models much easier, since you only need to focus on one thing at a time.
", "votes": 1, "replies": [] } ] }, { "id": 677203, "author_name": "corochann", "author_url": "", "post_date": "2019-11-19T23:17:56.830000", "content": "Congrats!
", "votes": 0, "replies": [ { "id": 678692, "author_name": "Xuan Cao", "author_url": "", "post_date": "2019-11-21T18:23:21.023000", "content": "Thanks.
", "votes": 0, "replies": [] } ] }, { "id": 677149, "author_name": "Rubnenkov Ivan", "author_url": "", "post_date": "2019-11-19T21:25:01.337000", "content": "Congratulations, great work and post !
", "votes": 0, "replies": [ { "id": 678693, "author_name": "Xuan Cao", "author_url": "", "post_date": "2019-11-21T18:23:27.427000", "content": "Thanks.
", "votes": 0, "replies": [] } ] }, { "id": 677106, "author_name": "Psi", "author_url": "", "post_date": "2019-11-19T19:59:45.377000", "content": "Congratz!
", "votes": 0, "replies": [ { "id": 678695, "author_name": "Xuan Cao", "author_url": "", "post_date": "2019-11-21T18:23:35.620000", "content": "Thanks.
", "votes": 0, "replies": [] } ] }, { "id": 677008, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-11-19T17:53:03.477000", "content": "Congratulations Xuan on solo Gold and becoming Grand Master.
\n\nI like your def dice_only_pos(logits, labels, labels_fc) function. I plan to use that in the future. That is key for Kaggle competitions with their discontinuous Dice metric.
I also noticed how different segmentation models would achieve different empty mask prediction accuracies. I even added a metric to my models:
\n\ndef kaggle_acc(y_true, y_pred0, pix=0.5, area=24000, dim=(384,576)):\n\n# PIXEL THRESHOLD\ny_pred = K.cast( K.greater(y_pred0,pix), K.floatx() )\n\n# MIN AREA THRESHOLD\ns = K.sum(y_pred, axis=(1,2))\ns = K.cast( K.greater(s, area), K.floatx() )\n\n# REMOVE MIN AREA\ns = K.reshape(s,(-1,1))\ns = K.repeat(s,dim[0]*dim[1])\ns = K.reshape(s,(-1,1))\ny_pred = K.permute_dimensions(y_pred,(0,3,1,2))\ny_pred = K.reshape(y_pred,shape=(-1,1))\ny_pred = s*y_pred\ny_pred = K.reshape(y_pred,(-1,y_pred0.shape[3],dim[0],dim[1]))\ny_pred = K.permute_dimensions(y_pred,(0,2,3,1))\n\n# COMPUTE KAGGLE ACC\ntotal_y_true = K.sum(y_true, axis=(1,2))\ntotal_y_true = K.cast( K.greater(total_y_true, 0), K.floatx() )\n\ntotal_y_pred = K.sum(y_pred, axis=(1,2))\ntotal_y_pred = K.cast( K.greater(total_y_pred, 0), K.floatx() )\n\nreturn 1 - K.mean( K.abs( total_y_pred - total_y_true ) )\nAnd then similar to you, I noticed that some models had low LB score but very high accuracy predicting empty masks. While other models had higher LB score but lower accuracy predicting empty masks. I did something similar to you and converted the low LB high ACC models into classifiers to remove false positives from the high LB low ACC models.
", "votes": 0, "replies": [ { "id": 677348, "author_name": "Xuan Cao", "author_url": "", "post_date": "2019-11-20T04:05:08.257000", "content": "\n\n\nThat is key for Kaggle competitions with their discontinuous Dice metric.
\n
Yes, I realized it in steel, but was unable to overcome it there. There were too many empty masks in steel, and this approach would make each batch very unstable. I was thinking about training 4 segmentation models but gave up at last considering the inference time limit.
\n\nGood luck to your image competition journal, wish you get your first CV gold medal soon!
", "votes": 1, "replies": [] } ] }, { "id": 676988, "author_name": "TheGruffalo", "author_url": "", "post_date": "2019-11-19T17:28:52.513000", "content": "Very nice and congratulations! I really like the write up. It is nice to be in the head of the winners sometimes ;-)
\n\nThe \"things worth mentioning\" I think are probably very important - especially point 1! Always trust your CV
", "votes": 0, "replies": [] }, { "id": 676685, "author_name": "Giba", "author_url": "", "post_date": "2019-11-19T12:23:24.150000", "content": "Double congrats @naivelamb !
", "votes": 0, "replies": [ { "id": 677066, "author_name": "Xuan Cao", "author_url": "", "post_date": "2019-11-19T18:49:35.027000", "content": "Thanks Giba!
", "votes": 0, "replies": [] } ] }, { "id": 676660, "author_name": "Ram Ramrakhya", "author_url": "", "post_date": "2019-11-19T11:55:39.980000", "content": "Congratulations @naivelamb , great work with your solution. Your GM title is well deserved :)
", "votes": 0, "replies": [] }, { "id": 676641, "author_name": "Ivan Bagmut", "author_url": "", "post_date": "2019-11-19T11:39:23.020000", "content": "Сongratulations and thanks!
\n\nseg1: a multi-label segmentation model trained with BCE loss\nseg2: a multi-label segmentation model trained with pos-only soft DICE loss\ncls: a multi-label classifier trained with BCE loss.
\n\nGood idea!
", "votes": 0, "replies": [] }, { "id": 676612, "author_name": "Marco Gorelli", "author_url": "", "post_date": "2019-11-19T10:54:07.650000", "content": "Great work, thanks for your write up, and congratulations!
\n\nJust out of interest - were you using Kaggle Kernels, or do you have your own GPU?
", "votes": 0, "replies": [ { "id": 677065, "author_name": "Xuan Cao", "author_url": "", "post_date": "2019-11-19T18:49:25.317000", "content": "I used my own GPU.
", "votes": 1, "replies": [] } ] }, { "id": 676401, "author_name": "Hamish", "author_url": "", "post_date": "2019-11-19T06:18:57.220000", "content": "Great write up! Well deserved GM!!
", "votes": 0, "replies": [] }, { "id": 676288, "author_name": "liuze", "author_url": "", "post_date": "2019-11-19T04:22:22.940000", "content": "Congratulations, good solution:)
", "votes": 0, "replies": [] }, { "id": 676213, "author_name": "lifengnan", "author_url": "", "post_date": "2019-11-19T03:02:06.337000", "content": "Congratulations！\nI saw you trained several big models(with B5/B7 backbone), can you tell me how did you set the batchsize when you train your biggest model?
", "votes": 0, "replies": [ { "id": 676233, "author_name": "Xuan Cao", "author_url": "", "post_date": "2019-11-19T03:23:41.593000", "content": "I have mentioned the batch size, 16 for all models.
", "votes": 1, "replies": [] }, { "id": 676242, "author_name": "lifengnan", "author_url": "", "post_date": "2019-11-19T03:30:14.200000", "content": "Oh, sorry, I comment after I read your topic roughtly. Your topic includes so much information which is useful for me, thanks for your sharing!
", "votes": 0, "replies": [] }, { "id": 677124, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2019-11-19T20:44:50.183000", "content": "Congratulations for your performance ! \nDid batch size of 16 fit in your GPU memory even for large architectures ? Did you use fp16 ?
", "votes": 0, "replies": [] }, { "id": 677323, "author_name": "Xuan Cao", "author_url": "", "post_date": "2019-11-20T03:35:13.970000", "content": "apex is your good friend.
", "votes": 1, "replies": [] } ] }, { "id": 676206, "author_name": "Gary", "author_url": "", "post_date": "2019-11-19T02:53:15.597000", "content": "Congratulations😄 😄
", "votes": 0, "replies": [ { "id": 676235, "author_name": "Xuan Cao", "author_url": "", "post_date": "2019-11-19T03:24:31.123000", "content": "Same to you.
", "votes": 0, "replies": [] } ] }, { "id": 676174, "author_name": "jionie", "author_url": "", "post_date": "2019-11-19T02:18:59.670000", "content": "Congrats!!!!! Pseudo labeling part is really impressive!!
", "votes": 0, "replies": [] }, { "id": 676172, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-11-19T02:15:46.563000", "content": "Hi, Congratulations for getting money and GM!!
\n\nI have a question on some part of your solution.
\n\nYou mentioned the below sentences.
\n\n> To predict the non-empty mask accurately, I decided to train 4 individual segmentation models for the non-empty images and then ensemble them together.
\n\nPlease correct my understanding.
\n\nSeg 1.
\n\nMake a 4-class segmentation model. and get sub1.
\n\nSeg 2.\nMake binary segmentation models for 4 each label. (4 segmentation models)\nImages of this competition can have multiple labels, so the same images can be used to train multiple labels.
\n\nAnd, replace the predictions in sub1 with the predictions from 4 segmentation models.
\n\ncls 1.\nMake a 4-class classifier(finding an empty mask). and remove the false-postive masks in sub1.
", "votes": 0, "replies": [ { "id": 676175, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-11-19T02:19:19.450000", "content": "At first, you did that. Later on, you made one seg2 model with special loss (only pos), right..?
", "votes": 0, "replies": [] }, { "id": 676232, "author_name": "Xuan Cao", "author_url": "", "post_date": "2019-11-19T03:23:15.030000", "content": "The very first version of seg2 is 4 binary segmentation models trained on non-empty images only (channel wise, so different channels will have different numbers of train images).
\n\nThe non-empty predictions in sub1 is replaced by those from seg2, since seg2 is better in giving accurate masks.
", "votes": 0, "replies": [] } ] }, { "id": 676160, "author_name": "llh1818", "author_url": "", "post_date": "2019-11-19T02:04:17.497000", "content": "Congrats Cao.
", "votes": 0, "replies": [] }, { "id": 676155, "author_name": "Vishy", "author_url": "", "post_date": "2019-11-19T01:57:26.630000", "content": "Congrats on winning and thanks for sharing the solution over to us.
", "votes": 0, "replies": [] }, { "id": 676148, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-11-19T01:48:41.437000", "content": "Double congratulations @naivelamb and well done.
\n\nYour solution is interesting and thanks for sharing. Would you be rreleasing your code?
", "votes": 0, "replies": [ { "id": 676234, "author_name": "Xuan Cao", "author_url": "", "post_date": "2019-11-19T03:24:20.337000", "content": "Thanks. I haven't decided for the code, it is messy.
", "votes": 1, "replies": [] }, { "id": 677181, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-11-19T22:23:05.120000", "content": "@naivelamb, thanks for answering my question. I do not mind a messy code as I would like to reproduce the top results of this comeptition with the aim of beating the 1st place scores. I want to do that with a single model that combines different top ideas in a couple of weeks when I have some free time.
", "votes": 0, "replies": [] } ] }, { "id": 676121, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2019-11-19T01:16:22.683000", "content": "Congratulations :)
", "votes": 0, "replies": [] }, { "id": 676061, "author_name": "atfujita", "author_url": "", "post_date": "2019-11-19T00:14:44.130000", "content": "Thank you for sharing and congratulations!
", "votes": 0, "replies": [] }, { "id": 676056, "author_name": "Laevatein", "author_url": "", "post_date": "2019-11-19T00:07:12.440000", "content": "Congrats! Thanks for sharing, very impressive!
", "votes": 0, "replies": [] }, { "id": 676053, "author_name": "Hieu Phung", "author_url": "", "post_date": "2019-11-19T00:05:16.813000", "content": "Congratulations! And thanks for sharing!!!
", "votes": 0, "replies": [] }, { "id": 676349, "author_name": "", "author_url": "", "post_date": "2019-11-19T05:22:39.070000", "content": "", "votes": 0, "replies": [] }, { "id": 676714, "author_name": "bassbone(Junichi Ito)", "author_url": "", "post_date": "2019-11-19T12:59:36.957000", "content": "Congrats! Thanks for sharing!
", "votes": 0, "replies": [] }, { "id": 676397, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2019-11-19T06:15:05.733000", "content": "Congrats Xuan! Thanks for sharing!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "676052": "UPDATE: code is now available [here](https://github.com/naivelamb/kaggle-cloud-organization).\n\nThanks for Max Planck Institute for Meteorology and Kaggle for hosting such an interesting competition. Congrats to all the winners.\n\nThe key in my solution is training two segmentation models: **seg1** trained on all data with BCE loss, and **seg2** trained on non-empty images only with soft DICE loss. I think it works because this competition basically has two tasks: 1) detect the empty images; 2) predict accurate masks for the non-empty images. The two segmentation models address these two tasks respectively. \n## How I come up with this. \nI started the competition with resnet34-FPN using BCE loss (**seg1**). This model achieves ~0.608 on LB and the major contribution comes from capturing the empty mask: it captures ~80% empty masks. I tried a lot to improve the non-empty part, like using combo loss of BCE and DICE, but it is hard to improve the neg-dice (dice score for the empty masks) and pos-dice (dice score for the non-empty makes) simultaneously.\n\nTo predict the non-empty mask accurately, I decided to train 4 individual segmentation models for the non-empty images and then ensemble them together. Since all the train images are non-empty, we can use soft DICE loss directly and the model would focus on predicting accurate masks. I used exactly the same network structure, resnet34-FPN (**seg2**). Then I simply replace all the non-empty predictions from **seg1** model using the predictions from ‘seg2’. Only 1 fold of this 2-stage segmentation pipeline, no TTA, no min-size remover, no classifier, no threshold adjustment (all 0.5) could achieve LB 0.652. After including a resnet34 classifier (0.5 threshold), I got LB 0.655. \n\nLater on, I managed to train all 4 classes in one model by implementing pos-only soft DICE loss. The code looks like:\n\n```python\ndef dice_only_pos(logits, labels, labels_fc):\n # logits -> pixel level predictions\n # labels -> pixel level labels\n # labels_fc -> image/channel level labels\n pos_idx = (labels_fc > 0.5)\n neg_idx = (labels_fc < 0.5)\n loss = SoftDiceLoss()(logits[pos_idx], labels[pos_idx])\n return loss\n```\nThis loss only counts the non-empty channels and ignores all the empty channels.\n\nIn summary the pipeline looks like: \n>1. **seg1**: a multi-label segmentation model trained with BCE loss\n>2. **seg2**: a multi-label segmentation model trained with pos-only soft DICE loss\n>3. **cls**: a multi-label classifier trained with BCE loss. \n\nThe final submission is achieved by the following steps:\n>1. Get predictions using **seg1**\n>2. Replacing the non-empty masks from **seg1** by predictions from **seg2**\n>3. Removing more empty masks using **cls**\n\nBoth pixel-level (segmentation) and image-level (classifier) thresholds are 0.5. \n\n## Baseline results for the 2-stage segmentation\nModel summary:\n>Network: Resnet34-FPN \n>Image size: 384x576\n>Batch size: 16\n>Optimizer: Adam\n>Scheduler: reduceLR for seg1, warmRestart for seg2.\n>Augmentations: H/V flip, ShiftScalerRotate and GridDistortion\n>TTA: raw, Horizontal Flip, Vertical Flip\n\nResults:\n>1-fold: 0.664 \n>5-fold + TTA3: 0.669\n>5-fold + TTA3 + classifier: 0.670. \n\n*TTA1 means only raw images; TTA3 means raw + H/V flip.*\n\nThe rest of my work is just trying different backbones to find the best one. My final models are:\n\n>seg1: densenet121-FPN, TTA1\n>seg2: b7-FPN, TTA3\n>cls: b1, TTA1\n\nResults:\n>1-fold LB: 0.673\n>5-fold LB: 0.6788\n\n## Ensemble\n\nI ensembled multiple seg2 models using major vote. By including 4 models (b5-Unet, InceptionResnetV2-FPN, b7-FPN and b7-Unet), I achieved 0.6792 on LB. \n\n## Pseudo Labeling\nI selected the pseudo labels based a LB 0.6790 submission with the following rules:\n>1. Empty channels with classifier prediction < 0.3\n>2. Non-empty channels with classifier prediction > 0.7\n\nAn image is selected when all the 4 channels satisfy one of the conditions. 835 images are selected. I retrained the b7-FPN and b1-classifier including the pseudo labeling samples, and the final models are:\n>seg1: densenet121-FPN, TTA1\n>seg2: b5-Unet + InceptionResnetV2-FPN + b7-Unet + b7-FPN + b7-FPN-PL, TTA3\n>cls: b1-PL, TTA3\n\n*PL means the model is retrained with pseudo labels*\n\nThis model achieves 0.6794 LB. \n\nOn the last day, I decide to optimize the classifier threshold channel wise to achieve the best local CV, which gives me 0.6805 LB. \n\n## Other things worth mentioning\n1. My CV aligns pretty well with the LB. 1-fold CV = LB +- 0.005. 5-fold CV = LB - (0.010 ~ 0.012). This helps a lot during the model development.\n2. Resizing the image before training could significantly reduce the training time. My resnet34-FPN could finish 1 epoch of training and validation in around 1 mins on a 2080Ti. \n3. For **seg1** and **cls**, complicated networks do not work. This is probably due to the noisy labels. For **seg2**, I cannot make seresnext50 and seresnext101 work and I have no idea why. ", "676104": "> Finally GM& 1st time won prize money! And 3rd place solution.\n\nI have 3x congratulations for you!!! Well done and well deserved.", "676132": "What you did is truly amazing! Congratulations, and thank you for sharing your solution.\n\nIf I understood correctly, we can say that you used **'seg model with pos-only dice loss'** as a base network and used **'seg model with bce loss'** and **'cls model'** to zero-out some masks to get public score of **0.670**. Then you changed backbones of those three models and got **0.6788**.\n\nIt is surprising that just changing backbones leaded to so much improvement of nearly **0.09**. Many of us didn't enjoy significant improvement from changing backbone architectures.\n\nWhat do you think made such an improvement possible?\n\nOne more, you said you used 'soft dice-loss'. I googled it and found [this](https://gist.github.com/jeremyjordan/9ea3032a32909f71dd2ab35fe3bacc08). (which references https://mediatum.ub.tum.de/doc/1395260/1395260.pdf page 72) It squares masks element-wise.\n\n```\ndef soft_dice_loss(y_true, y_pred, epsilon=1e-6): \n axes = tuple(range(1, len(y_pred.shape)-1)) \n numerator = 2. * np.sum(y_pred * y_true, axes)\n denominator = np.sum(np.square(y_pred) + np.square(y_true), axes)\n return 1 - np.mean(numerator / (denominator + epsilon))\n```\n\nDid using it instead of normal dice loss improve your score?\n\nThanks again for your kind writeup.", "676058": "Congratulations. Well deserved. That solo gold is a doozy. \n\nAhh I was so close. I have a variant of your non-empty loss as well. My rationale was that it would focus purely on improving the quality of the mask and less on reducing the false positives. Ended up not moving forward with it because I couldnt get anything out of it initially. Shouldve spent more time on it", "677203": "Congrats!", "677149": "Congratulations, great work and post !", "677106": "Congratz! ", "677008": "Congratulations Xuan on solo Gold and becoming Grand Master. \n\nI like your `def dice_only_pos(logits, labels, labels_fc)` function. I plan to use that in the future. That is key for Kaggle competitions with their discontinuous Dice metric. \n\nI also noticed how different segmentation models would achieve different empty mask prediction accuracies. I even added a metric to my models:\n\n def kaggle_acc(y_true, y_pred0, pix=0.5, area=24000, dim=(384,576)):\n \n # PIXEL THRESHOLD\n y_pred = K.cast( K.greater(y_pred0,pix), K.floatx() )\n \n # MIN AREA THRESHOLD\n s = K.sum(y_pred, axis=(1,2))\n s = K.cast( K.greater(s, area), K.floatx() )\n\n # REMOVE MIN AREA\n s = K.reshape(s,(-1,1))\n s = K.repeat(s,dim[0]*dim[1])\n s = K.reshape(s,(-1,1))\n y_pred = K.permute_dimensions(y_pred,(0,3,1,2))\n y_pred = K.reshape(y_pred,shape=(-1,1))\n y_pred = s*y_pred\n y_pred = K.reshape(y_pred,(-1,y_pred0.shape[3],dim[0],dim[1]))\n y_pred = K.permute_dimensions(y_pred,(0,2,3,1))\n\n # COMPUTE KAGGLE ACC\n total_y_true = K.sum(y_true, axis=(1,2))\n total_y_true = K.cast( K.greater(total_y_true, 0), K.floatx() )\n\n total_y_pred = K.sum(y_pred, axis=(1,2))\n total_y_pred = K.cast( K.greater(total_y_pred, 0), K.floatx() )\n\n return 1 - K.mean( K.abs( total_y_pred - total_y_true ) )\n\nAnd then similar to you, I noticed that some models had low LB score but very high accuracy predicting empty masks. While other models had higher LB score but lower accuracy predicting empty masks. I did something similar to you and converted the low LB high ACC models into classifiers to remove false positives from the high LB low ACC models.", "676988": "Very nice and congratulations! I really like the write up. It is nice to be in the head of the winners sometimes ;-)\n\nThe \"things worth mentioning\" I think are probably very important - especially point 1! Always trust your CV", "676685": "Double congrats @naivelamb ! ", "676660": "Congratulations @naivelamb , great work with your solution. Your GM title is well deserved :)", "676641": "Сongratulations and thanks!\n\nseg1: a multi-label segmentation model trained with BCE loss\nseg2: a multi-label segmentation model trained with pos-only soft DICE loss\ncls: a multi-label classifier trained with BCE loss.\n\nGood idea!", "676612": "Great work, thanks for your write up, and congratulations!\n\nJust out of interest - were you using Kaggle Kernels, or do you have your own GPU?", "676401": "Great write up! Well deserved GM!!", "676288": "Congratulations, good solution:)", "676213": "Congratulations！\nI saw you trained several big models(with B5/B7 backbone), can you tell me how did you set the batchsize when you train your biggest model? ", "676206": "Congratulations😄 😄 ", "676174": "Congrats!!!!! Pseudo labeling part is really impressive!!", "676172": "Hi, Congratulations for getting money and GM!!\n\nI have a question on some part of your solution.\n\nYou mentioned the below sentences.\n\n> To predict the non-empty mask accurately, I decided to train 4 individual segmentation models for the non-empty images and then ensemble them together.\n\nPlease correct my understanding. \n\nSeg 1.\n\nMake a 4-class segmentation model. and get sub1.\n\nSeg 2.\nMake binary segmentation models for 4 each label. (4 segmentation models)\nImages of this competition can have multiple labels, so the same images can be used to train multiple labels. \n\nAnd, replace the predictions in sub1 with the predictions from 4 segmentation models.\n\ncls 1.\nMake a 4-class classifier(finding an empty mask). and remove the false-postive masks in sub1.", "676160": "Congrats Cao. ", "676155": "Congrats on winning and thanks for sharing the solution over to us.", "676148": "Double congratulations @naivelamb and well done. \n\nYour solution is interesting and thanks for sharing. Would you be rreleasing your code?", "676121": "Congratulations :) ", "676061": "Thank you for sharing and congratulations!", "676056": "Congrats! Thanks for sharing, very impressive!", "676053": "Congratulations! And thanks for sharing!!!", "676349": "", "676714": "Congrats! Thanks for sharing!", "676397": "Congrats Xuan! Thanks for sharing!" } }