![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1829450%2Fb8053a93d2452d3417e3bfe0100ea953%2Fcutmix.png?generation=1574165517927124&alt=media\")
First I would like to thank Max Planck Institute and Kaggle for hosting this interesting competition.
\n\nI would like to share some of the key points of my 3rd place (silver) solution :-) It sounds cool right? (well I love to make that top silver medal the most out of it, forgive me :P)
\n\nNaturally thinking, cutmix is the best way to deal with this competition. We can cut a part of this image and paste to another image. This idea came off from my mind without knowing its existence academically, which I later found an official paper about it.\n
How to do that in code?\nI search for some augmentation package, but find it hard to flexibly code it my way. So I decided to do it manually.\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1829450%2Fbc80b1cb83404da2281afe27ec2f7f72%2Fcutmixcode.png?generation=1574165662592266&alt=media\")
\nThe different of doing cutmix or not is just an extra section of __getitem__ in data generator. Here, indexes_augment is the random indexes pick from the training data, w_cutmix and h_cutmix are width and height of the crop. So I just get the random starting position of width and height in the original drawn image (X) and insert part of other image (Xc) into it.
Cutmix boosted both LB and CV by 0.004.
\n\nPseudo-label only works if we correctly select good samples, as well as the correct number of samples. I did this by assessing the quality of each predicted validation image by calculating:\nquality = (number of pixels with probability > top) + w*(number of pixels with probability < bot). Here, top can take values from [.7, .75, .8, .85, .9], bot can take values from [.1, .15, .2, .25, .3], and w is the weight of low-value pixels as compared to high-value pixels, which can be taken from, say, [.1, .5, 1, 2, 10].
I get the quality of all validation data, rank it, and select nb_samples most confident samples from it, and see the score. I search through a full set of validation data and had a result something like this\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1829450%2Fbdb7de2de3781797e891e7eb8ec93508%2Fpseudo.png?generation=1574166277986726&alt=media\")
So, I can manually decide bot, top, w, and nb_samples as long as nb_samples are reasonable with the corresponding score. For example, bot=.1, top=.7, w=1, and nb_samples=1000 (with corresponding dice=0.77xx), which means the most 1000 confident predictions out of 5546 train images can have that good dice. Then I can pick up the same ratio of images from test predictions, which is (1000/5546*3698).
Pseudo labelling boosted around 0.003 on both CV and LB.
\n\nFirst, I did a test on private LB, based on this topic.
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1528571%2F43771b2c3fd35e39047f55b552604852%2FLB_probing.png?generation=1569149948996763&alt=media\")
From that probing result, we need to make 1 assumption:\n\"Train set and full test set should have the same distribution of classes.\"
\n\nThen, by submitting each class as empty (others as 1-pixel masks), we can know the percentage of each class in the public LB. Then the assumption I make will allow us to know the percentage of each class in the private LB. The result is:
\n\nTrain data: Fish 49.85%, Flower 57.35%, Gravel 47.00%, Sugar 32.36%.\nPrivate test data: Fish 49.94%, Flower 56.76%, Gravel 47.20%, Sugar 31.30%.
\n\nAs you can see, the distributions of private test and train very similar, allowing me to completely trust CV. Therefore during the whole competition, I never probed LB by submissions, but only stick with full k-fold to search for post-processing parameters. This is important, as it guides the way we do everyday in the competition. And you can see that I jumped on private LB, and I also selected my possibly best submission.
\n\nFinally, I still would like to emphasize again that late sharing should not be encouraged. I have a bad thought that whenever I see the excessive sharers around in future competitions, I would be very disappointed, and discouraged from competing. In other words I am somehow \"scared\" of their existence.
\n\nThanks for reading!
", "messages": [ { "id": 676697, "postDate": "2019-11-19T12:42:07.210Z", "content": "First I would like to thank Max Planck Institute and Kaggle for hosting this interesting competition.
\n\nI would like to share some of the key points of my 3rd place (silver) solution :-) It sounds cool right? (well I love to make that top silver medal the most out of it, forgive me :P)
\n\nNaturally thinking, cutmix is the best way to deal with this competition. We can cut a part of this image and paste to another image. This idea came off from my mind without knowing its existence academically, which I later found an official paper about it.\n
How to do that in code?\nI search for some augmentation package, but find it hard to flexibly code it my way. So I decided to do it manually.\n\nThe different of doing cutmix or not is just an extra section of __getitem__ in data generator. Here, indexes_augment is the random indexes pick from the training data, w_cutmix and h_cutmix are width and height of the crop. So I just get the random starting position of width and height in the original drawn image (X) and insert part of other image (Xc) into it.
Cutmix boosted both LB and CV by 0.004.
\n\nPseudo-label only works if we correctly select good samples, as well as the correct number of samples. I did this by assessing the quality of each predicted validation image by calculating:\nquality = (number of pixels with probability > top) + w*(number of pixels with probability < bot). Here, top can take values from [.7, .75, .8, .85, .9], bot can take values from [.1, .15, .2, .25, .3], and w is the weight of low-value pixels as compared to high-value pixels, which can be taken from, say, [.1, .5, 1, 2, 10].
I get the quality of all validation data, rank it, and select nb_samples most confident samples from it, and see the score. I search through a full set of validation data and had a result something like this\n
So, I can manually decide bot, top, w, and nb_samples as long as nb_samples are reasonable with the corresponding score. For example, bot=.1, top=.7, w=1, and nb_samples=1000 (with corresponding dice=0.77xx), which means the most 1000 confident predictions out of 5546 train images can have that good dice. Then I can pick up the same ratio of images from test predictions, which is (1000/5546*3698).
Pseudo labelling boosted around 0.003 on both CV and LB.
\n\nFirst, I did a test on private LB, based on this topic.
\n\n
From that probing result, we need to make 1 assumption:\n\"Train set and full test set should have the same distribution of classes.\"
\n\nThen, by submitting each class as empty (others as 1-pixel masks), we can know the percentage of each class in the public LB. Then the assumption I make will allow us to know the percentage of each class in the private LB. The result is:
\n\nTrain data: Fish 49.85%, Flower 57.35%, Gravel 47.00%, Sugar 32.36%.\nPrivate test data: Fish 49.94%, Flower 56.76%, Gravel 47.20%, Sugar 31.30%.
\n\nAs you can see, the distributions of private test and train very similar, allowing me to completely trust CV. Therefore during the whole competition, I never probed LB by submissions, but only stick with full k-fold to search for post-processing parameters. This is important, as it guides the way we do everyday in the competition. And you can see that I jumped on private LB, and I also selected my possibly best submission.
\n\nFinally, I still would like to emphasize again that late sharing should not be encouraged. I have a bad thought that whenever I see the excessive sharers around in future competitions, I would be very disappointed, and discouraged from competing. In other words I am somehow \"scared\" of their existence.
\n\nThanks for reading!
", "rawMarkdown": "First I would like to thank Max Planck Institute and Kaggle for hosting this interesting competition.\n\nI would like to share some of the key points of my 3rd place (silver) solution :-) It sounds cool right? (well I love to make that top silver medal the most out of it, forgive me :P)\n\n### 1) Cutmix augmentation\nNaturally thinking, cutmix is the best way to deal with this competition. We can cut a part of this image and paste to another image. This idea came off from my mind without knowing its existence academically, which I later found an official paper about it.\n\n\nHow to do that in code?\nI search for some augmentation package, but find it hard to flexibly code it my way. So I decided to do it manually.\n\nThe different of doing cutmix or not is just an extra section of `__getitem__` in data generator. Here, `indexes_augment` is the random indexes pick from the training data, `w_cutmix` and `h_cutmix` are width and height of the crop. So I just get the random starting position of width and height in the original drawn image (`X`) and insert part of other image (`Xc`) into it. \n\n**Cutmix boosted both LB and CV by 0.004.**\n\n### 2) Pseudo-label\nPseudo-label only works if we correctly select good samples, as well as the correct number of samples. I did this by assessing the `quality` of each predicted validation image by calculating:\n`quality = (number of pixels with probability > top) + w*(number of pixels with probability < bot)`. Here, `top` can take values from [.7, .75, .8, .85, .9], `bot` can take values from [.1, .15, .2, .25, .3], and `w` is the weight of low-value pixels as compared to high-value pixels, which can be taken from, say, [.1, .5, 1, 2, 10]. \n\nI get the `quality` of all validation data, rank it, and select `nb_samples` most confident samples from it, and see the score. I search through a full set of validation data and had a result something like this\n\n\nSo, I can manually decide `bot`, `top`, `w`, and `nb_samples` as long as `nb_samples` are reasonable with the corresponding score. For example, `bot=.1`, `top=.7`, `w=1`, and `nb_samples`=1000 (with corresponding `dice=0.77xx`), which means the most 1000 confident predictions out of 5546 train images can have that good dice. Then I can pick up the same ratio of images from test predictions, which is (1000/5546*3698).\n\n**Pseudo labelling boosted around 0.003 on both CV and LB.**\n\n### 3) Estimating private LB distribution and decide to trust CV\nFirst, I did a test on private LB, based on [this topic](https://www.kaggle.com/c/understanding_cloud_organization/discussion/109793#latest-631950). \n\n\n\nFrom that probing result, we need to make 1 assumption:\n\"Train set and full test set should have the same distribution of classes.\"\n\nThen, by submitting each class as empty (others as 1-pixel masks), we can know the percentage of each class in the public LB. Then the assumption I make will allow us to know the percentage of each class in the private LB. The result is:\n\n\nTrain data: Fish 49.85%, Flower 57.35%, Gravel 47.00%, Sugar 32.36%.\nPrivate test data: Fish 49.94%, Flower 56.76%, Gravel 47.20%, Sugar 31.30%.\n\nAs you can see, the distributions of private test and train very similar, allowing me to completely trust CV. Therefore during the whole competition, I never probed LB by submissions, but only stick with full k-fold to search for post-processing parameters. **This is important, as it guides the way we do everyday in the competition**. And you can see that I jumped on private LB, and I also selected my possibly best submission.\n\nFinally, I still would like to emphasize again that late sharing should not be encouraged. I have a bad thought that whenever I see the excessive sharers around in future competitions, I would be very disappointed, and discouraged from competing. In other words I am somehow \"scared\" of their existence. \n\nThanks for reading!", "votes": 26 }, { "id": 678494, "postDate": "2019-11-21T13:19:39.840Z", "content": "congrats !!
", "rawMarkdown": "congrats !!", "votes": 3 }, { "id": 677270, "postDate": "2019-11-20T01:33:58.817Z", "content": "Congrats and thank you for sharing the cutmix augmentation technique. I will try it in the next competition.
", "rawMarkdown": "Congrats and thank you for sharing the cutmix augmentation technique. I will try it in the next competition. ", "votes": 1 }, { "id": 677016, "postDate": "2019-11-19T17:59:45.810Z", "content": "Congrats KhaVo. Wow, CutMix augmentation is cool. And your method for selecting Pseudo labeling is ingenious. I plan to use both in the future. Thanks for sharing.
\n\nI saw that you jumped 20 spots on the last day. Was it one of these two tricks?
", "rawMarkdown": "Congrats KhaVo. Wow, CutMix augmentation is cool. And your method for selecting Pseudo labeling is ingenious. I plan to use both in the future. Thanks for sharing.\n\nI saw that you jumped 20 spots on the last day. Was it one of these two tricks? ", "votes": 1, "replies": [ { "id": 677297, "postDate": "2019-11-20T02:31:23.430Z", "content": "Thanks Chris. The jump was just from ensembling, which I did only in the last 2 days. Before jumping, my single models scored around 0.669 public LB, after ensemble public 0.674 LB.
", "rawMarkdown": "Thanks Chris. The jump was just from ensembling, which I did only in the last 2 days. Before jumping, my single models scored around 0.669 public LB, after ensemble public 0.674 LB." } ] }, { "id": 676852, "postDate": "2019-11-19T14:47:34.663Z", "content": "Congratulations\nGreat Write-Up\nThank you for Sharing your Insights & Approach! @khahuras
", "rawMarkdown": "Congratulations\nGreat Write-Up\nThank you for Sharing your Insights & Approach! @khahuras ", "votes": 1 }, { "id": 676707, "postDate": "2019-11-19T12:52:10.423Z", "content": "Thank you for your writeup. Cutmix always looked promising and I am glad it worked for you. \nI'm curious about your models and loss functions
", "rawMarkdown": "Thank you for your writeup. Cutmix always looked promising and I am glad it worked for you. \nI'm curious about your models and loss functions", "votes": 1, "replies": [ { "id": 676710, "postDate": "2019-11-19T12:55:24.043Z", "content": "My models are not special, since I am completely new to computer vision and segmentation in particular. What worked is that I trained a bunch of different encoders, pseudo-label thresholds, then combine around 10 models, search for best postprocessing parameters through CV. That's simply it.
", "rawMarkdown": "My models are not special, since I am completely new to computer vision and segmentation in particular. What worked is that I trained a bunch of different encoders, pseudo-label thresholds, then combine around 10 models, search for best postprocessing parameters through CV. That's simply it.", "votes": 1 } ] }, { "id": 677081, "postDate": "2019-11-19T19:14:23.660Z", "content": "Hey glad to see cutmix helped you even we tried this type of augmentation in our initial training but model took a lot of time to converge so we dropped it did it happened with you too?
", "rawMarkdown": "Hey glad to see cutmix helped you even we tried this type of augmentation in our initial training but model took a lot of time to converge so we dropped it did it happened with you too?", "votes": 2, "replies": [ { "id": 677294, "postDate": "2019-11-20T02:30:25.733Z", "content": "It needs around 20 epochs I think
", "rawMarkdown": "It needs around 20 epochs I think", "votes": 1 }, { "id": 677303, "postDate": "2019-11-20T02:43:54.943Z", "content": "I think then we dropped the idea too early
", "rawMarkdown": "I think then we dropped the idea too early" } ] }, { "id": 680911, "postDate": "2019-11-25T11:45:16.067Z", "rawMarkdown": "", "isDeleted": true }, { "id": 678492, "postDate": "2019-11-21T13:19:13.990Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 678494, "author_name": "ravi tanwar", "author_url": "", "post_date": "2019-11-21T13:19:39.840000", "content": "congrats !!
", "votes": 3, "replies": [] }, { "id": 677270, "author_name": "sabari nathan", "author_url": "", "post_date": "2019-11-20T01:33:58.817000", "content": "Congrats and thank you for sharing the cutmix augmentation technique. I will try it in the next competition.
", "votes": 1, "replies": [] }, { "id": 677016, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-11-19T17:59:45.810000", "content": "Congrats KhaVo. Wow, CutMix augmentation is cool. And your method for selecting Pseudo labeling is ingenious. I plan to use both in the future. Thanks for sharing.
\n\nI saw that you jumped 20 spots on the last day. Was it one of these two tricks?
", "votes": 1, "replies": [ { "id": 677297, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-11-20T02:31:23.430000", "content": "Thanks Chris. The jump was just from ensembling, which I did only in the last 2 days. Before jumping, my single models scored around 0.669 public LB, after ensemble public 0.674 LB.
", "votes": 0, "replies": [] } ] }, { "id": 676852, "author_name": "Ailurophile", "author_url": "", "post_date": "2019-11-19T14:47:34.663000", "content": "Congratulations\nGreat Write-Up\nThank you for Sharing your Insights & Approach! @khahuras
", "votes": 1, "replies": [] }, { "id": 676707, "author_name": "Bibek", "author_url": "", "post_date": "2019-11-19T12:52:10.423000", "content": "Thank you for your writeup. Cutmix always looked promising and I am glad it worked for you. \nI'm curious about your models and loss functions
", "votes": 1, "replies": [ { "id": 676710, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-11-19T12:55:24.043000", "content": "My models are not special, since I am completely new to computer vision and segmentation in particular. What worked is that I trained a bunch of different encoders, pseudo-label thresholds, then combine around 10 models, search for best postprocessing parameters through CV. That's simply it.
", "votes": 1, "replies": [] } ] }, { "id": 677081, "author_name": "Udbhav Bamba", "author_url": "", "post_date": "2019-11-19T19:14:23.660000", "content": "Hey glad to see cutmix helped you even we tried this type of augmentation in our initial training but model took a lot of time to converge so we dropped it did it happened with you too?
", "votes": 2, "replies": [ { "id": 677294, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-11-20T02:30:25.733000", "content": "It needs around 20 epochs I think
", "votes": 1, "replies": [] }, { "id": 677303, "author_name": "Udbhav Bamba", "author_url": "", "post_date": "2019-11-20T02:43:54.943000", "content": "I think then we dropped the idea too early
", "votes": 0, "replies": [] } ] }, { "id": 680911, "author_name": "", "author_url": "", "post_date": "2019-11-25T11:45:16.067000", "content": "", "votes": 0, "replies": [] }, { "id": 678492, "author_name": "", "author_url": "", "post_date": "2019-11-21T13:19:13.990000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "676697": "First I would like to thank Max Planck Institute and Kaggle for hosting this interesting competition.\n\nI would like to share some of the key points of my 3rd place (silver) solution :-) It sounds cool right? (well I love to make that top silver medal the most out of it, forgive me :P)\n\n### 1) Cutmix augmentation\nNaturally thinking, cutmix is the best way to deal with this competition. We can cut a part of this image and paste to another image. This idea came off from my mind without knowing its existence academically, which I later found an official paper about it.\n\n\nHow to do that in code?\nI search for some augmentation package, but find it hard to flexibly code it my way. So I decided to do it manually.\n\nThe different of doing cutmix or not is just an extra section of `__getitem__` in data generator. Here, `indexes_augment` is the random indexes pick from the training data, `w_cutmix` and `h_cutmix` are width and height of the crop. So I just get the random starting position of width and height in the original drawn image (`X`) and insert part of other image (`Xc`) into it. \n\n**Cutmix boosted both LB and CV by 0.004.**\n\n### 2) Pseudo-label\nPseudo-label only works if we correctly select good samples, as well as the correct number of samples. I did this by assessing the `quality` of each predicted validation image by calculating:\n`quality = (number of pixels with probability > top) + w*(number of pixels with probability < bot)`. Here, `top` can take values from [.7, .75, .8, .85, .9], `bot` can take values from [.1, .15, .2, .25, .3], and `w` is the weight of low-value pixels as compared to high-value pixels, which can be taken from, say, [.1, .5, 1, 2, 10]. \n\nI get the `quality` of all validation data, rank it, and select `nb_samples` most confident samples from it, and see the score. I search through a full set of validation data and had a result something like this\n\n\nSo, I can manually decide `bot`, `top`, `w`, and `nb_samples` as long as `nb_samples` are reasonable with the corresponding score. For example, `bot=.1`, `top=.7`, `w=1`, and `nb_samples`=1000 (with corresponding `dice=0.77xx`), which means the most 1000 confident predictions out of 5546 train images can have that good dice. Then I can pick up the same ratio of images from test predictions, which is (1000/5546*3698).\n\n**Pseudo labelling boosted around 0.003 on both CV and LB.**\n\n### 3) Estimating private LB distribution and decide to trust CV\nFirst, I did a test on private LB, based on [this topic](https://www.kaggle.com/c/understanding_cloud_organization/discussion/109793#latest-631950). \n\n\n\nFrom that probing result, we need to make 1 assumption:\n\"Train set and full test set should have the same distribution of classes.\"\n\nThen, by submitting each class as empty (others as 1-pixel masks), we can know the percentage of each class in the public LB. Then the assumption I make will allow us to know the percentage of each class in the private LB. The result is:\n\n\nTrain data: Fish 49.85%, Flower 57.35%, Gravel 47.00%, Sugar 32.36%.\nPrivate test data: Fish 49.94%, Flower 56.76%, Gravel 47.20%, Sugar 31.30%.\n\nAs you can see, the distributions of private test and train very similar, allowing me to completely trust CV. Therefore during the whole competition, I never probed LB by submissions, but only stick with full k-fold to search for post-processing parameters. **This is important, as it guides the way we do everyday in the competition**. And you can see that I jumped on private LB, and I also selected my possibly best submission.\n\nFinally, I still would like to emphasize again that late sharing should not be encouraged. I have a bad thought that whenever I see the excessive sharers around in future competitions, I would be very disappointed, and discouraged from competing. In other words I am somehow \"scared\" of their existence. \n\nThanks for reading!", "678494": "congrats !!", "677270": "Congrats and thank you for sharing the cutmix augmentation technique. I will try it in the next competition. ", "677016": "Congrats KhaVo. Wow, CutMix augmentation is cool. And your method for selecting Pseudo labeling is ingenious. I plan to use both in the future. Thanks for sharing.\n\nI saw that you jumped 20 spots on the last day. Was it one of these two tricks? ", "676852": "Congratulations\nGreat Write-Up\nThank you for Sharing your Insights & Approach! @khahuras ", "676707": "Thank you for your writeup. Cutmix always looked promising and I am glad it worked for you. \nI'm curious about your models and loss functions", "677081": "Hey glad to see cutmix helped you even we tried this type of augmentation in our initial training but model took a lot of time to converge so we dropped it did it happened with you too?", "680911": "", "678492": "" } }