\n
![\"\"](\"https://f.easyuploader.app/20210511095415_66516e4b.jpg\")
Congratulations to the winners and all participants who finished this competition.
\nA number of problems arose during the competition, and the competition was held for six months. I have to admit that it was quite a challenge for me to keep participating until the end.
\nMy ranking was not so great, but I would like to share it with you.
The resolution is not very good, so you may not be able to see the fine details, but the model I created is shown in the following figure.
\n
I would like to explain the main points.
\n1. Cutting out patches
\nEarly in the competition, I recognized through my experiments that the difference in image size (original size to be cut out and image size after downscaling) might affect the score. Therefore, I chose several image sizes that were large enough to ensure a sufficient batch size for training, and eventually ensembled 2 models for 352 x 4 and 512 x 2 image sizes.
\nAlso, I did not include the black and white regions, which are not very informative regions, in the training patches, referring to @iafoss kernel.
\nIf we simply shifted the image by augmentation for the patch, we would have to do some kind of interpolation such as reflection for the missing regions. Since my experimental results showed that this was not a good idea, I shifted the image by a quarter of the patch size when cutting out the patches from the WSIs.
2. Preprocessing / Augmentation
\nThe images were simply normalized by 255, and then I trained the models on 5 folds, where each fold consists of 3 WSIs.
\nI don't know about the private WSIs, but at least some of the public images were darker than the training WSIs, or had some kind of dirt on them, so I applied a very strong augmentation as shown below.
As a result, I think that the evaluation of the model with 5 folds by WSI and this strong augmentation contributed to the stable correlation between local CV and LB.
\n3. Training
\nMy training was very simple.
\nI used BCE + Adam to train ImageNet Pretrained EfficientNetB3 backbone U-Net. As mentioned in some discussions, the complexity of the model did not seem to have any effect, probably because it is a rather simple task. I also tried some losses such as Focal loss and Combo loss, but there was no noticeable gain, probably because I excluded the black and white parts beforehand. In addition, I don't use losses based on discrete metrics such as Dice loss for ensembles, so I thought that BCE would be sufficient for this project when there was no gain from focal loss.
\nAlso, since I used the psuedo label in the latter half like many of the participants, I trained the image size combinations (group A and group B in the model pipeline figure) alternately to avoid overtraining. I used the psuedo label of the model trained in group A for training group B to avoid the model from becoming overly dependent on the public data. However, since the psuedo label was almost meaningless in private score, I think it was almost a meaningless attempt.
\n(I used the d48 hand label at the end of the competition, but I did not do any hand labeling by myself. I think that the d48 hand label actually contributed nothing to the improvement of the private score.)
The following graph shows the correlation between the local CV and the public LB, and since the public LB is in increments of 0.001, I think the correlation is actually greater than what can be seen in the graph. There are three clusters, starting from the bottom one: no psuedo at all, psuedo without d48 hand label, and psuedo based on d48 hand label.
\n![\"\"](\"https://f.easyuploader.app/20210511104838_62546152.jpg\")
4-5. Prediction + Tiling predicted patches
\nI think the prediction was the same method as many of you.
In my experiments, I found that there was no significant difference in the scores if the overlap was 1/16 or more, so I decided to use that value to save inference time.
\nThe only regret I have is that I set the threshold in the direction of better public score. As I will show later, if I had set the threshold to the theoretical value determined by the train data, I would have been sure to get the gold medal.
6. Submission
\nd48... image)Finally…
\nI think this competition was about how much you can believe in your Local CV. If there is a domain shift between the training data and the test data, the trust CV is not always correct, so I don't think the participants who did the training with more public are wrong. It would be nice if the host would mention the presence or absence of domain shift at least a little in advance, but there is a degree to which this is the case, and as long as the number of data is finite, it may be a permanent problem in data science.
By the way, my sbumission in descending order of private score was as follows. It would have been better if I had used the correct threshold value, but it was a good lesson for me that I didn't trust all my locals.
\n![\"\"](\"https://f.easyuploader.app/20210511105807_7a4f4f78.jpg\")
See you at the next competition!
", "messages": [ { "id": "1301260", "postDate": "05/11/2021 02:04:58", "content": "Congratulations to the winners and all participants who finished this competition.
\nA number of problems arose during the competition, and the competition was held for six months. I have to admit that it was quite a challenge for me to keep participating until the end.
\nMy ranking was not so great, but I would like to share it with you.
The resolution is not very good, so you may not be able to see the fine details, but the model I created is shown in the following figure.
\n
I would like to explain the main points.
\n1. Cutting out patches
\nEarly in the competition, I recognized through my experiments that the difference in image size (original size to be cut out and image size after downscaling) might affect the score. Therefore, I chose several image sizes that were large enough to ensure a sufficient batch size for training, and eventually ensembled 2 models for 352 x 4 and 512 x 2 image sizes.
\nAlso, I did not include the black and white regions, which are not very informative regions, in the training patches, referring to @iafoss kernel.
\nIf we simply shifted the image by augmentation for the patch, we would have to do some kind of interpolation such as reflection for the missing regions. Since my experimental results showed that this was not a good idea, I shifted the image by a quarter of the patch size when cutting out the patches from the WSIs.
2. Preprocessing / Augmentation
\nThe images were simply normalized by 255, and then I trained the models on 5 folds, where each fold consists of 3 WSIs.
\nI don't know about the private WSIs, but at least some of the public images were darker than the training WSIs, or had some kind of dirt on them, so I applied a very strong augmentation as shown below.
As a result, I think that the evaluation of the model with 5 folds by WSI and this strong augmentation contributed to the stable correlation between local CV and LB.
\n3. Training
\nMy training was very simple.
\nI used BCE + Adam to train ImageNet Pretrained EfficientNetB3 backbone U-Net. As mentioned in some discussions, the complexity of the model did not seem to have any effect, probably because it is a rather simple task. I also tried some losses such as Focal loss and Combo loss, but there was no noticeable gain, probably because I excluded the black and white parts beforehand. In addition, I don't use losses based on discrete metrics such as Dice loss for ensembles, so I thought that BCE would be sufficient for this project when there was no gain from focal loss.
\nAlso, since I used the psuedo label in the latter half like many of the participants, I trained the image size combinations (group A and group B in the model pipeline figure) alternately to avoid overtraining. I used the psuedo label of the model trained in group A for training group B to avoid the model from becoming overly dependent on the public data. However, since the psuedo label was almost meaningless in private score, I think it was almost a meaningless attempt.
\n(I used the d48 hand label at the end of the competition, but I did not do any hand labeling by myself. I think that the d48 hand label actually contributed nothing to the improvement of the private score.)
The following graph shows the correlation between the local CV and the public LB, and since the public LB is in increments of 0.001, I think the correlation is actually greater than what can be seen in the graph. There are three clusters, starting from the bottom one: no psuedo at all, psuedo without d48 hand label, and psuedo based on d48 hand label.
\n
4-5. Prediction + Tiling predicted patches
\nI think the prediction was the same method as many of you.
In my experiments, I found that there was no significant difference in the scores if the overlap was 1/16 or more, so I decided to use that value to save inference time.
\nThe only regret I have is that I set the threshold in the direction of better public score. As I will show later, if I had set the threshold to the theoretical value determined by the train data, I would have been sure to get the gold medal.
6. Submission
\nd48... image)Finally…
\nI think this competition was about how much you can believe in your Local CV. If there is a domain shift between the training data and the test data, the trust CV is not always correct, so I don't think the participants who did the training with more public are wrong. It would be nice if the host would mention the presence or absence of domain shift at least a little in advance, but there is a degree to which this is the case, and as long as the number of data is finite, it may be a permanent problem in data science.
By the way, my sbumission in descending order of private score was as follows. It would have been better if I had used the correct threshold value, but it was a good lesson for me that I didn't trust all my locals.
\n
See you at the next competition!
", "rawMarkdown": "Congratulations to the winners and all participants who finished this competition.\nA number of problems arose during the competition, and the competition was held for six months. I have to admit that it was quite a challenge for me to keep participating until the end.\nMy ranking was not so great, but I would like to share it with you.\n \n---\n \nThe resolution is not very good, so you may not be able to see the fine details, but the model I created is shown in the following figure.\n\n\nI would like to explain the main points.\n\n**1. Cutting out patches**\nEarly in the competition, I recognized through my experiments that the difference in image size (original size to be cut out and image size after downscaling) might affect the score. Therefore, I chose several image sizes that were large enough to ensure a sufficient batch size for training, and eventually ensembled 2 models for 352 x 4 and 512 x 2 image sizes. \nAlso, I did not include the black and white regions, which are not very informative regions, in the training patches, referring to @iafoss kernel. \nIf we simply shifted the image by augmentation for the patch, we would have to do some kind of interpolation such as reflection for the missing regions. Since my experimental results showed that this was not a good idea, I shifted the image by a quarter of the patch size when cutting out the patches from the WSIs.\n\n**2. Preprocessing / Augmentation**\nThe images were simply normalized by 255, and then I trained the models on 5 folds, where each fold consists of 3 WSIs.\nI don't know about the private WSIs, but at least some of the public images were darker than the training WSIs, or had some kind of dirt on them, so I applied a very strong augmentation as shown below.\n- Horizontal/Vertical Flip (p=0.5)\n- RandomRotate90 (p=0.5)\n- Rotate (-40/+40, reflect101, p=0.5)\n- ShiftScaleRotate (scale: -0.2/0.1, p=0.5)\n- OneOf (p=0.5)\n\tRandomBrightnessContrast (B: 0.5, C: 0.1)\n\tHueSaturationValue (H: +-20, S: +-100, V: +-80)\n- One of (p=0.5)\n\tCutout (holes: 100, size: 1/64, white RGB)\n\tGaussianNoise\n- One of (p=0.5)\n\tElasticTransform\n\tGridDistortion (steps: 5, limit: 0.3)\n\tOpticalDistortion (distort: 0.5, shift: 0.0)\n\nAs a result, I think that the evaluation of the model with 5 folds by WSI and this strong augmentation contributed to the stable correlation between local CV and LB.\n\n**3. Training**\nMy training was very simple. \nI used BCE + Adam to train ImageNet Pretrained EfficientNetB3 backbone U-Net. As mentioned in some discussions, the complexity of the model did not seem to have any effect, probably because it is a rather simple task. I also tried some losses such as Focal loss and Combo loss, but there was no noticeable gain, probably because I excluded the black and white parts beforehand. In addition, I don't use losses based on discrete metrics such as Dice loss for ensembles, so I thought that BCE would be sufficient for this project when there was no gain from focal loss. \nAlso, since I used the psuedo label in the latter half like many of the participants, I trained the image size combinations (group A and group B in the model pipeline figure) alternately to avoid overtraining. I used the psuedo label of the model trained in group A for training group B to avoid the model from becoming overly dependent on the public data. However, since the psuedo label was almost meaningless in private score, I think it was almost a meaningless attempt.\n(I used the d48 hand label at the end of the competition, but I did not do any hand labeling by myself. I think that the d48 hand label actually contributed nothing to the improvement of the private score.)\n \nThe following graph shows the correlation between the local CV and the public LB, and since the public LB is in increments of 0.001, I think the correlation is actually greater than what can be seen in the graph. There are three clusters, starting from the bottom one: no psuedo at all, psuedo without d48 hand label, and psuedo based on d48 hand label.\n\n\n**4-5. Prediction + Tiling predicted patches**\nI think the prediction was the same method as many of you.\n\n- TTA: Raw, HorizontalFlip, VerticalFlip\n- Overlapping by 1/16 image size for tiling\n- Naive averaging for ensemble (folds and models)\n- Threshold optimization with train images (th: 0.40 for public LB, 0.44-0.48 for local CV)\n\nIn my experiments, I found that there was no significant difference in the scores if the overlap was 1/16 or more, so I decided to use that value to save inference time.\nThe only regret I have is that I set the threshold in the direction of better public score. As I will show later, if I had set the threshold to the theoretical value determined by the train data, I would have been sure to get the gold medal.\n\n**6. Submission**\n- Approximately 8 hours running (5 folds, 3 TTA, 2 models)\n- Good correlation between local CV and public LB\n\t(Alghough there are deviations..., maybe due to the notorious `d48...` image)\n- Local CV: 0.9429\n\tPublic LB: 0.927\n\tPrivate LB: 0.947\n\n**Finally...**\nI think this competition was about how much you can believe in your Local CV. If there is a domain shift between the training data and the test data, the `trust CV` is not always correct, so I don't think the participants who did the training with more public are wrong. It would be nice if the host would mention the presence or absence of domain shift at least a little in advance, but there is a degree to which this is the case, and as long as the number of data is finite, it may be a permanent problem in data science.\n\nBy the way, my sbumission in descending order of private score was as follows. It would have been better if I had used the correct threshold value, but it was a good lesson for me that I didn't trust all my locals.\n\n\nSee you at the next competition!", "votes": null }, { "id": "1301270", "postDate": "05/11/2021 02:12:38", "content": "@maxwell110 Congratulations and Thanks for sharing the approach
", "rawMarkdown": "maxwell110 Congratulations and Thanks for sharing the approach", "votes": null }, { "id": "1301422", "postDate": "05/11/2021 04:27:19", "content": "Congrats on 17th place and thanks for sharing details solution @maxwell110
", "rawMarkdown": "Congrats on 17th place and thanks for sharing details solution @maxwell110", "votes": null }, { "id": "1301514", "postDate": "05/11/2021 05:48:25", "content": "Congratulation for getting to the 17th place. And this is very informative, thanks for sharing your work.
", "rawMarkdown": "Congratulation for getting to the 17th place. And this is very informative, thanks for sharing your work.", "votes": null } ], "comments": [ { "id": 1301270, "author_name": "usharengaraju", "author_url": "", "post_date": "05/11/2021 02:12:38", "content": "@maxwell110 Congratulations and Thanks for sharing the approach
", "votes": null, "replies": [] }, { "id": 1301422, "author_name": "duykhanh99", "author_url": "", "post_date": "05/11/2021 04:27:19", "content": "Congrats on 17th place and thanks for sharing details solution @maxwell110
", "votes": null, "replies": [] }, { "id": 1301514, "author_name": "gssdatamanager", "author_url": "", "post_date": "05/11/2021 05:48:25", "content": "Congratulation for getting to the 17th place. And this is very informative, thanks for sharing your work.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1301260": "Congratulations to the winners and all participants who finished this competition.\nA number of problems arose during the competition, and the competition was held for six months. I have to admit that it was quite a challenge for me to keep participating until the end.\nMy ranking was not so great, but I would like to share it with you.\n \n---\n \nThe resolution is not very good, so you may not be able to see the fine details, but the model I created is shown in the following figure.\n\n\nI would like to explain the main points.\n\n**1. Cutting out patches**\nEarly in the competition, I recognized through my experiments that the difference in image size (original size to be cut out and image size after downscaling) might affect the score. Therefore, I chose several image sizes that were large enough to ensure a sufficient batch size for training, and eventually ensembled 2 models for 352 x 4 and 512 x 2 image sizes. \nAlso, I did not include the black and white regions, which are not very informative regions, in the training patches, referring to @iafoss kernel. \nIf we simply shifted the image by augmentation for the patch, we would have to do some kind of interpolation such as reflection for the missing regions. Since my experimental results showed that this was not a good idea, I shifted the image by a quarter of the patch size when cutting out the patches from the WSIs.\n\n**2. Preprocessing / Augmentation**\nThe images were simply normalized by 255, and then I trained the models on 5 folds, where each fold consists of 3 WSIs.\nI don't know about the private WSIs, but at least some of the public images were darker than the training WSIs, or had some kind of dirt on them, so I applied a very strong augmentation as shown below.\n- Horizontal/Vertical Flip (p=0.5)\n- RandomRotate90 (p=0.5)\n- Rotate (-40/+40, reflect101, p=0.5)\n- ShiftScaleRotate (scale: -0.2/0.1, p=0.5)\n- OneOf (p=0.5)\n\tRandomBrightnessContrast (B: 0.5, C: 0.1)\n\tHueSaturationValue (H: +-20, S: +-100, V: +-80)\n- One of (p=0.5)\n\tCutout (holes: 100, size: 1/64, white RGB)\n\tGaussianNoise\n- One of (p=0.5)\n\tElasticTransform\n\tGridDistortion (steps: 5, limit: 0.3)\n\tOpticalDistortion (distort: 0.5, shift: 0.0)\n\nAs a result, I think that the evaluation of the model with 5 folds by WSI and this strong augmentation contributed to the stable correlation between local CV and LB.\n\n**3. Training**\nMy training was very simple. \nI used BCE + Adam to train ImageNet Pretrained EfficientNetB3 backbone U-Net. As mentioned in some discussions, the complexity of the model did not seem to have any effect, probably because it is a rather simple task. I also tried some losses such as Focal loss and Combo loss, but there was no noticeable gain, probably because I excluded the black and white parts beforehand. In addition, I don't use losses based on discrete metrics such as Dice loss for ensembles, so I thought that BCE would be sufficient for this project when there was no gain from focal loss. \nAlso, since I used the psuedo label in the latter half like many of the participants, I trained the image size combinations (group A and group B in the model pipeline figure) alternately to avoid overtraining. I used the psuedo label of the model trained in group A for training group B to avoid the model from becoming overly dependent on the public data. However, since the psuedo label was almost meaningless in private score, I think it was almost a meaningless attempt.\n(I used the d48 hand label at the end of the competition, but I did not do any hand labeling by myself. I think that the d48 hand label actually contributed nothing to the improvement of the private score.)\n \nThe following graph shows the correlation between the local CV and the public LB, and since the public LB is in increments of 0.001, I think the correlation is actually greater than what can be seen in the graph. There are three clusters, starting from the bottom one: no psuedo at all, psuedo without d48 hand label, and psuedo based on d48 hand label.\n\n\n**4-5. Prediction + Tiling predicted patches**\nI think the prediction was the same method as many of you.\n\n- TTA: Raw, HorizontalFlip, VerticalFlip\n- Overlapping by 1/16 image size for tiling\n- Naive averaging for ensemble (folds and models)\n- Threshold optimization with train images (th: 0.40 for public LB, 0.44-0.48 for local CV)\n\nIn my experiments, I found that there was no significant difference in the scores if the overlap was 1/16 or more, so I decided to use that value to save inference time.\nThe only regret I have is that I set the threshold in the direction of better public score. As I will show later, if I had set the threshold to the theoretical value determined by the train data, I would have been sure to get the gold medal.\n\n**6. Submission**\n- Approximately 8 hours running (5 folds, 3 TTA, 2 models)\n- Good correlation between local CV and public LB\n\t(Alghough there are deviations..., maybe due to the notorious `d48...` image)\n- Local CV: 0.9429\n\tPublic LB: 0.927\n\tPrivate LB: 0.947\n\n**Finally...**\nI think this competition was about how much you can believe in your Local CV. If there is a domain shift between the training data and the test data, the `trust CV` is not always correct, so I don't think the participants who did the training with more public are wrong. It would be nice if the host would mention the presence or absence of domain shift at least a little in advance, but there is a degree to which this is the case, and as long as the number of data is finite, it may be a permanent problem in data science.\n\nBy the way, my sbumission in descending order of private score was as follows. It would have been better if I had used the correct threshold value, but it was a good lesson for me that I didn't trust all my locals.\n\n\nSee you at the next competition!", "1301270": "maxwell110 Congratulations and Thanks for sharing the approach", "1301422": "Congrats on 17th place and thanks for sharing details solution @maxwell110", "1301514": "Congratulation for getting to the 17th place. And this is very informative, thanks for sharing your work." }, "source": "meta" }