![\"\"](\"https://raw.githubusercontent.com/Shujun-He/Hubmap-3rd-place-solution/main/expansion_tiles.PNG\")
Congrats to everyone who participated in this competition! It's been a roller coaster of emotions for me. Thanks to the organizers for hosting an interesting competition with meaningful data. I also would like to specially thank @iafoss without his starter notebooks, I would never have been able to do anything here, since I've never worked on segmentation before. BTW, congrats on getting one step closer to gm! It seems like we have both been doing ok in this shakeup business
\nNear the end of the competition, I was getting really tilted by the labeling issues, thinking nothing could be done. But i didn't want to give up either, so I took the path of just doing the simplest things which I think would generalize as well as possible. Here I will discuss the most important details and also release all code used to produce the final result.
\nFull training code: https://github.com/Shujun-He/Hubmap-3rd-place-solution/
\nInference code (forked from @iafoss) https://www.kaggle.com/shujun717/hubmap-3rd-place-inference?scriptVersionId=62198789
We used reduce=2, and sz=1024 in our final ensemble. The notebook we used was originally released by iafoss: https://www.kaggle.com/iafoss/256x256-images
\nWe used cutout in all our training:
\ndef cutout(tensor,alpha=0.5):\n x=int(alpha*tensor.shape[2])\n y=int(alpha*tensor.shape[3])\n center=np.random.randint(0,tensor.shape[2],size=(2))\n #perm = torch.randperm(img.shape[0])\n cut_tensor=tensor.clone()\n cut_tensor[:,:,center[0]-x//2:center[0]+x//2,center[1]-y//2:center[1]+y//2]=0\n return cut_tensor \nIn addition, we used the following albumentation augmentations:
\ndef get_aug(p=1.0):\n return Compose([\n HorizontalFlip(),\n VerticalFlip(),\n RandomRotate90(),\n ShiftScaleRotate(shift_limit=0.0625, scale_limit=0.2, rotate_limit=15, p=0.9,\n border_mode=cv2.BORDER_REFLECT),\n OneOf([\n ElasticTransform(p=.3),\n GaussianBlur(p=.3),\n GaussNoise(p=.3),\n OpticalDistortion(p=0.3),\n GridDistortion(p=.1),\n IAAPiecewiseAffine(p=0.3),\n ], p=0.3),\n OneOf([\n HueSaturationValue(15,25,0),\n CLAHE(clip_limit=2),\n RandomBrightnessContrast(brightness_limit=0.3, contrast_limit=0.3),\n ], p=0.3),\n ], p=p)\nStarting from iafoss's starter notebooks and changing them to pure pytorch with heavier augmentation, I ensembled 2 sets of 5-fold models: one with resnext50 and one with resnext101. Effnets have never worked for me, and here that trend continues. Not much more could be said here tbh.
\nWe used something we called expansion tiles but only during inference, which my teammate jj originally came up with in the PANDA competition. You can see how this is done in our inference notebook and also in the graphic below. The basic idea is to eliminate edge effects. To achieve this, for each tile we run the model on, we expand the tile by a certain number of pixels in all 4 directions, but only use the center region's predictions which do not have edges. Effectively there are no edge effects. For the threshold, I decided to simply use 0.5 to minimize any sort of bias the training data might have.
\n
I wrote this very quickly since I'm eager to share my results/code, so lmk if you have any questions.
", "messages": [ { "id": "1301176", "postDate": "05/11/2021 01:13:57", "content": "Congrats to everyone who participated in this competition! It's been a roller coaster of emotions for me. Thanks to the organizers for hosting an interesting competition with meaningful data. I also would like to specially thank @iafoss without his starter notebooks, I would never have been able to do anything here, since I've never worked on segmentation before. BTW, congrats on getting one step closer to gm! It seems like we have both been doing ok in this shakeup business
\nNear the end of the competition, I was getting really tilted by the labeling issues, thinking nothing could be done. But i didn't want to give up either, so I took the path of just doing the simplest things which I think would generalize as well as possible. Here I will discuss the most important details and also release all code used to produce the final result.
\nFull training code: https://github.com/Shujun-He/Hubmap-3rd-place-solution/
\nInference code (forked from @iafoss) https://www.kaggle.com/shujun717/hubmap-3rd-place-inference?scriptVersionId=62198789
We used reduce=2, and sz=1024 in our final ensemble. The notebook we used was originally released by iafoss: https://www.kaggle.com/iafoss/256x256-images
\nWe used cutout in all our training:
\ndef cutout(tensor,alpha=0.5):\n x=int(alpha*tensor.shape[2])\n y=int(alpha*tensor.shape[3])\n center=np.random.randint(0,tensor.shape[2],size=(2))\n #perm = torch.randperm(img.shape[0])\n cut_tensor=tensor.clone()\n cut_tensor[:,:,center[0]-x//2:center[0]+x//2,center[1]-y//2:center[1]+y//2]=0\n return cut_tensor \nIn addition, we used the following albumentation augmentations:
\ndef get_aug(p=1.0):\n return Compose([\n HorizontalFlip(),\n VerticalFlip(),\n RandomRotate90(),\n ShiftScaleRotate(shift_limit=0.0625, scale_limit=0.2, rotate_limit=15, p=0.9,\n border_mode=cv2.BORDER_REFLECT),\n OneOf([\n ElasticTransform(p=.3),\n GaussianBlur(p=.3),\n GaussNoise(p=.3),\n OpticalDistortion(p=0.3),\n GridDistortion(p=.1),\n IAAPiecewiseAffine(p=0.3),\n ], p=0.3),\n OneOf([\n HueSaturationValue(15,25,0),\n CLAHE(clip_limit=2),\n RandomBrightnessContrast(brightness_limit=0.3, contrast_limit=0.3),\n ], p=0.3),\n ], p=p)\nStarting from iafoss's starter notebooks and changing them to pure pytorch with heavier augmentation, I ensembled 2 sets of 5-fold models: one with resnext50 and one with resnext101. Effnets have never worked for me, and here that trend continues. Not much more could be said here tbh.
\nWe used something we called expansion tiles but only during inference, which my teammate jj originally came up with in the PANDA competition. You can see how this is done in our inference notebook and also in the graphic below. The basic idea is to eliminate edge effects. To achieve this, for each tile we run the model on, we expand the tile by a certain number of pixels in all 4 directions, but only use the center region's predictions which do not have edges. Effectively there are no edge effects. For the threshold, I decided to simply use 0.5 to minimize any sort of bias the training data might have.
\n
I wrote this very quickly since I'm eager to share my results/code, so lmk if you have any questions.
", "rawMarkdown": "Congrats to everyone who participated in this competition! It's been a roller coaster of emotions for me. Thanks to the organizers for hosting an interesting competition with meaningful data. I also would like to specially thank @iafoss without his starter notebooks, I would never have been able to do anything here, since I've never worked on segmentation before. BTW, congrats on getting one step closer to gm! It seems like we have both been doing ok in this shakeup business\n\nNear the end of the competition, I was getting really tilted by the labeling issues, thinking nothing could be done. But i didn't want to give up either, so I took the path of just doing the simplest things which I think would generalize as well as possible. Here I will discuss the most important details and also release all code used to produce the final result. \n\nFull training code: https://github.com/Shujun-He/Hubmap-3rd-place-solution/\nInference code (forked from @iafoss) https://www.kaggle.com/shujun717/hubmap-3rd-place-inference?scriptVersionId=62198789\n\n# Preprocessing\nWe used reduce=2, and sz=1024 in our final ensemble. The notebook we used was originally released by iafoss: https://www.kaggle.com/iafoss/256x256-images\n\n# Augmentation\nWe used cutout in all our training:\n\n```python\ndef cutout(tensor,alpha=0.5):\n x=int(alpha*tensor.shape[2])\n y=int(alpha*tensor.shape[3])\n center=np.random.randint(0,tensor.shape[2],size=(2))\n #perm = torch.randperm(img.shape[0])\n cut_tensor=tensor.clone()\n cut_tensor[:,:,center[0]-x//2:center[0]+x//2,center[1]-y//2:center[1]+y//2]=0\n return cut_tensor \n```\n\nIn addition, we used the following albumentation augmentations:\n\n```python\ndef get_aug(p=1.0):\n return Compose([\n HorizontalFlip(),\n VerticalFlip(),\n RandomRotate90(),\n ShiftScaleRotate(shift_limit=0.0625, scale_limit=0.2, rotate_limit=15, p=0.9,\n border_mode=cv2.BORDER_REFLECT),\n OneOf([\n ElasticTransform(p=.3),\n GaussianBlur(p=.3),\n GaussNoise(p=.3),\n OpticalDistortion(p=0.3),\n GridDistortion(p=.1),\n IAAPiecewiseAffine(p=0.3),\n ], p=0.3),\n OneOf([\n HueSaturationValue(15,25,0),\n CLAHE(clip_limit=2),\n RandomBrightnessContrast(brightness_limit=0.3, contrast_limit=0.3),\n ], p=0.3),\n ], p=p)\n\n```\n\n\n# Models\nStarting from iafoss's starter notebooks and changing them to pure pytorch with heavier augmentation, I ensembled 2 sets of 5-fold models: one with resnext50 and one with resnext101. Effnets have never worked for me, and here that trend continues. Not much more could be said here tbh.\n\n# Inference\nWe used something we called expansion tiles but only during inference, which my teammate jj originally came up with in the PANDA competition. You can see how this is done in our inference notebook and also in the graphic below. The basic idea is to eliminate edge effects. To achieve this, for each tile we run the model on, we expand the tile by a certain number of pixels in all 4 directions, but only use the center region's predictions which do not have edges. Effectively there are no edge effects. For the threshold, I decided to simply use 0.5 to minimize any sort of bias the training data might have.\n\n\n\n\n\nI wrote this very quickly since I'm eager to share my results/code, so lmk if you have any questions.", "votes": null }, { "id": "1301186", "postDate": "05/11/2021 01:23:12", "content": "Congratz man! Tomorrow i will definitely take a look at your code to see if i can learn a trick or two.
\nDid you picked your best private score? Or you got better ones that were left out?
", "rawMarkdown": "Congratz man! Tomorrow i will definitely take a look at your code to see if i can learn a trick or two.\n\nDid you picked your best private score? Or you got better ones that were left out?", "votes": null }, { "id": "1301188", "postDate": "05/11/2021 01:24:36", "content": "Thanks! I picked my best score, which was also my last sub without using hand label or pseudo labels
", "rawMarkdown": "Thanks! I picked my best score, which was also my last sub without using hand label or pseudo labels", "votes": null }, { "id": "1301294", "postDate": "05/11/2021 02:32:55", "content": "@shujun717 Big congratulations! I'm happy that u are doing quite well in competitions.
\nSurprisingly, one of our best submissions in this competition (done a while ago before we did labeling of external data) uses quite similar approach as your setup: ResNeXt50, ASPP [based on my public kernel], 1024 tiles at res/2, ignore 256 boundary pixels around at inference also got 0.950… I guess your result at private LB is rather consistent than a random luck.
Congrats to you as well! You're one gold medal away from GM and I think you will get it soon. As for the private lb result, I'd like to think its not dumb luck as well haha. In fact, by the end of the competition, I saw only two ways how the private lb would turn out. The first one is that the d4 image was more or less inconsistently labeled compared to the private set, which I think turned out to be the case. The second, which imo would be very bizarre, is that the private set score would somehow not differ too much from the public. I have no idea how the second could be true, but it was not impossible. So I chose my subs accordingly based on these two senarios, one sub without training on pl/hl and one with. By design you also have 2 sub selections, so it worked out perfectly. In the end, my 2 subs got scores:
\nno hand/pseudo label: 0.918/0.95
\nwith d4 hand/pseudo labels: 0.943/0.936.
The first one being my best private score out of all subs. On another note, the hand labels did not catastrophically swing the model in a bad direction, which I expected as well because of my previous experiments with hand labels. In short, I found that resnext101 worked significantly better than resnext50, which makes sense because a larger network should be better at dealing with noise and confrontation in the data.
", "rawMarkdown": "Congrats to you as well! You're one gold medal away from GM and I think you will get it soon. As for the private lb result, I'd like to think its not dumb luck as well haha. In fact, by the end of the competition, I saw only two ways how the private lb would turn out. The first one is that the d4 image was more or less inconsistently labeled compared to the private set, which I think turned out to be the case. The second, which imo would be very bizarre, is that the private set score would somehow not differ too much from the public. I have no idea how the second could be true, but it was not impossible. So I chose my subs accordingly based on these two senarios, one sub without training on pl/hl and one with. By design you also have 2 sub selections, so it worked out perfectly. In the end, my 2 subs got scores:\n\nno hand/pseudo label: 0.918/0.95\nwith d4 hand/pseudo labels: 0.943/0.936.\n\nThe first one being my best private score out of all subs. On another note, the hand labels did not catastrophically swing the model in a bad direction, which I expected as well because of my previous experiments with hand labels. In short, I found that resnext101 worked significantly better than resnext50, which makes sense because a larger network should be better at dealing with noise and confrontation in the data.", "votes": null }, { "id": "1301345", "postDate": "05/11/2021 03:18:39", "content": "Thanks. I agree that d488c759a should be rather ignored (though we also had a plan B).
\nQuite interesting observation about ResNeXt101… I do not remember if I tried it. The most surprising thing about this competition is that the EfficientNet worked oO. It worked. It's unbelievable, but it worked for the first time for me… New Nvidia 11 drivers fixed the CUDA issue I previously had, and probably I did the split not in a right way previously, idk. And I'll need to review my old experiments I ran 2 month ago… they got surprisingly high score, and they were rather my first attempts after the competition reset, without almost any checking, rather using my public kernel with tile sampling from RAM and advanced inference. ResNext50+MHSA+r/4 : 0.948/0.915, ResNext50+ASPP+r/4 : 0.947/0.913, ResNext50+ASPP+r/2 :0.950/0.906… There is something in this setup that worked really well. But to be honest I would never select it. Just straight use of PL, though, degraded the performance at private LB.
\"the simplest way is the best way\" really fits it
", "rawMarkdown": "Thanks. I agree that d488c759a should be rather ignored (though we also had a plan B).\nQuite interesting observation about ResNeXt101... I do not remember if I tried it. The most surprising thing about this competition is that the EfficientNet worked oO. It worked. It's unbelievable, but it worked for the first time for me... New Nvidia 11 drivers fixed the CUDA issue I previously had, and probably I did the split not in a right way previously, idk. And I'll need to review my old experiments I ran 2 month ago... they got surprisingly high score, and they were rather my first attempts after the competition reset, without almost any checking, rather using my public kernel with tile sampling from RAM and advanced inference. ResNext50+MHSA+r/4 : 0.948/0.915, ResNext50+ASPP+r/4 : 0.947/0.913, ResNext50+ASPP+r/2 :0.950/0.906... There is something in this setup that worked really well. But to be honest I would never select it. Just straight use of PL, though, degraded the performance at private LB.\n\n\"the simplest way is the best way\" really fits it", "votes": null }, { "id": "1301371", "postDate": "05/11/2021 03:47:12", "content": "It's very surprising effnet worked for you. Honestly I did not try it that much, i think just a few experiments and I just gave up, which is partly also due to the bad impression effnet's it has left on me. idk… there's something about resnexts, the way the res blocks are split are like multiheaded self-attention. The arch is so simple and it just generalizes really really well. I remember you mentioned to me that you though effnet overfitted to imagenet, and I think that's true. I mean, how can it not when you use RL to do architecture search to fit to imagenet? That completely goes against the idea of agnostic model design.
\nAs for the degradation the pl labels cause, @cdeotte talked about it in this post:https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/237999 and it did not affect him that much. I think this depends a lot of the way pl is done during training. For me, it really degraded model performance since I used it without consideration of leakage, but really when you start doing pl, there's always leakage
", "rawMarkdown": "It's very surprising effnet worked for you. Honestly I did not try it that much, i think just a few experiments and I just gave up, which is partly also due to the bad impression effnet's it has left on me. idk... there's something about resnexts, the way the res blocks are split are like multiheaded self-attention. The arch is so simple and it just generalizes really really well. I remember you mentioned to me that you though effnet overfitted to imagenet, and I think that's true. I mean, how can it not when you use RL to do architecture search to fit to imagenet? That completely goes against the idea of agnostic model design.\n\nAs for the degradation the pl labels cause, @cdeotte talked about it in this post:https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/237999 and it did not affect him that much. I think this depends a lot of the way pl is done during training. For me, it really degraded model performance since I used it without consideration of leakage, but really when you start doing pl, there's always leakage", "votes": null }, { "id": "1301387", "postDate": "05/11/2021 04:01:18", "content": "Yes, I agree with this. For last nearly 1-2 years (10+ projects) the combo semi-superwised pretrained ResNeXt + RangerLars optimizer gives insane performance, superfast convergence, and good computational cost. Efficintnet is just overfitted in terms of the architecture… but I may also assume that I just have a bad idea how to train it. This time I used a new madgrad optimizer with it (and also rewritten efficientnet with predefined splits). BTW, u may be also very interested in Swin Transforemer backbone, really cool stuff: segmentation and detection SOTA without architecture search. T version is as small as B1 or ~ResNeXt50-ResNeXt101.
\nRegarding PL, there is a way to make them 100% leak free by considering your folds for PL generation.
Swin transformers look really interesting and the design is quite intuitive as well. It looks more akin to how humans actually perceive than traditional type of CNN. I guess long range dependencies can actually be captured by swin transformers whereas CNNs really cant. I need to give madgrad a try it sounds like, but mostly adam just works well enough for me
\nAnd you're right, leak free PL is possible actually; but I guess I was more talking about the hand labels I used which also included pseudo labels of someone else's model so I couldn't really separate it like that. Overall tho, PL is really risky business sometimes it works well sometimes it doesnt
", "rawMarkdown": "Swin transformers look really interesting and the design is quite intuitive as well. It looks more akin to how humans actually perceive than traditional type of CNN. I guess long range dependencies can actually be captured by swin transformers whereas CNNs really cant. I need to give madgrad a try it sounds like, but mostly adam just works well enough for me\n\nAnd you're right, leak free PL is possible actually; but I guess I was more talking about the hand labels I used which also included pseudo labels of someone else's model so I couldn't really separate it like that. Overall tho, PL is really risky business sometimes it works well sometimes it doesnt", "votes": null }, { "id": "1301428", "postDate": "05/11/2021 04:32:00", "content": "On a side note, i also noticed my local CV was always much higher than LB, and when I changed the validation scheme, so validation was done on full resolution images (by upsampling before evaluating just like how inference is done), my CV was pushing 0.96. That also gave me confidence to just train models without much consideration for other stuff. Reading your solution I think your approach is much more systematic than mine, but I think placings at the top probably came down to luck as 2-4 places have the same score 3 digits past the decimal
", "rawMarkdown": "On a side note, i also noticed my local CV was always much higher than LB, and when I changed the validation scheme, so validation was done on full resolution images (by upsampling before evaluating just like how inference is done), my CV was pushing 0.96. That also gave me confidence to just train models without much consideration for other stuff. Reading your solution I think your approach is much more systematic than mine, but I think placings at the top probably came down to luck as 2-4 places have the same score 3 digits past the decimal", "votes": null }, { "id": "1301437", "postDate": "05/11/2021 04:43:05", "content": "Quite good CV. Unfortunately I didn't validate my very first models at full res(
", "rawMarkdown": "Quite good CV. Unfortunately I didn't validate my very first models at full res(", "votes": null }, { "id": "1301548", "postDate": "05/11/2021 06:17:01", "content": "Congratz on the 3rd place ! Did you get any improvement from using res/2 1024x1024 tiles ?
", "rawMarkdown": "Congratz on the 3rd place ! Did you get any improvement from using res/2 1024x1024 tiles ?", "votes": null }, { "id": "1301674", "postDate": "05/11/2021 07:32:37", "content": "Congrats with great result!
", "rawMarkdown": "Congrats with great result!", "votes": null }, { "id": "1301961", "postDate": "05/11/2021 10:34:35", "content": "Great work, Congratulation for this great solution !
", "rawMarkdown": "Great work, Congratulation for this great solution !", "votes": null }, { "id": "1301986", "postDate": "05/11/2021 10:55:04", "content": "I think your \"expansion tiles\" modification is interesting, in the original u-net paper Ronneberger et al. did the same during training and inference, how big is your performance gain when you only do this during inference?
", "rawMarkdown": "I think your \"expansion tiles\" modification is interesting, in the [original u-net paper](https://arxiv.org/pdf/1505.04597.pdf) Ronneberger et al. did the same during training and inference, how big is your performance gain when you only do this during inference?", "votes": null }, { "id": "1302665", "postDate": "05/11/2021 16:51:37", "content": "Thanks! Congrats on your finish as well! Honestly, I'm not sure how much high resolution helped, when I was doing experiments with PL, resolution (reduce=2 as opposed to 4) helped public lb by 0.001. I did it in the final subs because I have seen many times higher res tends to work better. However, most of the improvements in the private lb came from expansion tiles and a neutral threshold I think. Many public notebooks use low threshold because of the d4 image. A single model with reduce=2,sz=2048, and th=0.4 without expansion tiles only give 0.945 private lb score
", "rawMarkdown": "Thanks! Congrats on your finish as well! Honestly, I'm not sure how much high resolution helped, when I was doing experiments with PL, resolution (reduce=2 as opposed to 4) helped public lb by 0.001. I did it in the final subs because I have seen many times higher res tends to work better. However, most of the improvements in the private lb came from expansion tiles and a neutral threshold I think. Many public notebooks use low threshold because of the d4 image. A single model with reduce=2,sz=2048, and th=0.4 without expansion tiles only give 0.945 private lb score", "votes": null }, { "id": "1302668", "postDate": "05/11/2021 16:52:14", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": null }, { "id": "1302680", "postDate": "05/11/2021 16:55:03", "content": "That's quite interesting! The main reason we used it is because I thought tho iafoss' tile method is very efficient and effective, the edge effect was the only downside I could think of. I'm not sure exactly how much improvement I got. On public lb it was maybe 0.001. But on private probably at least 0.002, so enough to push me from silver to the prize zone.
", "rawMarkdown": "That's quite interesting! The main reason we used it is because I thought tho iafoss' tile method is very efficient and effective, the edge effect was the only downside I could think of. I'm not sure exactly how much improvement I got. On public lb it was maybe 0.001. But on private probably at least 0.002, so enough to push me from silver to the prize zone.", "votes": null }, { "id": "1304656", "postDate": "05/12/2021 18:58:37", "content": "Congratulations. I like your inference trick. I will use that in future competitions to remove edge effects.
", "rawMarkdown": "Congratulations. I like your inference trick. I will use that in future competitions to remove edge effects.", "votes": null }, { "id": "1304851", "postDate": "05/13/2021 00:10:51", "content": "Thanks, I expect this trick will work well for segmentation tasks, since you will need to classify every pixel essentially. It might also be good for object detection, but it wont do much for classification
", "rawMarkdown": "Thanks, I expect this trick will work well for segmentation tasks, since you will need to classify every pixel essentially. It might also be good for object detection, but it wont do much for classification", "votes": null }, { "id": "1304852", "postDate": "05/13/2021 00:11:23", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": null }, { "id": "2301745", "postDate": "06/14/2023 06:27:46", "content": "Can you please guide us a bit descriptively on your inference trick that you mentioned
", "rawMarkdown": "Can you please guide us a bit descriptively on your inference trick that you mentioned", "votes": null } ], "comments": [ { "id": 1301186, "author_name": "victorasso", "author_url": "", "post_date": "05/11/2021 01:23:12", "content": "Congratz man! Tomorrow i will definitely take a look at your code to see if i can learn a trick or two.
\nDid you picked your best private score? Or you got better ones that were left out?
", "votes": null, "replies": [ { "id": 1301188, "author_name": "shujun717", "author_url": "", "post_date": "05/11/2021 01:24:36", "content": "Thanks! I picked my best score, which was also my last sub without using hand label or pseudo labels
", "votes": null, "replies": [] } ] }, { "id": 1301294, "author_name": "iafoss", "author_url": "", "post_date": "05/11/2021 02:32:55", "content": "@shujun717 Big congratulations! I'm happy that u are doing quite well in competitions.
\nSurprisingly, one of our best submissions in this competition (done a while ago before we did labeling of external data) uses quite similar approach as your setup: ResNeXt50, ASPP [based on my public kernel], 1024 tiles at res/2, ignore 256 boundary pixels around at inference also got 0.950… I guess your result at private LB is rather consistent than a random luck.
Congrats to you as well! You're one gold medal away from GM and I think you will get it soon. As for the private lb result, I'd like to think its not dumb luck as well haha. In fact, by the end of the competition, I saw only two ways how the private lb would turn out. The first one is that the d4 image was more or less inconsistently labeled compared to the private set, which I think turned out to be the case. The second, which imo would be very bizarre, is that the private set score would somehow not differ too much from the public. I have no idea how the second could be true, but it was not impossible. So I chose my subs accordingly based on these two senarios, one sub without training on pl/hl and one with. By design you also have 2 sub selections, so it worked out perfectly. In the end, my 2 subs got scores:
\nno hand/pseudo label: 0.918/0.95
\nwith d4 hand/pseudo labels: 0.943/0.936.
The first one being my best private score out of all subs. On another note, the hand labels did not catastrophically swing the model in a bad direction, which I expected as well because of my previous experiments with hand labels. In short, I found that resnext101 worked significantly better than resnext50, which makes sense because a larger network should be better at dealing with noise and confrontation in the data.
", "votes": null, "replies": [] }, { "id": 1301345, "author_name": "iafoss", "author_url": "", "post_date": "05/11/2021 03:18:39", "content": "Thanks. I agree that d488c759a should be rather ignored (though we also had a plan B).
\nQuite interesting observation about ResNeXt101… I do not remember if I tried it. The most surprising thing about this competition is that the EfficientNet worked oO. It worked. It's unbelievable, but it worked for the first time for me… New Nvidia 11 drivers fixed the CUDA issue I previously had, and probably I did the split not in a right way previously, idk. And I'll need to review my old experiments I ran 2 month ago… they got surprisingly high score, and they were rather my first attempts after the competition reset, without almost any checking, rather using my public kernel with tile sampling from RAM and advanced inference. ResNext50+MHSA+r/4 : 0.948/0.915, ResNext50+ASPP+r/4 : 0.947/0.913, ResNext50+ASPP+r/2 :0.950/0.906… There is something in this setup that worked really well. But to be honest I would never select it. Just straight use of PL, though, degraded the performance at private LB.
\"the simplest way is the best way\" really fits it
", "votes": null, "replies": [] }, { "id": 1301371, "author_name": "shujun717", "author_url": "", "post_date": "05/11/2021 03:47:12", "content": "It's very surprising effnet worked for you. Honestly I did not try it that much, i think just a few experiments and I just gave up, which is partly also due to the bad impression effnet's it has left on me. idk… there's something about resnexts, the way the res blocks are split are like multiheaded self-attention. The arch is so simple and it just generalizes really really well. I remember you mentioned to me that you though effnet overfitted to imagenet, and I think that's true. I mean, how can it not when you use RL to do architecture search to fit to imagenet? That completely goes against the idea of agnostic model design.
\nAs for the degradation the pl labels cause, @cdeotte talked about it in this post:https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/237999 and it did not affect him that much. I think this depends a lot of the way pl is done during training. For me, it really degraded model performance since I used it without consideration of leakage, but really when you start doing pl, there's always leakage
", "votes": null, "replies": [] }, { "id": 1301387, "author_name": "iafoss", "author_url": "", "post_date": "05/11/2021 04:01:18", "content": "Yes, I agree with this. For last nearly 1-2 years (10+ projects) the combo semi-superwised pretrained ResNeXt + RangerLars optimizer gives insane performance, superfast convergence, and good computational cost. Efficintnet is just overfitted in terms of the architecture… but I may also assume that I just have a bad idea how to train it. This time I used a new madgrad optimizer with it (and also rewritten efficientnet with predefined splits). BTW, u may be also very interested in Swin Transforemer backbone, really cool stuff: segmentation and detection SOTA without architecture search. T version is as small as B1 or ~ResNeXt50-ResNeXt101.
\nRegarding PL, there is a way to make them 100% leak free by considering your folds for PL generation.
Swin transformers look really interesting and the design is quite intuitive as well. It looks more akin to how humans actually perceive than traditional type of CNN. I guess long range dependencies can actually be captured by swin transformers whereas CNNs really cant. I need to give madgrad a try it sounds like, but mostly adam just works well enough for me
\nAnd you're right, leak free PL is possible actually; but I guess I was more talking about the hand labels I used which also included pseudo labels of someone else's model so I couldn't really separate it like that. Overall tho, PL is really risky business sometimes it works well sometimes it doesnt
", "votes": null, "replies": [] }, { "id": 1301428, "author_name": "shujun717", "author_url": "", "post_date": "05/11/2021 04:32:00", "content": "On a side note, i also noticed my local CV was always much higher than LB, and when I changed the validation scheme, so validation was done on full resolution images (by upsampling before evaluating just like how inference is done), my CV was pushing 0.96. That also gave me confidence to just train models without much consideration for other stuff. Reading your solution I think your approach is much more systematic than mine, but I think placings at the top probably came down to luck as 2-4 places have the same score 3 digits past the decimal
", "votes": null, "replies": [] }, { "id": 1301437, "author_name": "iafoss", "author_url": "", "post_date": "05/11/2021 04:43:05", "content": "Quite good CV. Unfortunately I didn't validate my very first models at full res(
", "votes": null, "replies": [] } ] }, { "id": 1301548, "author_name": "theoviel", "author_url": "", "post_date": "05/11/2021 06:17:01", "content": "Congratz on the 3rd place ! Did you get any improvement from using res/2 1024x1024 tiles ?
", "votes": null, "replies": [ { "id": 1302665, "author_name": "shujun717", "author_url": "", "post_date": "05/11/2021 16:51:37", "content": "Thanks! Congrats on your finish as well! Honestly, I'm not sure how much high resolution helped, when I was doing experiments with PL, resolution (reduce=2 as opposed to 4) helped public lb by 0.001. I did it in the final subs because I have seen many times higher res tends to work better. However, most of the improvements in the private lb came from expansion tiles and a neutral threshold I think. Many public notebooks use low threshold because of the d4 image. A single model with reduce=2,sz=2048, and th=0.4 without expansion tiles only give 0.945 private lb score
", "votes": null, "replies": [] } ] }, { "id": 1301674, "author_name": "aksell7", "author_url": "", "post_date": "05/11/2021 07:32:37", "content": "Congrats with great result!
", "votes": null, "replies": [ { "id": 1302668, "author_name": "shujun717", "author_url": "", "post_date": "05/11/2021 16:52:14", "content": "Thank you!
", "votes": null, "replies": [] } ] }, { "id": 1301961, "author_name": "salimkhazem", "author_url": "", "post_date": "05/11/2021 10:34:35", "content": "Great work, Congratulation for this great solution !
", "votes": null, "replies": [ { "id": 1304852, "author_name": "shujun717", "author_url": "", "post_date": "05/13/2021 00:11:23", "content": "Thank you!
", "votes": null, "replies": [] } ] }, { "id": 1301986, "author_name": "theudas", "author_url": "", "post_date": "05/11/2021 10:55:04", "content": "I think your \"expansion tiles\" modification is interesting, in the original u-net paper Ronneberger et al. did the same during training and inference, how big is your performance gain when you only do this during inference?
", "votes": null, "replies": [ { "id": 1302680, "author_name": "shujun717", "author_url": "", "post_date": "05/11/2021 16:55:03", "content": "That's quite interesting! The main reason we used it is because I thought tho iafoss' tile method is very efficient and effective, the edge effect was the only downside I could think of. I'm not sure exactly how much improvement I got. On public lb it was maybe 0.001. But on private probably at least 0.002, so enough to push me from silver to the prize zone.
", "votes": null, "replies": [] } ] }, { "id": 1304656, "author_name": "cdeotte", "author_url": "", "post_date": "05/12/2021 18:58:37", "content": "Congratulations. I like your inference trick. I will use that in future competitions to remove edge effects.
", "votes": null, "replies": [ { "id": 1304851, "author_name": "shujun717", "author_url": "", "post_date": "05/13/2021 00:10:51", "content": "Thanks, I expect this trick will work well for segmentation tasks, since you will need to classify every pixel essentially. It might also be good for object detection, but it wont do much for classification
", "votes": null, "replies": [] } ] }, { "id": 2301745, "author_name": "bhavesjain", "author_url": "", "post_date": "06/14/2023 06:27:46", "content": "Can you please guide us a bit descriptively on your inference trick that you mentioned
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1301176": "Congrats to everyone who participated in this competition! It's been a roller coaster of emotions for me. Thanks to the organizers for hosting an interesting competition with meaningful data. I also would like to specially thank @iafoss without his starter notebooks, I would never have been able to do anything here, since I've never worked on segmentation before. BTW, congrats on getting one step closer to gm! It seems like we have both been doing ok in this shakeup business\n\nNear the end of the competition, I was getting really tilted by the labeling issues, thinking nothing could be done. But i didn't want to give up either, so I took the path of just doing the simplest things which I think would generalize as well as possible. Here I will discuss the most important details and also release all code used to produce the final result. \n\nFull training code: https://github.com/Shujun-He/Hubmap-3rd-place-solution/\nInference code (forked from @iafoss) https://www.kaggle.com/shujun717/hubmap-3rd-place-inference?scriptVersionId=62198789\n\n# Preprocessing\nWe used reduce=2, and sz=1024 in our final ensemble. The notebook we used was originally released by iafoss: https://www.kaggle.com/iafoss/256x256-images\n\n# Augmentation\nWe used cutout in all our training:\n\n```python\ndef cutout(tensor,alpha=0.5):\n x=int(alpha*tensor.shape[2])\n y=int(alpha*tensor.shape[3])\n center=np.random.randint(0,tensor.shape[2],size=(2))\n #perm = torch.randperm(img.shape[0])\n cut_tensor=tensor.clone()\n cut_tensor[:,:,center[0]-x//2:center[0]+x//2,center[1]-y//2:center[1]+y//2]=0\n return cut_tensor \n```\n\nIn addition, we used the following albumentation augmentations:\n\n```python\ndef get_aug(p=1.0):\n return Compose([\n HorizontalFlip(),\n VerticalFlip(),\n RandomRotate90(),\n ShiftScaleRotate(shift_limit=0.0625, scale_limit=0.2, rotate_limit=15, p=0.9,\n border_mode=cv2.BORDER_REFLECT),\n OneOf([\n ElasticTransform(p=.3),\n GaussianBlur(p=.3),\n GaussNoise(p=.3),\n OpticalDistortion(p=0.3),\n GridDistortion(p=.1),\n IAAPiecewiseAffine(p=0.3),\n ], p=0.3),\n OneOf([\n HueSaturationValue(15,25,0),\n CLAHE(clip_limit=2),\n RandomBrightnessContrast(brightness_limit=0.3, contrast_limit=0.3),\n ], p=0.3),\n ], p=p)\n\n```\n\n\n# Models\nStarting from iafoss's starter notebooks and changing them to pure pytorch with heavier augmentation, I ensembled 2 sets of 5-fold models: one with resnext50 and one with resnext101. Effnets have never worked for me, and here that trend continues. Not much more could be said here tbh.\n\n# Inference\nWe used something we called expansion tiles but only during inference, which my teammate jj originally came up with in the PANDA competition. You can see how this is done in our inference notebook and also in the graphic below. The basic idea is to eliminate edge effects. To achieve this, for each tile we run the model on, we expand the tile by a certain number of pixels in all 4 directions, but only use the center region's predictions which do not have edges. Effectively there are no edge effects. For the threshold, I decided to simply use 0.5 to minimize any sort of bias the training data might have.\n\n\n\n\n\nI wrote this very quickly since I'm eager to share my results/code, so lmk if you have any questions.", "1301186": "Congratz man! Tomorrow i will definitely take a look at your code to see if i can learn a trick or two.\n\nDid you picked your best private score? Or you got better ones that were left out?", "1301188": "Thanks! I picked my best score, which was also my last sub without using hand label or pseudo labels", "1301294": "shujun717 Big congratulations! I'm happy that u are doing quite well in competitions. \nSurprisingly, one of our best submissions in this competition (done a while ago before we did labeling of external data) uses quite similar approach as your setup: ResNeXt50, ASPP [based on my public kernel], 1024 tiles at res/2, ignore 256 boundary pixels around at inference also got 0.950... I guess your result at private LB is rather consistent than a random luck.", "1301314": "Congrats to you as well! You're one gold medal away from GM and I think you will get it soon. As for the private lb result, I'd like to think its not dumb luck as well haha. In fact, by the end of the competition, I saw only two ways how the private lb would turn out. The first one is that the d4 image was more or less inconsistently labeled compared to the private set, which I think turned out to be the case. The second, which imo would be very bizarre, is that the private set score would somehow not differ too much from the public. I have no idea how the second could be true, but it was not impossible. So I chose my subs accordingly based on these two senarios, one sub without training on pl/hl and one with. By design you also have 2 sub selections, so it worked out perfectly. In the end, my 2 subs got scores:\n\nno hand/pseudo label: 0.918/0.95\nwith d4 hand/pseudo labels: 0.943/0.936.\n\nThe first one being my best private score out of all subs. On another note, the hand labels did not catastrophically swing the model in a bad direction, which I expected as well because of my previous experiments with hand labels. In short, I found that resnext101 worked significantly better than resnext50, which makes sense because a larger network should be better at dealing with noise and confrontation in the data.", "1301345": "Thanks. I agree that d488c759a should be rather ignored (though we also had a plan B).\nQuite interesting observation about ResNeXt101... I do not remember if I tried it. The most surprising thing about this competition is that the EfficientNet worked oO. It worked. It's unbelievable, but it worked for the first time for me... New Nvidia 11 drivers fixed the CUDA issue I previously had, and probably I did the split not in a right way previously, idk. And I'll need to review my old experiments I ran 2 month ago... they got surprisingly high score, and they were rather my first attempts after the competition reset, without almost any checking, rather using my public kernel with tile sampling from RAM and advanced inference. ResNext50+MHSA+r/4 : 0.948/0.915, ResNext50+ASPP+r/4 : 0.947/0.913, ResNext50+ASPP+r/2 :0.950/0.906... There is something in this setup that worked really well. But to be honest I would never select it. Just straight use of PL, though, degraded the performance at private LB.\n\n\"the simplest way is the best way\" really fits it", "1301371": "It's very surprising effnet worked for you. Honestly I did not try it that much, i think just a few experiments and I just gave up, which is partly also due to the bad impression effnet's it has left on me. idk... there's something about resnexts, the way the res blocks are split are like multiheaded self-attention. The arch is so simple and it just generalizes really really well. I remember you mentioned to me that you though effnet overfitted to imagenet, and I think that's true. I mean, how can it not when you use RL to do architecture search to fit to imagenet? That completely goes against the idea of agnostic model design.\n\nAs for the degradation the pl labels cause, @cdeotte talked about it in this post:https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/237999 and it did not affect him that much. I think this depends a lot of the way pl is done during training. For me, it really degraded model performance since I used it without consideration of leakage, but really when you start doing pl, there's always leakage", "1301387": "Yes, I agree with this. For last nearly 1-2 years (10+ projects) the combo semi-superwised pretrained ResNeXt + RangerLars optimizer gives insane performance, superfast convergence, and good computational cost. Efficintnet is just overfitted in terms of the architecture... but I may also assume that I just have a bad idea how to train it. This time I used a new madgrad optimizer with it (and also rewritten efficientnet with predefined splits). BTW, u may be also very interested in Swin Transforemer backbone, really cool stuff: segmentation and detection SOTA without architecture search. T version is as small as B1 or ~ResNeXt50-ResNeXt101.\nRegarding PL, there is a way to make them 100% leak free by considering your folds for PL generation.", "1301421": "Swin transformers look really interesting and the design is quite intuitive as well. It looks more akin to how humans actually perceive than traditional type of CNN. I guess long range dependencies can actually be captured by swin transformers whereas CNNs really cant. I need to give madgrad a try it sounds like, but mostly adam just works well enough for me\n\nAnd you're right, leak free PL is possible actually; but I guess I was more talking about the hand labels I used which also included pseudo labels of someone else's model so I couldn't really separate it like that. Overall tho, PL is really risky business sometimes it works well sometimes it doesnt", "1301428": "On a side note, i also noticed my local CV was always much higher than LB, and when I changed the validation scheme, so validation was done on full resolution images (by upsampling before evaluating just like how inference is done), my CV was pushing 0.96. That also gave me confidence to just train models without much consideration for other stuff. Reading your solution I think your approach is much more systematic than mine, but I think placings at the top probably came down to luck as 2-4 places have the same score 3 digits past the decimal", "1301437": "Quite good CV. Unfortunately I didn't validate my very first models at full res(", "1301548": "Congratz on the 3rd place ! Did you get any improvement from using res/2 1024x1024 tiles ?", "1301674": "Congrats with great result!", "1301961": "Great work, Congratulation for this great solution !", "1301986": "I think your \"expansion tiles\" modification is interesting, in the [original u-net paper](https://arxiv.org/pdf/1505.04597.pdf) Ronneberger et al. did the same during training and inference, how big is your performance gain when you only do this during inference?", "1302665": "Thanks! Congrats on your finish as well! Honestly, I'm not sure how much high resolution helped, when I was doing experiments with PL, resolution (reduce=2 as opposed to 4) helped public lb by 0.001. I did it in the final subs because I have seen many times higher res tends to work better. However, most of the improvements in the private lb came from expansion tiles and a neutral threshold I think. Many public notebooks use low threshold because of the d4 image. A single model with reduce=2,sz=2048, and th=0.4 without expansion tiles only give 0.945 private lb score", "1302668": "Thank you!", "1302680": "That's quite interesting! The main reason we used it is because I thought tho iafoss' tile method is very efficient and effective, the edge effect was the only downside I could think of. I'm not sure exactly how much improvement I got. On public lb it was maybe 0.001. But on private probably at least 0.002, so enough to push me from silver to the prize zone.", "1304656": "Congratulations. I like your inference trick. I will use that in future competitions to remove edge effects.", "1304851": "Thanks, I expect this trick will work well for segmentation tasks, since you will need to classify every pixel essentially. It might also be good for object detection, but it wont do much for classification", "1304852": "Thank you!", "2301745": "Can you please guide us a bit descriptively on your inference trick that you mentioned" }, "source": "meta" }