![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9735064%2F7cbdc92edfa4491a4405db81528f99ce%2F1.png?generation=1686897867291657&alt=media\")
First of all, we want to thank the hosts who brought us a great opportunity to be a part of resurrecting an ancient library, which is really awesome! And congrats to all winners!
\nOur final solution chosen is a 15-unet-ensemble, which only use ir-CSN as 3d encoder with simple mean pooling bridging to 2d decoder. This idea refers to @samfc10's Notebook: Vesuvius Challenge - 3D ResNet Training. And our training and inference pipeline refer to @tanakar's great scalable sharing: 2.5d segmentaion baseline. Let me explain the details below.
\nAt first we just use the original 3 fragments for our local validation, and we got a very bad score on our first ensemble whose fold 2 scored the worst. So we thought it might caused by less training sample on fold 2.
\nWe split fragment 2 to 3 fragments referring to @junxhuang's comment and @tattaka's discussion.
\n
And we found it interesting that our best public score 0.77 is a single ir-CSN-r50 unet trained on fold 2 with 4 rotation tta which gets 0.66 locally and 0.73 without tta. Of all our experiments, fold 2 always gets high score on public lb comparing to other folds.
\nWe had struggled a long time on how many slices can bring us higher score. We chose slice following @tanakar's baseline, and experimented 8*islices with i select from 1, 2, 3, 4, 6. 24, 32 and 48 slices all gave us robust performance according to lb and cv. After seeing @pavelgonchar's discussion, we stay in 24 slices for most of our experiments.
We spent almost 2/3 time in tuning model on 224x224 cropping size patches, because the smaller the size is the more training samples we can get. And cropping stride 224//2 gave us 13272 samples in total, which is sufficient to train a r50 type unet. The most robust single r50 type model trained on 224x224 cropping size and 224//2 cropping stride is the fold 5 scoring 0.74(with 4 rotation tta) on lb and 0.74 cv.
\nIn the last two weeks, after reading many brilliant winning solutions, e.g. @hesene's 3rd place solution, we decide to use bigger resolution and use multi-size cropping inference as our final submission. Because it is easy to get in mind that big cropping size can get the entire character for models. But there is a problem that the bigger patch we crop the little samples we get. When using a 864x864 size patch with 864//6 , our r152 type unet can easily overfit within 2 epochs. So we make some trade-off there, doing experiments on 500-700 size patch, and these sizes work for us. The best single big resolution model is 576x576 patch size r152 type unet on fold 5, scoring 0.77 locally.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9735064%2F0598a60aad240ecc111899ca897d5975%2F2.png?generation=1686897881870354&alt=media\")
Only unet is used for our segmentation models.
\nIn most of our experiments, we chose ir-CSN-r50 as encoder of our unets. After reading @selimsef's 4th place solution and @nvnnghia's 1st place solution, we stick to openmmlab's great implementation and pretrained weights.
\nThis is our first time participating a segmentation competition. At the very beginning, we spent so much time to build a better decoder, but after reading some solutions, we found that the channels of decoder is the most unconcerned part. From the experiments, light weight decoders perform good enough, so all our unets' decoder channel start from 256. Our decoder code is using selimsef's xview3_solution, and we made some modifications.
\nAt first we gave model with 1 channel voxel. We need to sum the first conv pretrained weights to 1 channel like @samfc10 did.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9735064%2Fe683a7577522d400d937e953a738270f%2F3.png?generation=1686897894991869&alt=media\")
But after reading @pelegshilo's comment and @hengck23's great topic, we thought maybe we can make a full use of pretrained weights of first conv and let model choose the best z location. We simply do concatenation from temporal dimension to channel dimension with 3 overlapped voxels.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9735064%2Fd1c1046526fca219835eaff4e0bdb42e%2F4.png?generation=1686897907062423&alt=media\")
In our experiments, input voxel size (28,224,224) with overlapped start indices [0,2,4] which give 3 24-slice voxels, boosting cv from 0.01 to 0.03 on different folds.
\nWe use 2.5d segmentaion baseline provided augmentations and change the probabilities for different experiment.
We set the ShiftScaleRotate's rotate_limit to 180 referring to @hengck23's topic.
Using mixup and cutmix boost cv differently on each fold and with different mix probabilities. We comment out cutout when use cutmix.
Segmenting on normalized pixels works, referring to @yoyobar's sharing.
We use clean input meaning we filtered out all zero input.
We use adamw optimizer and onecycle scheduler.
We use BCE and hard dice with different weighted average in our experiments.
We use EMA and it boosts cv sometimes.
DDP training and validation is used when training models on big resolution patches.
Gradient accumulation is used when training models on big resolution patches. We use batch size 16 in all our experiments.
Our final solution is a 224-384-576-15-unet ensemble with no tta which scoring 0.76 on public. In this setting we use 28-24 slice selection that means slice file id range from 18 to 46 and copying it to 3 24-slice voxels with slice start index [0,2,4]. For 576 size model we scale up to r152 and the rest are r50. This give us 3rd place on private.
\nWe find that rotation tta do not boost much when add some big resolution models.
\nBut we have 6 different ensemble settings higher than 0.682 private score. The best one scoring 0.687873 on private with 0.749338 on public. In this solution we add 5 more r152 unets with training input size 640 which has 36-32 slice with start index [0,2,4]. We didn't choose it because the low lb and some overfitting performance between train loss and validation loss.
\nHere we list two of them.
| ensemble models | \ninput size | \nnumber of slices | \ncropping stride | \npublic score | \nprivate score | \n
|---|---|---|---|---|---|
| r50,r50,r152 | \n224,384,576 | \n28-24,28-24,28-24 | \n224//8,384//6,576//8 | \n0.763384 | \n0.681137 | \n
| r50,r50,r152,r152 | \n224,384,576,640 | \n28-24,28-24,28-24, 36-32 | \n224//2,384//3,576//4,640//5 | \n0.749338 | \n0.687873 | \n
| r152 | \n576 | \n28-24 | \n576//12 | \n0.720028 | \n0.683273 | \n
We want to thank all those kagglers we mentioned above. We have learned a lot from your sharing, and there is still a lot we need to learn.
\nWe also want to thank the kaggle community, everyone here is free to share their insight that is really helpful for one's growing up.
\nAgain, Thanks to the hosts who bring such great competition! We have seen a lot of brilliant teams with wonderful solution.
\nThis is my teammate @yufujiang's first time participating in a kaggle competition who is also my roommate at college, many thanks to him. And this is my 3rd time competition on kaggle, finally I got my first medal!
\n3rd place inference code
\n3rd place training code repo
First of all, we want to thank the hosts who brought us a great opportunity to be a part of resurrecting an ancient library, which is really awesome! And congrats to all winners!
\nOur final solution chosen is a 15-unet-ensemble, which only use ir-CSN as 3d encoder with simple mean pooling bridging to 2d decoder. This idea refers to @samfc10's Notebook: Vesuvius Challenge - 3D ResNet Training. And our training and inference pipeline refer to @tanakar's great scalable sharing: 2.5d segmentaion baseline. Let me explain the details below.
\nAt first we just use the original 3 fragments for our local validation, and we got a very bad score on our first ensemble whose fold 2 scored the worst. So we thought it might caused by less training sample on fold 2.
\nWe split fragment 2 to 3 fragments referring to @junxhuang's comment and @tattaka's discussion.
\n
And we found it interesting that our best public score 0.77 is a single ir-CSN-r50 unet trained on fold 2 with 4 rotation tta which gets 0.66 locally and 0.73 without tta. Of all our experiments, fold 2 always gets high score on public lb comparing to other folds.
\nWe had struggled a long time on how many slices can bring us higher score. We chose slice following @tanakar's baseline, and experimented 8*islices with i select from 1, 2, 3, 4, 6. 24, 32 and 48 slices all gave us robust performance according to lb and cv. After seeing @pavelgonchar's discussion, we stay in 24 slices for most of our experiments.
We spent almost 2/3 time in tuning model on 224x224 cropping size patches, because the smaller the size is the more training samples we can get. And cropping stride 224//2 gave us 13272 samples in total, which is sufficient to train a r50 type unet. The most robust single r50 type model trained on 224x224 cropping size and 224//2 cropping stride is the fold 5 scoring 0.74(with 4 rotation tta) on lb and 0.74 cv.
\nIn the last two weeks, after reading many brilliant winning solutions, e.g. @hesene's 3rd place solution, we decide to use bigger resolution and use multi-size cropping inference as our final submission. Because it is easy to get in mind that big cropping size can get the entire character for models. But there is a problem that the bigger patch we crop the little samples we get. When using a 864x864 size patch with 864//6 , our r152 type unet can easily overfit within 2 epochs. So we make some trade-off there, doing experiments on 500-700 size patch, and these sizes work for us. The best single big resolution model is 576x576 patch size r152 type unet on fold 5, scoring 0.77 locally.
\n
Only unet is used for our segmentation models.
\nIn most of our experiments, we chose ir-CSN-r50 as encoder of our unets. After reading @selimsef's 4th place solution and @nvnnghia's 1st place solution, we stick to openmmlab's great implementation and pretrained weights.
\nThis is our first time participating a segmentation competition. At the very beginning, we spent so much time to build a better decoder, but after reading some solutions, we found that the channels of decoder is the most unconcerned part. From the experiments, light weight decoders perform good enough, so all our unets' decoder channel start from 256. Our decoder code is using selimsef's xview3_solution, and we made some modifications.
\nAt first we gave model with 1 channel voxel. We need to sum the first conv pretrained weights to 1 channel like @samfc10 did.
\n
But after reading @pelegshilo's comment and @hengck23's great topic, we thought maybe we can make a full use of pretrained weights of first conv and let model choose the best z location. We simply do concatenation from temporal dimension to channel dimension with 3 overlapped voxels.
\n
In our experiments, input voxel size (28,224,224) with overlapped start indices [0,2,4] which give 3 24-slice voxels, boosting cv from 0.01 to 0.03 on different folds.
\nWe use 2.5d segmentaion baseline provided augmentations and change the probabilities for different experiment.
We set the ShiftScaleRotate's rotate_limit to 180 referring to @hengck23's topic.
Using mixup and cutmix boost cv differently on each fold and with different mix probabilities. We comment out cutout when use cutmix.
Segmenting on normalized pixels works, referring to @yoyobar's sharing.
We use clean input meaning we filtered out all zero input.
We use adamw optimizer and onecycle scheduler.
We use BCE and hard dice with different weighted average in our experiments.
We use EMA and it boosts cv sometimes.
DDP training and validation is used when training models on big resolution patches.
Gradient accumulation is used when training models on big resolution patches. We use batch size 16 in all our experiments.
Our final solution is a 224-384-576-15-unet ensemble with no tta which scoring 0.76 on public. In this setting we use 28-24 slice selection that means slice file id range from 18 to 46 and copying it to 3 24-slice voxels with slice start index [0,2,4]. For 576 size model we scale up to r152 and the rest are r50. This give us 3rd place on private.
\nWe find that rotation tta do not boost much when add some big resolution models.
\nBut we have 6 different ensemble settings higher than 0.682 private score. The best one scoring 0.687873 on private with 0.749338 on public. In this solution we add 5 more r152 unets with training input size 640 which has 36-32 slice with start index [0,2,4]. We didn't choose it because the low lb and some overfitting performance between train loss and validation loss.
\nHere we list two of them.
| ensemble models | \ninput size | \nnumber of slices | \ncropping stride | \npublic score | \nprivate score | \n
|---|---|---|---|---|---|
| r50,r50,r152 | \n224,384,576 | \n28-24,28-24,28-24 | \n224//8,384//6,576//8 | \n0.763384 | \n0.681137 | \n
| r50,r50,r152,r152 | \n224,384,576,640 | \n28-24,28-24,28-24, 36-32 | \n224//2,384//3,576//4,640//5 | \n0.749338 | \n0.687873 | \n
| r152 | \n576 | \n28-24 | \n576//12 | \n0.720028 | \n0.683273 | \n
We want to thank all those kagglers we mentioned above. We have learned a lot from your sharing, and there is still a lot we need to learn.
\nWe also want to thank the kaggle community, everyone here is free to share their insight that is really helpful for one's growing up.
\nAgain, Thanks to the hosts who bring such great competition! We have seen a lot of brilliant teams with wonderful solution.
\nThis is my teammate @yufujiang's first time participating in a kaggle competition who is also my roommate at college, many thanks to him. And this is my 3rd time competition on kaggle, finally I got my first medal!
\n3rd place inference code
\n3rd place training code repo
Congratulations! Extensive and meticulous work, this gold medal is well-deserved!
", "rawMarkdown": "Congratulations! Extensive and meticulous work, this gold medal is well-deserved!", "votes": null }, { "id": "2311746", "postDate": "06/21/2023 12:32:37", "content": "Congratulations! I'm a journalist with Scientific American. I'd love to talk about your approach. Could we arrange a video call? email: tomasjweber@gmail.com. Thanks!
", "rawMarkdown": "Congratulations! I'm a journalist with Scientific American. I'd love to talk about your approach. Could we arrange a video call? email: tomasjweber@gmail.com. Thanks!", "votes": null }, { "id": "2325604", "postDate": "07/01/2023 13:14:48", "content": "Our training code is released here.
\nAnd we release our checkpoints for inference notebook.
Congratulations! Extensive and meticulous work, this gold medal is well-deserved!
", "votes": null, "replies": [] }, { "id": 2311746, "author_name": "tomasjweber", "author_url": "", "post_date": "06/21/2023 12:32:37", "content": "Congratulations! I'm a journalist with Scientific American. I'd love to talk about your approach. Could we arrange a video call? email: tomasjweber@gmail.com. Thanks!
", "votes": null, "replies": [] }, { "id": 2325604, "author_name": "traptinblur", "author_url": "", "post_date": "07/01/2023 13:14:48", "content": "Our training code is released here.
\nAnd we release our checkpoints for inference notebook.