Hey Everyone,
\nThis post is mostly for me, but feel free to add your experiences to the comments!
\nI have to say, this was a really fun competition for me. I began this competition about about a month ago and it's my first time working on a segmentation problem. I'll admit, it was a little intimidating at first, but I really enjoy the satisfying visuals you can create when your model overlaps well with the ground truth. I'm already looking forward to the next segmentation problem Kaggle has for us :). From a competition standpoint it's a little discouraging I haven't been able to match the public notebooks with my own code thus far, but I'm still really happy with everything I've learned from this competition. Thanks to everyone who's been so helpful on these forums! Kaggle is a fun place to spend my free time and I'm always very grateful for the open sharing of ideas between competitors on these forums. Now, to highlight some of the things I've learned.
\n1) How to even begin a semantic segmentation task. While I've taken a deep learning course on Coursera that had a brief general introduction to the topic, this was my first experience actually working on a semantic segmentation problem. In the end, I was pleasantly surprised how similar it is to classification problems, with the added bonus of really satisfying and easily interpreted visuals you can create with the predicted and ground truth masks.
\n2) I became much more familiar with fully convolutional networks and built resnet blocks and squeeze and excite blocks from \"scratch\" to build my u-nets. I did this knowing that I would potentially not place as well, but I'd learn more and that's why I'm here. I read the original papers for ResNet, ResNext, Squeeze and Excitation Networks, as well as ConvNext, which was edifying and actually quite fun. I had hoped to build some Unet++ models, but I didn't have time as I spent a lot of time trying to improve my image preprocessing and augmentation to boost my LB score. I think preprocessing and augmentation are the places where I need to improve the most to start scoring higher on the leaderboard, so I'm very much looking forward to seeing how the winning solutions approached this bit of the competition.
\n3) Custom loss functions: this was relatively new for me with this competition. I've played with implementing custom losses a little before, but I really dove in here for this competition. One thing I'm particularly happy with was my implementation of some of the loss functions in this paper: https://doi.org/10.48550/arXiv.1904.10030. Distance transforms of images were a new concept for me, as were metrics such as dice, surface-dice, and Hausdorff Distance, so implementing different variants of dice loss and incorporating the loss functions in the linked paper was super fun. In the end, combinations of dice loss and the dt_loss described in this paper didn't substantially improve my score for my best models. That said, I was able to train a model that scored equally well as my best dice loss based model by using the dt_loss + binary_crossentropy. Qualitatively, this model had really nice segmentation borders when it detected the vessel, but it struggled to identify small vessels. I'm trying again with a balanced binary crossentropy loss function I'm working on (in an attempt to help it find the smaller vessels) and perhaps that will be my last experiment for the competition.
\n4) How to properly conduct ML experiments. In previous competitions (I've competed in 2 other competitions), as well as the beginning of this one, I'm often scrambling to get a pipeline together so I can understand the task at hand, then try a bunch of model architectures and see what works. In the past, I've also been limited to the compute time available on Kaggle, which further restricts my experimentation. This was my first competition where I had Kaggle's docker image spun up as a container on my local machine from the beginning and could train networks on my RTX 3060. While my 3060 doesn't have as much VRAM as the Kaggle machines (12 gb for my 3060 vs 16 gb for Kaggle notebooks), I was able to spend a lot more time testing things and I was able to learn a lot from training more models (even if they were slower to train or smaller than what I can do on Kaggle). This allowed me to realize that was neglecting the importance of a robust pipeline for systematic experiments. I was pretty surprised to see that I could dramatically change the model parameters (example: u-net starting with 8 filters or u-net starting with 64 filters) and I could even change the model architecture (include SE blocks or don't) and only marginal changes in the quality of the resulting model was observed, if any. However, when I changed the data pipeline with better augmentations or standardizations, I could see much larger improvements (up to a point, I have some work to do to get better at this). In hindsight it's obvious, but this really showed me that the way you present the data to the model is far more important than the model architecture. It's a valuable lesson to learn that tuning the model architecture simply allows you to get the most out of the data as you've represented it, but the representation of the data is what really what makes the biggest difference.
\nThanks to the organizers for hosting this interesting competition and thanks to the Kaggle community for being so engaging and supportive!
", "messages": [ { "id": "2634192", "postDate": "02/03/2024 15:17:42", "content": "Hey Everyone,
\nThis post is mostly for me, but feel free to add your experiences to the comments!
\nI have to say, this was a really fun competition for me. I began this competition about about a month ago and it's my first time working on a segmentation problem. I'll admit, it was a little intimidating at first, but I really enjoy the satisfying visuals you can create when your model overlaps well with the ground truth. I'm already looking forward to the next segmentation problem Kaggle has for us :). From a competition standpoint it's a little discouraging I haven't been able to match the public notebooks with my own code thus far, but I'm still really happy with everything I've learned from this competition. Thanks to everyone who's been so helpful on these forums! Kaggle is a fun place to spend my free time and I'm always very grateful for the open sharing of ideas between competitors on these forums. Now, to highlight some of the things I've learned.
\n1) How to even begin a semantic segmentation task. While I've taken a deep learning course on Coursera that had a brief general introduction to the topic, this was my first experience actually working on a semantic segmentation problem. In the end, I was pleasantly surprised how similar it is to classification problems, with the added bonus of really satisfying and easily interpreted visuals you can create with the predicted and ground truth masks.
\n2) I became much more familiar with fully convolutional networks and built resnet blocks and squeeze and excite blocks from \"scratch\" to build my u-nets. I did this knowing that I would potentially not place as well, but I'd learn more and that's why I'm here. I read the original papers for ResNet, ResNext, Squeeze and Excitation Networks, as well as ConvNext, which was edifying and actually quite fun. I had hoped to build some Unet++ models, but I didn't have time as I spent a lot of time trying to improve my image preprocessing and augmentation to boost my LB score. I think preprocessing and augmentation are the places where I need to improve the most to start scoring higher on the leaderboard, so I'm very much looking forward to seeing how the winning solutions approached this bit of the competition.
\n3) Custom loss functions: this was relatively new for me with this competition. I've played with implementing custom losses a little before, but I really dove in here for this competition. One thing I'm particularly happy with was my implementation of some of the loss functions in this paper: https://doi.org/10.48550/arXiv.1904.10030. Distance transforms of images were a new concept for me, as were metrics such as dice, surface-dice, and Hausdorff Distance, so implementing different variants of dice loss and incorporating the loss functions in the linked paper was super fun. In the end, combinations of dice loss and the dt_loss described in this paper didn't substantially improve my score for my best models. That said, I was able to train a model that scored equally well as my best dice loss based model by using the dt_loss + binary_crossentropy. Qualitatively, this model had really nice segmentation borders when it detected the vessel, but it struggled to identify small vessels. I'm trying again with a balanced binary crossentropy loss function I'm working on (in an attempt to help it find the smaller vessels) and perhaps that will be my last experiment for the competition.
\n4) How to properly conduct ML experiments. In previous competitions (I've competed in 2 other competitions), as well as the beginning of this one, I'm often scrambling to get a pipeline together so I can understand the task at hand, then try a bunch of model architectures and see what works. In the past, I've also been limited to the compute time available on Kaggle, which further restricts my experimentation. This was my first competition where I had Kaggle's docker image spun up as a container on my local machine from the beginning and could train networks on my RTX 3060. While my 3060 doesn't have as much VRAM as the Kaggle machines (12 gb for my 3060 vs 16 gb for Kaggle notebooks), I was able to spend a lot more time testing things and I was able to learn a lot from training more models (even if they were slower to train or smaller than what I can do on Kaggle). This allowed me to realize that was neglecting the importance of a robust pipeline for systematic experiments. I was pretty surprised to see that I could dramatically change the model parameters (example: u-net starting with 8 filters or u-net starting with 64 filters) and I could even change the model architecture (include SE blocks or don't) and only marginal changes in the quality of the resulting model was observed, if any. However, when I changed the data pipeline with better augmentations or standardizations, I could see much larger improvements (up to a point, I have some work to do to get better at this). In hindsight it's obvious, but this really showed me that the way you present the data to the model is far more important than the model architecture. It's a valuable lesson to learn that tuning the model architecture simply allows you to get the most out of the data as you've represented it, but the representation of the data is what really what makes the biggest difference.
\nThanks to the organizers for hosting this interesting competition and thanks to the Kaggle community for being so engaging and supportive!
", "rawMarkdown": "Hey Everyone,\n\nThis post is mostly for me, but feel free to add your experiences to the comments!\n\nI have to say, this was a really fun competition for me. I began this competition about about a month ago and it's my first time working on a segmentation problem. I'll admit, it was a little intimidating at first, but I really enjoy the satisfying visuals you can create when your model overlaps well with the ground truth. I'm already looking forward to the next segmentation problem Kaggle has for us :). From a competition standpoint it's a little discouraging I haven't been able to match the public notebooks with my own code thus far, but I'm still really happy with everything I've learned from this competition. Thanks to everyone who's been so helpful on these forums! Kaggle is a fun place to spend my free time and I'm always very grateful for the open sharing of ideas between competitors on these forums. Now, to highlight some of the things I've learned.\n\n1) How to even begin a semantic segmentation task. While I've taken a deep learning course on Coursera that had a brief general introduction to the topic, this was my first experience actually working on a semantic segmentation problem. In the end, I was pleasantly surprised how similar it is to classification problems, with the added bonus of really satisfying and easily interpreted visuals you can create with the predicted and ground truth masks.\n\n2) I became much more familiar with fully convolutional networks and built resnet blocks and squeeze and excite blocks from \"scratch\" to build my u-nets. I did this knowing that I would potentially not place as well, but I'd learn more and that's why I'm here. I read the original papers for ResNet, ResNext, Squeeze and Excitation Networks, as well as ConvNext, which was edifying and actually quite fun. I had hoped to build some Unet++ models, but I didn't have time as I spent a lot of time trying to improve my image preprocessing and augmentation to boost my LB score. I think preprocessing and augmentation are the places where I need to improve the most to start scoring higher on the leaderboard, so I'm very much looking forward to seeing how the winning solutions approached this bit of the competition.\n\n3) Custom loss functions: this was relatively new for me with this competition. I've played with implementing custom losses a little before, but I really dove in here for this competition. One thing I'm particularly happy with was my implementation of some of the loss functions in this paper: https://doi.org/10.48550/arXiv.1904.10030. Distance transforms of images were a new concept for me, as were metrics such as dice, surface-dice, and Hausdorff Distance, so implementing different variants of dice loss and incorporating the loss functions in the linked paper was super fun. In the end, combinations of dice loss and the dt_loss described in this paper didn't substantially improve my score for my best models. That said, I was able to train a model that scored equally well as my best dice loss based model by using the dt_loss + binary_crossentropy. Qualitatively, this model had really nice segmentation borders when it detected the vessel, but it struggled to identify small vessels. I'm trying again with a balanced binary crossentropy loss function I'm working on (in an attempt to help it find the smaller vessels) and perhaps that will be my last experiment for the competition.\n\n4) How to properly conduct ML experiments. In previous competitions (I've competed in 2 other competitions), as well as the beginning of this one, I'm often scrambling to get a pipeline together so I can understand the task at hand, then try a bunch of model architectures and see what works. In the past, I've also been limited to the compute time available on Kaggle, which further restricts my experimentation. This was my first competition where I had Kaggle's docker image spun up as a container on my local machine from the beginning and could train networks on my RTX 3060. While my 3060 doesn't have as much VRAM as the Kaggle machines (12 gb for my 3060 vs 16 gb for Kaggle notebooks), I was able to spend a lot more time testing things and I was able to learn a lot from training more models (even if they were slower to train or smaller than what I can do on Kaggle). This allowed me to realize that was neglecting the importance of a robust pipeline for systematic experiments. I was pretty surprised to see that I could dramatically change the model parameters (example: u-net starting with 8 filters or u-net starting with 64 filters) and I could even change the model architecture (include SE blocks or don't) and only marginal changes in the quality of the resulting model was observed, if any. However, when I changed the data pipeline with better augmentations or standardizations, I could see much larger improvements (up to a point, I have some work to do to get better at this). In hindsight it's obvious, but this really showed me that the way you present the data to the model is far more important than the model architecture. It's a valuable lesson to learn that tuning the model architecture simply allows you to get the most out of the data as you've represented it, but the representation of the data is what really what makes the biggest difference.\n\nThanks to the organizers for hosting this interesting competition and thanks to the Kaggle community for being so engaging and supportive!", "votes": null }, { "id": "2640470", "postDate": "02/07/2024 00:32:12", "content": "Recapping upon the end of the competition. I ended up benefiting positively from the shake, gaining almost 500 places, but that's not the most interesting thing. The interesting thing is, I didn't pick my best model by a difference of about 0.11! With the pipeline I had set up, I don't think I could have reasonably done a better job of picking models so I'm not upset I didn't pick my best. What's really interesting is I can see how if I had a different evaluation pipeline, I very likely could have! Or at least, I could have picked one of the many better ones. This gets to points 3 and 4 above. Almost all of my notebooks that incorporated the dt_loss function (point 3) ended up scoring better as single models than the models I picked (including ensembles) and there are clear things I could have improved about my training and validation pipeline (point 4) that I believe would have identified this. What an awesome competition, I learned so much and I'm grateful to the organizers for the opportunity to model their data! I can't wait to see how the folks who won medals in this competition approached things :).
", "rawMarkdown": "Recapping upon the end of the competition. I ended up benefiting positively from the shake, gaining almost 500 places, but that's not the most interesting thing. The interesting thing is, I didn't pick my best model by a difference of about 0.11! With the pipeline I had set up, I don't think I could have reasonably done a better job of picking models so I'm not upset I didn't pick my best. What's really interesting is I can see how if I had a different evaluation pipeline, I very likely could have! Or at least, I could have picked one of the many better ones. This gets to points 3 and 4 above. Almost all of my notebooks that incorporated the dt_loss function (point 3) ended up scoring better as single models than the models I picked (including ensembles) and there are clear things I could have improved about my training and validation pipeline (point 4) that I believe would have identified this. What an awesome competition, I learned so much and I'm grateful to the organizers for the opportunity to model their data! I can't wait to see how the folks who won medals in this competition approached things :).", "votes": null } ], "comments": [ { "id": 2640470, "author_name": "chemdatafarmer", "author_url": "", "post_date": "02/07/2024 00:32:12", "content": "Recapping upon the end of the competition. I ended up benefiting positively from the shake, gaining almost 500 places, but that's not the most interesting thing. The interesting thing is, I didn't pick my best model by a difference of about 0.11! With the pipeline I had set up, I don't think I could have reasonably done a better job of picking models so I'm not upset I didn't pick my best. What's really interesting is I can see how if I had a different evaluation pipeline, I very likely could have! Or at least, I could have picked one of the many better ones. This gets to points 3 and 4 above. Almost all of my notebooks that incorporated the dt_loss function (point 3) ended up scoring better as single models than the models I picked (including ensembles) and there are clear things I could have improved about my training and validation pipeline (point 4) that I believe would have identified this. What an awesome competition, I learned so much and I'm grateful to the organizers for the opportunity to model their data! I can't wait to see how the folks who won medals in this competition approached things :).
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2634192": "Hey Everyone,\n\nThis post is mostly for me, but feel free to add your experiences to the comments!\n\nI have to say, this was a really fun competition for me. I began this competition about about a month ago and it's my first time working on a segmentation problem. I'll admit, it was a little intimidating at first, but I really enjoy the satisfying visuals you can create when your model overlaps well with the ground truth. I'm already looking forward to the next segmentation problem Kaggle has for us :). From a competition standpoint it's a little discouraging I haven't been able to match the public notebooks with my own code thus far, but I'm still really happy with everything I've learned from this competition. Thanks to everyone who's been so helpful on these forums! Kaggle is a fun place to spend my free time and I'm always very grateful for the open sharing of ideas between competitors on these forums. Now, to highlight some of the things I've learned.\n\n1) How to even begin a semantic segmentation task. While I've taken a deep learning course on Coursera that had a brief general introduction to the topic, this was my first experience actually working on a semantic segmentation problem. In the end, I was pleasantly surprised how similar it is to classification problems, with the added bonus of really satisfying and easily interpreted visuals you can create with the predicted and ground truth masks.\n\n2) I became much more familiar with fully convolutional networks and built resnet blocks and squeeze and excite blocks from \"scratch\" to build my u-nets. I did this knowing that I would potentially not place as well, but I'd learn more and that's why I'm here. I read the original papers for ResNet, ResNext, Squeeze and Excitation Networks, as well as ConvNext, which was edifying and actually quite fun. I had hoped to build some Unet++ models, but I didn't have time as I spent a lot of time trying to improve my image preprocessing and augmentation to boost my LB score. I think preprocessing and augmentation are the places where I need to improve the most to start scoring higher on the leaderboard, so I'm very much looking forward to seeing how the winning solutions approached this bit of the competition.\n\n3) Custom loss functions: this was relatively new for me with this competition. I've played with implementing custom losses a little before, but I really dove in here for this competition. One thing I'm particularly happy with was my implementation of some of the loss functions in this paper: https://doi.org/10.48550/arXiv.1904.10030. Distance transforms of images were a new concept for me, as were metrics such as dice, surface-dice, and Hausdorff Distance, so implementing different variants of dice loss and incorporating the loss functions in the linked paper was super fun. In the end, combinations of dice loss and the dt_loss described in this paper didn't substantially improve my score for my best models. That said, I was able to train a model that scored equally well as my best dice loss based model by using the dt_loss + binary_crossentropy. Qualitatively, this model had really nice segmentation borders when it detected the vessel, but it struggled to identify small vessels. I'm trying again with a balanced binary crossentropy loss function I'm working on (in an attempt to help it find the smaller vessels) and perhaps that will be my last experiment for the competition.\n\n4) How to properly conduct ML experiments. In previous competitions (I've competed in 2 other competitions), as well as the beginning of this one, I'm often scrambling to get a pipeline together so I can understand the task at hand, then try a bunch of model architectures and see what works. In the past, I've also been limited to the compute time available on Kaggle, which further restricts my experimentation. This was my first competition where I had Kaggle's docker image spun up as a container on my local machine from the beginning and could train networks on my RTX 3060. While my 3060 doesn't have as much VRAM as the Kaggle machines (12 gb for my 3060 vs 16 gb for Kaggle notebooks), I was able to spend a lot more time testing things and I was able to learn a lot from training more models (even if they were slower to train or smaller than what I can do on Kaggle). This allowed me to realize that was neglecting the importance of a robust pipeline for systematic experiments. I was pretty surprised to see that I could dramatically change the model parameters (example: u-net starting with 8 filters or u-net starting with 64 filters) and I could even change the model architecture (include SE blocks or don't) and only marginal changes in the quality of the resulting model was observed, if any. However, when I changed the data pipeline with better augmentations or standardizations, I could see much larger improvements (up to a point, I have some work to do to get better at this). In hindsight it's obvious, but this really showed me that the way you present the data to the model is far more important than the model architecture. It's a valuable lesson to learn that tuning the model architecture simply allows you to get the most out of the data as you've represented it, but the representation of the data is what really what makes the biggest difference.\n\nThanks to the organizers for hosting this interesting competition and thanks to the Kaggle community for being so engaging and supportive!", "2640470": "Recapping upon the end of the competition. I ended up benefiting positively from the shake, gaining almost 500 places, but that's not the most interesting thing. The interesting thing is, I didn't pick my best model by a difference of about 0.11! With the pipeline I had set up, I don't think I could have reasonably done a better job of picking models so I'm not upset I didn't pick my best. What's really interesting is I can see how if I had a different evaluation pipeline, I very likely could have! Or at least, I could have picked one of the many better ones. This gets to points 3 and 4 above. Almost all of my notebooks that incorporated the dt_loss function (point 3) ended up scoring better as single models than the models I picked (including ensembles) and there are clear things I could have improved about my training and validation pipeline (point 4) that I believe would have identified this. What an awesome competition, I learned so much and I'm grateful to the organizers for the opportunity to model their data! I can't wait to see how the folks who won medals in this competition approached things :)." }, "source": "meta" }