![\"\"](\"https://i.imgur.com/zhEn4Ow.png\")
60th Place Solution
\nThank you Kaggle and RANZCR for hosting this competition. It was the first competition I really immersed myself in and I have learned a tremendous amount. Thank you to all the competitors, especially those that so openly shared their results and techniques. Kaggle is a wonderful medium through which to learn data science, and it would not be the same without everyone sharing and communicating.
\nI am not pleased with my final standing, but I believe that all Kagglers should share their procedure, irrespective of their final position, because it is highly unlikely that anyone else did exactly what you did, so there is always something to learn from reading other's write ups. Here it goes.
\nImage resolutions
\nMy approach was a brute force approach and not particularly elegant. I began with @yasufuminakama’s kernels on 3 stage training with annotations and changed several parameters like learning rate, batch size, training augmentations, and optimizer (AdamP showed better CV results than Adam). I also generated stratified folds by coupling @sin's kernel with some useful information about incorrect labels / inverted images shared by @raddar and others. Once I had established a CV strategy, I started to experiment with different image resolutions, all with a ResNet200D. I tried image sizes from 512 to 736 in increments of 16 (all with a batch size of 32). I saw an increase in AUC + decrease in BCE loss each time I bumped up the image resolution. I validated all models against my first fold to save time (more on that later).
There was some discussion as to whether or not this multi stage approach with teacher-student training on annotated images actually gave any performance boost. I continued to use it anyways for one reason: you can use the stage 2 weights to lower training time. I trained stages 1 and 2 on a lower image resolution and then trained many different stage 3 models of various image resolutions from these weights. In other words, instead of repeating all 3 stages anytime I changed image resolutions / other parameters, I simply reused stage 2 weights and skipped to stage 3. This is not ideal, but it saved me a ton of training time and allowed me to run more experiments. I figured it would work seeing as we frequently fine tune models that have been trained on ImageNet with different image resolutions than the ones they were initially trained on.
\nPreprocessing
\nI did minimal preprocessing. It was mentioned that some images have black borders around them. I removed them during inference and saw an increase in score. I hesitated to remove them from training as these black boxes could serve as a form of regularization, but decided to remove them for the last couple models I trained. I figured if they helped with inference, they would help with training, but I did not run any real experiments to verify this.
\nForward Selection
\nI then created a fold prediction correlation heatmap for these different image sizes and saw a surprisingly low correlation between the ResNet200D model when trained on different image sizes (0.85 - 0.91). This showed me that one could get good model diversity by simply changing image resolutions, which got me thinking about ensembling. I remembered reading @chrisdeotte’s kernel on forward selection from the Melanoma competition and decided to try it. I saw nice improvements when doing so and decided to experiment with different model architectures as well. I trained a SEResNet152D on 672 and 720 image resolutions and checked the fold prediction correlations between them with my previous ResNet200D models: the correlation was even lower, so I threw them into forward selection with my other ResNet200D models.
The above procedure got me to CV 0.966 and public leaderboard 0.968. I then experimented with RegNetYs, EfficientNets, and ResNeSts. RegNetYs took too long to train and EfficientNets were never able to reach the same CV score as the ResNet200D / SEResNet152D. Only the ResNeSt’s got similar scores, but simply took too long to train. I had reached a bit of an impasse and was not sure how to proceed.
Pseudo labels
\nI then saw this post on duplicate images between the NIH Chest XRays dataset and the training / public test images given to use in this competition. Seeing as I already had an ensemble of models with good diversity, I was confident I could generate fairly high quality pseudo labels from this external dataset. (Note that you must generate pseudo labels from models that have not seen the fold you are validating against otherwise you get subtle CV leakage). I first tried to remove duplicates using CNN embeddings and RAPIDS NearestNeighbors, but this approach still left around 500 duplicate images, so I combined this approach with ImageHash to remove the remaining duplicates. I then started generating pseudo labels. I decided to use soft pseudo labels where at least one catheter class is present (any column with prediction above 0.7) and all other classes without any catheters (remaining columns with predictions below 0.3). I added Gaussian noise to the pseudo labels in an attempt to reduce confirmation bias.
Instead of re-training the model jointly with concatenated labeled + pseudo labels, I decided to introduce another training stage in which I train on only NIH pseudo labels. The final stage is then fine tuning on labeled training data. I also decided to train the last few epochs without any data augmentation, based on another discussion post, and saw further improvements. I carried out this procedure for my top-3 highest scoring models and threw them back into my forward selection algorithm to compare CV and public leaderboard score. It indeed increased.
\nI am still not sure if my CV was entirely leak free. I think many teams that used external data had some form of leakage. For this reason, I only included 2 NIH pseudo label pretrained models in the folds I forward selected against. Had I invested more time in examining these duplicate images found by RAPIDS NN and ImageHash, I think I would have had a more robust CV and better public / private score.
\nTraining / validation folds
\nUp to this point, I had only validated against the first fold, so in some sense I was ‘overfitting’ my own validation set. This is not ideal; it would be much better to validate each model against all folds and use the full OOF to ensemble, as opposed to single fold predictions. That being said, I joined this competition fairly late so I needed to save time somehow while simultaneously running many experiments. I also trusted my fold splits and was not overly concerned with validating against only one fold. However, I did want my final predictions to encompass all training data, so I repeated what I did on the first fold with the second fold, so that my models (in total) had been trained on all data folds.
\nAt the end, my approach was:
\nI had around 10 models validated against fold 1 and 10 validated against fold 2. I forward selected each fold separately and averaged their predictions. Seeing as I never got around to full KFold training, I thought that including public models would add some beneficial diversity, so I added @sin’s public 5Fold ResNet200D models to inference. My final submission was a convex combination of my own models weighted 0.85 and sin’s 5 models weighted at 0.15.
I chose my best public leaderboard score for my first submission (seeing as I trusted CV and public correlation) and chose my best CV score for my second submission. They were both 0.971 on the private leaderboard and not surprisingly, my best CV score was also my best private leaderboard score.
What didn’t really work
\n512What I wanted to test, but didn’t
\nAcknowledgements
\nI would first like to thank @yasufuminakama and @hengck23 for sharing their approach on how to use annotated images to increase CV scores. Without their generous contributions in this competition, I would not have made much progress during these past couple months.
\nI also want to thank @chrisdeotte for his wonderful notebooks on forward selection, pseudo-labeling, and finding duplicate images with RAPIDS. Forward selection was very useful for generating high quality pseudo labels and he inspired me to experiment with ensembling models trained on different image sizes. Thank you Chris.
\nAdditionally, I would like to thank @raddar for sharing domain knowledge so openly during this competition and @sin for sharing his use of heavy augmentations and his cross-validation strategy. In coupling their ideas, I was able to generate a robust CV strategy that informed all my decisions during this competition.
\nLastly, I want to thank Z by HP & NVIDIA for providing me with a Z8 workstation and ZBook Studio laptop. Without these GPUs, I would not have been able to experiment with these larger networks and different image resolutions so quickly. I ran smaller experiments / debugged my code on the laptop and ran the main experiments on the workstation. GPU memory was the bottleneck in this competition so having powerful local GPUs was integral to my solution. (RTX Quadro 8000 and RTX Quadro 5000 mobile, for reference). When it comes to deep learning and GPUs, bigger seems to be better. That being said, I wish I found more elegant approaches to circumvent this problem, but I am still learning. Next time.
", "messages": [ { "id": "1241203", "postDate": "03/17/2021 00:15:48", "content": "60th Place Solution
\nThank you Kaggle and RANZCR for hosting this competition. It was the first competition I really immersed myself in and I have learned a tremendous amount. Thank you to all the competitors, especially those that so openly shared their results and techniques. Kaggle is a wonderful medium through which to learn data science, and it would not be the same without everyone sharing and communicating.
\nI am not pleased with my final standing, but I believe that all Kagglers should share their procedure, irrespective of their final position, because it is highly unlikely that anyone else did exactly what you did, so there is always something to learn from reading other's write ups. Here it goes.
\nImage resolutions
\nMy approach was a brute force approach and not particularly elegant. I began with @yasufuminakama’s kernels on 3 stage training with annotations and changed several parameters like learning rate, batch size, training augmentations, and optimizer (AdamP showed better CV results than Adam). I also generated stratified folds by coupling @sin's kernel with some useful information about incorrect labels / inverted images shared by @raddar and others. Once I had established a CV strategy, I started to experiment with different image resolutions, all with a ResNet200D. I tried image sizes from 512 to 736 in increments of 16 (all with a batch size of 32). I saw an increase in AUC + decrease in BCE loss each time I bumped up the image resolution. I validated all models against my first fold to save time (more on that later).
There was some discussion as to whether or not this multi stage approach with teacher-student training on annotated images actually gave any performance boost. I continued to use it anyways for one reason: you can use the stage 2 weights to lower training time. I trained stages 1 and 2 on a lower image resolution and then trained many different stage 3 models of various image resolutions from these weights. In other words, instead of repeating all 3 stages anytime I changed image resolutions / other parameters, I simply reused stage 2 weights and skipped to stage 3. This is not ideal, but it saved me a ton of training time and allowed me to run more experiments. I figured it would work seeing as we frequently fine tune models that have been trained on ImageNet with different image resolutions than the ones they were initially trained on.
\nPreprocessing
\nI did minimal preprocessing. It was mentioned that some images have black borders around them. I removed them during inference and saw an increase in score. I hesitated to remove them from training as these black boxes could serve as a form of regularization, but decided to remove them for the last couple models I trained. I figured if they helped with inference, they would help with training, but I did not run any real experiments to verify this.
\nForward Selection
\nI then created a fold prediction correlation heatmap for these different image sizes and saw a surprisingly low correlation between the ResNet200D model when trained on different image sizes (0.85 - 0.91). This showed me that one could get good model diversity by simply changing image resolutions, which got me thinking about ensembling. I remembered reading @chrisdeotte’s kernel on forward selection from the Melanoma competition and decided to try it. I saw nice improvements when doing so and decided to experiment with different model architectures as well. I trained a SEResNet152D on 672 and 720 image resolutions and checked the fold prediction correlations between them with my previous ResNet200D models: the correlation was even lower, so I threw them into forward selection with my other ResNet200D models.
The above procedure got me to CV 0.966 and public leaderboard 0.968. I then experimented with RegNetYs, EfficientNets, and ResNeSts. RegNetYs took too long to train and EfficientNets were never able to reach the same CV score as the ResNet200D / SEResNet152D. Only the ResNeSt’s got similar scores, but simply took too long to train. I had reached a bit of an impasse and was not sure how to proceed.
Pseudo labels
\nI then saw this post on duplicate images between the NIH Chest XRays dataset and the training / public test images given to use in this competition. Seeing as I already had an ensemble of models with good diversity, I was confident I could generate fairly high quality pseudo labels from this external dataset. (Note that you must generate pseudo labels from models that have not seen the fold you are validating against otherwise you get subtle CV leakage). I first tried to remove duplicates using CNN embeddings and RAPIDS NearestNeighbors, but this approach still left around 500 duplicate images, so I combined this approach with ImageHash to remove the remaining duplicates. I then started generating pseudo labels. I decided to use soft pseudo labels where at least one catheter class is present (any column with prediction above 0.7) and all other classes without any catheters (remaining columns with predictions below 0.3). I added Gaussian noise to the pseudo labels in an attempt to reduce confirmation bias.
Instead of re-training the model jointly with concatenated labeled + pseudo labels, I decided to introduce another training stage in which I train on only NIH pseudo labels. The final stage is then fine tuning on labeled training data. I also decided to train the last few epochs without any data augmentation, based on another discussion post, and saw further improvements. I carried out this procedure for my top-3 highest scoring models and threw them back into my forward selection algorithm to compare CV and public leaderboard score. It indeed increased.
\nI am still not sure if my CV was entirely leak free. I think many teams that used external data had some form of leakage. For this reason, I only included 2 NIH pseudo label pretrained models in the folds I forward selected against. Had I invested more time in examining these duplicate images found by RAPIDS NN and ImageHash, I think I would have had a more robust CV and better public / private score.
\nTraining / validation folds
\nUp to this point, I had only validated against the first fold, so in some sense I was ‘overfitting’ my own validation set. This is not ideal; it would be much better to validate each model against all folds and use the full OOF to ensemble, as opposed to single fold predictions. That being said, I joined this competition fairly late so I needed to save time somehow while simultaneously running many experiments. I also trusted my fold splits and was not overly concerned with validating against only one fold. However, I did want my final predictions to encompass all training data, so I repeated what I did on the first fold with the second fold, so that my models (in total) had been trained on all data folds.
\nAt the end, my approach was:
\nI had around 10 models validated against fold 1 and 10 validated against fold 2. I forward selected each fold separately and averaged their predictions. Seeing as I never got around to full KFold training, I thought that including public models would add some beneficial diversity, so I added @sin’s public 5Fold ResNet200D models to inference. My final submission was a convex combination of my own models weighted 0.85 and sin’s 5 models weighted at 0.15.
I chose my best public leaderboard score for my first submission (seeing as I trusted CV and public correlation) and chose my best CV score for my second submission. They were both 0.971 on the private leaderboard and not surprisingly, my best CV score was also my best private leaderboard score.
What didn’t really work
\n512What I wanted to test, but didn’t
\nAcknowledgements
\nI would first like to thank @yasufuminakama and @hengck23 for sharing their approach on how to use annotated images to increase CV scores. Without their generous contributions in this competition, I would not have made much progress during these past couple months.
\nI also want to thank @chrisdeotte for his wonderful notebooks on forward selection, pseudo-labeling, and finding duplicate images with RAPIDS. Forward selection was very useful for generating high quality pseudo labels and he inspired me to experiment with ensembling models trained on different image sizes. Thank you Chris.
\nAdditionally, I would like to thank @raddar for sharing domain knowledge so openly during this competition and @sin for sharing his use of heavy augmentations and his cross-validation strategy. In coupling their ideas, I was able to generate a robust CV strategy that informed all my decisions during this competition.
\nLastly, I want to thank Z by HP & NVIDIA for providing me with a Z8 workstation and ZBook Studio laptop. Without these GPUs, I would not have been able to experiment with these larger networks and different image resolutions so quickly. I ran smaller experiments / debugged my code on the laptop and ran the main experiments on the workstation. GPU memory was the bottleneck in this competition so having powerful local GPUs was integral to my solution. (RTX Quadro 8000 and RTX Quadro 5000 mobile, for reference). When it comes to deep learning and GPUs, bigger seems to be better. That being said, I wish I found more elegant approaches to circumvent this problem, but I am still learning. Next time.
", "rawMarkdown": "**60th Place Solution**\n\nThank you Kaggle and RANZCR for hosting this competition. It was the first competition I really immersed myself in and I have learned a tremendous amount. Thank you to all the competitors, especially those that so openly shared their results and techniques. Kaggle is a wonderful medium through which to learn data science, and it would not be the same without everyone sharing and communicating. \n\nI am not pleased with my final standing, but I believe that all Kagglers should share their procedure, irrespective of their final position, because it is highly unlikely that anyone else did exactly what you did, so there is always something to learn from reading other's write ups. Here it goes.\n\n**Image resolutions**\n\nMy approach was a brute force approach and not particularly elegant. I began with [@yasufuminakama](https://www.kaggle.com/yasufuminakama)’s kernels on 3 stage training with annotations and changed several parameters like learning rate, batch size, training augmentations, and optimizer (AdamP showed better CV results than Adam). I also generated stratified folds by coupling [@sin](https://www.kaggle.com/underwearfitting)'s [kernel](https://www.kaggle.com/underwearfitting/how-to-properly-split-folds) with some [useful information](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/210064) about incorrect labels / inverted images shared by [@raddar](https://www.kaggle.com/raddar) and others. Once I had established a CV strategy, I started to experiment with different image resolutions, all with a ResNet200D. I tried image sizes from `512` to `736` in increments of `16` (all with a batch size of `32`). I saw an increase in AUC + decrease in BCE loss each time I bumped up the image resolution. I validated all models against my first fold to save time (more on that later). \n\n[There was some discussion](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/217296) as to whether or not this multi stage approach with teacher-student training on annotated images actually gave any performance boost. I continued to use it anyways for one reason: you can use the stage 2 weights to lower training time. I trained stages 1 and 2 on a lower image resolution and then trained many different stage 3 models of various image resolutions from these weights. In other words, instead of repeating all 3 stages anytime I changed image resolutions / other parameters, I simply reused stage 2 weights and skipped to stage 3. This is not ideal, but it saved me a ton of training time and allowed me to run more experiments. I figured it would work seeing as we frequently fine tune models that have been trained on ImageNet with different image resolutions than the ones they were initially trained on. \n\n**Preprocessing**\n\nI did minimal preprocessing. [It was mentioned](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/224146) that some images have black borders around them. I removed them during inference and saw an increase in score. I hesitated to remove them from training as these black boxes could serve as a form of regularization, but decided to remove them for the last couple models I trained. I figured if they helped with inference, they would help with training, but I did not run any real experiments to verify this.\n\n**Forward Selection**\n\nI then created a fold prediction correlation heatmap for these different image sizes and saw a surprisingly low correlation between the ResNet200D model when trained on different image sizes (0.85 - 0.91). This showed me that one could get good model diversity by simply changing image resolutions, which got me thinking about ensembling. I remembered reading [@chrisdeotte](https://www.kaggle.com/cdeotte)’s [kernel on forward selection](https://www.kaggle.com/cdeotte/forward-selection-oof-ensemble-0-942-private) from the Melanoma competition and decided to try it. I saw nice improvements when doing so and decided to experiment with different model architectures as well. I trained a SEResNet152D on `672` and `720` image resolutions and checked the fold prediction correlations between them with my previous ResNet200D models: the correlation was even lower, so I threw them into forward selection with my other ResNet200D models. \n\nThe above procedure got me to CV `0.966` and public leaderboard `0.968`. I then experimented with RegNetYs, EfficientNets, and ResNeSts. RegNetYs took too long to train and EfficientNets were never able to reach the same CV score as the ResNet200D / SEResNet152D. Only the ResNeSt’s got similar scores, but simply took too long to train. I had reached a bit of an impasse and was not sure how to proceed.\n\n**Pseudo labels**\n\nI then saw [this post](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/221808) on duplicate images between the NIH Chest XRays dataset and the training / public test images given to use in this competition. Seeing as I already had an ensemble of models with good diversity, I was confident I could generate fairly high quality pseudo labels from this external dataset. (Note that you must generate pseudo labels from models that have not seen the fold you are validating against otherwise you get subtle CV leakage). I first tried to remove duplicates using CNN embeddings and RAPIDS NearestNeighbors, but this approach still left around 500 duplicate images, so I combined this approach with ImageHash to remove the remaining duplicates. I then started generating pseudo labels. I decided to use soft pseudo labels where at least one catheter class is present (any column with prediction above `0.7`) and all other classes without any catheters (remaining columns with predictions below `0.3`). I added Gaussian noise to the pseudo labels in an attempt to reduce confirmation bias.\n\nInstead of re-training the model jointly with concatenated labeled + pseudo labels, I decided to introduce another training stage in which I train on only NIH pseudo labels. The final stage is then fine tuning on labeled training data. I also decided to train the last few epochs without any data augmentation, based on [another discussion post](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/212532), and saw further improvements. I carried out this procedure for my top-3 highest scoring models and threw them back into my forward selection algorithm to compare CV and public leaderboard score. It indeed increased. \n\nI am still not sure if my CV was entirely leak free. I think many teams that used external data had some form of leakage. For this reason, I only included 2 NIH pseudo label pretrained models in the folds I forward selected against. Had I invested more time in examining these duplicate images found by RAPIDS NN and ImageHash, I think I would have had a more robust CV and better public / private score. \n\n**Training / validation folds**\n\nUp to this point, I had only validated against the first fold, so in some sense I was ‘overfitting’ my own validation set. This is not ideal; it would be much better to validate each model against all folds and use the full OOF to ensemble, as opposed to single fold predictions. That being said, I joined this competition fairly late so I needed to save time somehow while simultaneously running many experiments. I also trusted my fold splits and was not overly concerned with validating against only one fold. However, I did want my final predictions to encompass all training data, so I repeated what I did on the first fold with the second fold, so that my models (in total) had been trained on all data folds. \n\nAt the end, my approach was:\n\n* Train teacher on annotated samples (light augmentations for 4 epochs) \n* Transfer this information to student (light augmentations for 6-7 epochs)\n* Use these student weights as a starting point for NIH pseudo labeling training (light augmentations for 7-8 epochs)\n* Train on full labeled data (heavy augmentation for 7-8 epochs)\n* Train on full labeled data (no augmentation for 1-2 epochs)\n* Repeat on different image sizes and model architectures and forward select based on validation fold predictions\n\nI had around 10 models validated against fold 1 and 10 validated against fold 2. I forward selected each fold separately and averaged their predictions. Seeing as I never got around to full KFold training, I thought that including public models would add some beneficial diversity, so I added [@sin](https://www.kaggle.com/underwearfitting)’s public 5Fold ResNet200D models to inference. My final submission was a convex combination of my own models weighted `0.85` and [sin](https://www.kaggle.com/underwearfitting)’s 5 models weighted at `0.15`.\n\nI chose my best public leaderboard score for my first submission (seeing as I trusted CV and public correlation) and chose my best CV score for my second submission. They were both `0.971` on the private leaderboard and not surprisingly, my best CV score was also my best private leaderboard score. \n\n**What didn’t really work**\n\n* TTA beyond simple horizontal flipping only showed improvements when the number of steps was fairly high (>5) and this took too long to infer with\n* Models other than ResNet200D and SEResNet152D (ResNeSt200E worked as well but took too long to train)\n* Image resolutions below `512`\n\n**What I wanted to test, but didn’t**\n\n* Experiment with changing image resolutions during training as a means of regularization\n* Using shape priors and a segmentation model to create features (categorize catheter overlaps with HoG descriptors and intensity histograms)\n* Use masks and segmentation model outputs to categorize distance between various anatomical features and the catheters to create more features\n* Upsample images without any catheters present with NIH Chest XRay images and compare performance\n* Experiment with label smoothing / special loss functions\n* Use MixUp on pseudo labels to further reduce confirmation bias\n\n**Acknowledgements**\n\nI would first like to thank [@yasufuminakama](https://www.kaggle.com/yasufuminakama) and [@hengck23](https://www.kaggle.com/hengck23) for sharing their approach on how to use annotated images to increase CV scores. Without their generous contributions in this competition, I would not have made much progress during these past couple months. \n\nI also want to thank [@chrisdeotte](https://www.kaggle.com/cdeotte) for his wonderful notebooks on forward selection, [pseudo-labeling](https://www.kaggle.com/cdeotte/pseudo-labeling-qda-0-969), and [finding duplicate images with RAPIDS](https://www.kaggle.com/cdeotte/rapids-cuml-knn-find-duplicates). Forward selection was very useful for generating high quality pseudo labels and he inspired me to experiment with ensembling models trained on different image sizes. Thank you Chris.\n\nAdditionally, I would like to thank [@raddar](https://www.kaggle.com/raddar) for sharing domain knowledge so openly during this competition and [@sin](https://www.kaggle.com/underwearfitting) for sharing his use of heavy augmentations and his cross-validation strategy. In coupling their ideas, I was able to generate a robust CV strategy that informed all my decisions during this competition.\n\nLastly, I want to thank Z by HP & NVIDIA for providing me with a Z8 workstation and ZBook Studio laptop. Without these GPUs, I would not have been able to experiment with these larger networks and different image resolutions so quickly. I ran smaller experiments / debugged my code on the laptop and ran the main experiments on the workstation. GPU memory was the bottleneck in this competition so having powerful local GPUs was integral to my solution. (RTX Quadro 8000 and RTX Quadro 5000 mobile, for reference). When it comes to deep learning and GPUs, bigger seems to be better. That being said, I wish I found more elegant approaches to circumvent this problem, but I am still learning. Next time.", "votes": null }, { "id": "1241214", "postDate": "03/17/2021 00:30:35", "content": "Congratulations @tuckerarrants and thanks for sharing this detailed writeup :)
\n\n\nI trained stages 1 and 2 on a lower image resolution and then trained many different stage 3 models of various image resolutions from these weights.
\n
How low was the image resolution you used in stage 1 and 2? Because with my models, whenever I increase the image size in step 3, both CV and LB decrease.
\nDid you tweak some parameters other than batch size when you increase the image size?
Thank you Amin. The lowest resolution I used in stages 1 and 2 was 576. Sometimes I went as high as 768 for these stages, but only before I started reusing lower resolution weights to save time.
I used slightly heavier augmentations each time I increased the image size.
", "rawMarkdown": "Thank you Amin. The lowest resolution I used in stages 1 and 2 was `576`. Sometimes I went as high as `768` for these stages, but only before I started reusing lower resolution weights to save time.\n\nI used slightly heavier augmentations each time I increased the image size.", "votes": null }, { "id": "1241225", "postDate": "03/17/2021 00:39:27", "content": "Great job Tucker. Thanks for sharing. Strong solo finish.
", "rawMarkdown": "Great job Tucker. Thanks for sharing. Strong solo finish.", "votes": null }, { "id": "1241227", "postDate": "03/17/2021 00:41:03", "content": "No thank you, I could not have done it without all your contributions. Congratulations on 2nd place, very impressive indeed.
", "rawMarkdown": "No thank you, I could not have done it without all your contributions. Congratulations on 2nd place, very impressive indeed.", "votes": null }, { "id": "1241232", "postDate": "03/17/2021 00:44:40", "content": "Great write-up. When looking at your ensembling efforts did you assign weights at the macro level across all columns or did you ever try to optimize at the column level? I found that in my experiment when assigning weights my ensembling technique would basically just hard select whichever model had the best performance for a given column and any combination of models didn't really seem to yield very much.
\nNever got around to applying that knowledge to the leaderboard, but I was surprised to see that behavior.
\n
Thank you. Sadly, I only got around to assigning them at the macro level. Only at the last minute did I think to do it for each column. I may play around with the column-wise approach now that the competition is over to compare performance.
", "rawMarkdown": "Thank you. Sadly, I only got around to assigning them at the macro level. Only at the last minute did I think to do it for each column. I may play around with the column-wise approach now that the competition is over to compare performance.", "votes": null }, { "id": "1241336", "postDate": "03/17/2021 02:10:35", "content": "@tuckerarrants Congratulations Tucker Arrants for becoming Competition expert and also for Solo Silver . Great Writeup
", "rawMarkdown": "tuckerarrants Congratulations Tucker Arrants for becoming Competition expert and also for Solo Silver . Great Writeup", "votes": null }, { "id": "1241355", "postDate": "03/17/2021 02:30:03", "content": "At the column level I tried. Will be posting a discussion soon. Somehow at the column level the optimization doesn’t lead to good results.
", "rawMarkdown": "At the column level I tried. Will be posting a discussion soon. Somehow at the column level the optimization doesn’t lead to good results.", "votes": null }, { "id": "1241361", "postDate": "03/17/2021 02:33:44", "content": "Interesting…I look forward to reading your discussion post.
", "rawMarkdown": "Interesting...I look forward to reading your discussion post.", "votes": null }, { "id": "1241363", "postDate": "03/17/2021 02:34:42", "content": "Thank you Usha, I really appreciate it.
", "rawMarkdown": "Thank you Usha, I really appreciate it.", "votes": null }, { "id": "1241422", "postDate": "03/17/2021 03:18:09", "content": "Great write up, thank you. Can I ask how you were able to get the Z8 and Zbook from HP / NVIDIA 😛
\nTrying to figure out how I can overcome this bottleneck!
It was quite the surprise. They launched an ambassadorship program last year and I was asked to join, presumably based on some of my public Kaggle notebooks. (It was not something I applied for).
\nMy best advice is to continue your data science journey and to share what you learn: hopefully a similar opportunity will present itself to you. I think that such programs will become more prevalent as more companies recognize the importance of hardware in data science / machine learning / deep learning.
\nI feel your pain, I have been trying to secure NVIDIA GPUs for my own personal use and they are very illusive these days. Hopefully this shortage is temporary and we can all get our hands on some solid GPUs in the near future.
", "rawMarkdown": "It was quite the surprise. They launched an ambassadorship program last year and I was asked to join, presumably based on some of my public Kaggle notebooks. (It was not something I applied for). \n\nMy best advice is to continue your data science journey and to share what you learn: hopefully a similar opportunity will present itself to you. I think that such programs will become more prevalent as more companies recognize the importance of hardware in data science / machine learning / deep learning. \n\nI feel your pain, I have been trying to secure NVIDIA GPUs for my own personal use and they are very illusive these days. Hopefully this shortage is temporary and we can all get our hands on some solid GPUs in the near future.", "votes": null }, { "id": "1241518", "postDate": "03/17/2021 04:41:52", "content": "Congratulations. Well explained Great job ))
", "rawMarkdown": "Congratulations. Well explained Great job ))", "votes": null } ], "comments": [ { "id": 1241214, "author_name": "amiiiney", "author_url": "", "post_date": "03/17/2021 00:30:35", "content": "Congratulations @tuckerarrants and thanks for sharing this detailed writeup :)
\n\n\nI trained stages 1 and 2 on a lower image resolution and then trained many different stage 3 models of various image resolutions from these weights.
\n
How low was the image resolution you used in stage 1 and 2? Because with my models, whenever I increase the image size in step 3, both CV and LB decrease.
\nDid you tweak some parameters other than batch size when you increase the image size?
Thank you Amin. The lowest resolution I used in stages 1 and 2 was 576. Sometimes I went as high as 768 for these stages, but only before I started reusing lower resolution weights to save time.
I used slightly heavier augmentations each time I increased the image size.
", "votes": null, "replies": [] } ] }, { "id": 1241225, "author_name": "underwearfitting", "author_url": "", "post_date": "03/17/2021 00:39:27", "content": "Great job Tucker. Thanks for sharing. Strong solo finish.
", "votes": null, "replies": [ { "id": 1241227, "author_name": "tuckerarrants", "author_url": "", "post_date": "03/17/2021 00:41:03", "content": "No thank you, I could not have done it without all your contributions. Congratulations on 2nd place, very impressive indeed.
", "votes": null, "replies": [] } ] }, { "id": 1241232, "author_name": "ryches", "author_url": "", "post_date": "03/17/2021 00:44:40", "content": "Great write-up. When looking at your ensembling efforts did you assign weights at the macro level across all columns or did you ever try to optimize at the column level? I found that in my experiment when assigning weights my ensembling technique would basically just hard select whichever model had the best performance for a given column and any combination of models didn't really seem to yield very much.
\nNever got around to applying that knowledge to the leaderboard, but I was surprised to see that behavior.
\n
Thank you. Sadly, I only got around to assigning them at the macro level. Only at the last minute did I think to do it for each column. I may play around with the column-wise approach now that the competition is over to compare performance.
", "votes": null, "replies": [] }, { "id": 1241355, "author_name": "reighns", "author_url": "", "post_date": "03/17/2021 02:30:03", "content": "At the column level I tried. Will be posting a discussion soon. Somehow at the column level the optimization doesn’t lead to good results.
", "votes": null, "replies": [] }, { "id": 1241361, "author_name": "tuckerarrants", "author_url": "", "post_date": "03/17/2021 02:33:44", "content": "Interesting…I look forward to reading your discussion post.
", "votes": null, "replies": [] } ] }, { "id": 1241336, "author_name": "usharengaraju", "author_url": "", "post_date": "03/17/2021 02:10:35", "content": "@tuckerarrants Congratulations Tucker Arrants for becoming Competition expert and also for Solo Silver . Great Writeup
", "votes": null, "replies": [ { "id": 1241363, "author_name": "tuckerarrants", "author_url": "", "post_date": "03/17/2021 02:34:42", "content": "Thank you Usha, I really appreciate it.
", "votes": null, "replies": [] } ] }, { "id": 1241422, "author_name": "reubenschmidt", "author_url": "", "post_date": "03/17/2021 03:18:09", "content": "Great write up, thank you. Can I ask how you were able to get the Z8 and Zbook from HP / NVIDIA 😛
\nTrying to figure out how I can overcome this bottleneck!
It was quite the surprise. They launched an ambassadorship program last year and I was asked to join, presumably based on some of my public Kaggle notebooks. (It was not something I applied for).
\nMy best advice is to continue your data science journey and to share what you learn: hopefully a similar opportunity will present itself to you. I think that such programs will become more prevalent as more companies recognize the importance of hardware in data science / machine learning / deep learning.
\nI feel your pain, I have been trying to secure NVIDIA GPUs for my own personal use and they are very illusive these days. Hopefully this shortage is temporary and we can all get our hands on some solid GPUs in the near future.
", "votes": null, "replies": [] } ] }, { "id": 1241518, "author_name": "ammarali32", "author_url": "", "post_date": "03/17/2021 04:41:52", "content": "Congratulations. Well explained Great job ))
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1241203": "**60th Place Solution**\n\nThank you Kaggle and RANZCR for hosting this competition. It was the first competition I really immersed myself in and I have learned a tremendous amount. Thank you to all the competitors, especially those that so openly shared their results and techniques. Kaggle is a wonderful medium through which to learn data science, and it would not be the same without everyone sharing and communicating. \n\nI am not pleased with my final standing, but I believe that all Kagglers should share their procedure, irrespective of their final position, because it is highly unlikely that anyone else did exactly what you did, so there is always something to learn from reading other's write ups. Here it goes.\n\n**Image resolutions**\n\nMy approach was a brute force approach and not particularly elegant. I began with [@yasufuminakama](https://www.kaggle.com/yasufuminakama)’s kernels on 3 stage training with annotations and changed several parameters like learning rate, batch size, training augmentations, and optimizer (AdamP showed better CV results than Adam). I also generated stratified folds by coupling [@sin](https://www.kaggle.com/underwearfitting)'s [kernel](https://www.kaggle.com/underwearfitting/how-to-properly-split-folds) with some [useful information](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/210064) about incorrect labels / inverted images shared by [@raddar](https://www.kaggle.com/raddar) and others. Once I had established a CV strategy, I started to experiment with different image resolutions, all with a ResNet200D. I tried image sizes from `512` to `736` in increments of `16` (all with a batch size of `32`). I saw an increase in AUC + decrease in BCE loss each time I bumped up the image resolution. I validated all models against my first fold to save time (more on that later). \n\n[There was some discussion](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/217296) as to whether or not this multi stage approach with teacher-student training on annotated images actually gave any performance boost. I continued to use it anyways for one reason: you can use the stage 2 weights to lower training time. I trained stages 1 and 2 on a lower image resolution and then trained many different stage 3 models of various image resolutions from these weights. In other words, instead of repeating all 3 stages anytime I changed image resolutions / other parameters, I simply reused stage 2 weights and skipped to stage 3. This is not ideal, but it saved me a ton of training time and allowed me to run more experiments. I figured it would work seeing as we frequently fine tune models that have been trained on ImageNet with different image resolutions than the ones they were initially trained on. \n\n**Preprocessing**\n\nI did minimal preprocessing. [It was mentioned](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/224146) that some images have black borders around them. I removed them during inference and saw an increase in score. I hesitated to remove them from training as these black boxes could serve as a form of regularization, but decided to remove them for the last couple models I trained. I figured if they helped with inference, they would help with training, but I did not run any real experiments to verify this.\n\n**Forward Selection**\n\nI then created a fold prediction correlation heatmap for these different image sizes and saw a surprisingly low correlation between the ResNet200D model when trained on different image sizes (0.85 - 0.91). This showed me that one could get good model diversity by simply changing image resolutions, which got me thinking about ensembling. I remembered reading [@chrisdeotte](https://www.kaggle.com/cdeotte)’s [kernel on forward selection](https://www.kaggle.com/cdeotte/forward-selection-oof-ensemble-0-942-private) from the Melanoma competition and decided to try it. I saw nice improvements when doing so and decided to experiment with different model architectures as well. I trained a SEResNet152D on `672` and `720` image resolutions and checked the fold prediction correlations between them with my previous ResNet200D models: the correlation was even lower, so I threw them into forward selection with my other ResNet200D models. \n\nThe above procedure got me to CV `0.966` and public leaderboard `0.968`. I then experimented with RegNetYs, EfficientNets, and ResNeSts. RegNetYs took too long to train and EfficientNets were never able to reach the same CV score as the ResNet200D / SEResNet152D. Only the ResNeSt’s got similar scores, but simply took too long to train. I had reached a bit of an impasse and was not sure how to proceed.\n\n**Pseudo labels**\n\nI then saw [this post](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/221808) on duplicate images between the NIH Chest XRays dataset and the training / public test images given to use in this competition. Seeing as I already had an ensemble of models with good diversity, I was confident I could generate fairly high quality pseudo labels from this external dataset. (Note that you must generate pseudo labels from models that have not seen the fold you are validating against otherwise you get subtle CV leakage). I first tried to remove duplicates using CNN embeddings and RAPIDS NearestNeighbors, but this approach still left around 500 duplicate images, so I combined this approach with ImageHash to remove the remaining duplicates. I then started generating pseudo labels. I decided to use soft pseudo labels where at least one catheter class is present (any column with prediction above `0.7`) and all other classes without any catheters (remaining columns with predictions below `0.3`). I added Gaussian noise to the pseudo labels in an attempt to reduce confirmation bias.\n\nInstead of re-training the model jointly with concatenated labeled + pseudo labels, I decided to introduce another training stage in which I train on only NIH pseudo labels. The final stage is then fine tuning on labeled training data. I also decided to train the last few epochs without any data augmentation, based on [another discussion post](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/212532), and saw further improvements. I carried out this procedure for my top-3 highest scoring models and threw them back into my forward selection algorithm to compare CV and public leaderboard score. It indeed increased. \n\nI am still not sure if my CV was entirely leak free. I think many teams that used external data had some form of leakage. For this reason, I only included 2 NIH pseudo label pretrained models in the folds I forward selected against. Had I invested more time in examining these duplicate images found by RAPIDS NN and ImageHash, I think I would have had a more robust CV and better public / private score. \n\n**Training / validation folds**\n\nUp to this point, I had only validated against the first fold, so in some sense I was ‘overfitting’ my own validation set. This is not ideal; it would be much better to validate each model against all folds and use the full OOF to ensemble, as opposed to single fold predictions. That being said, I joined this competition fairly late so I needed to save time somehow while simultaneously running many experiments. I also trusted my fold splits and was not overly concerned with validating against only one fold. However, I did want my final predictions to encompass all training data, so I repeated what I did on the first fold with the second fold, so that my models (in total) had been trained on all data folds. \n\nAt the end, my approach was:\n\n* Train teacher on annotated samples (light augmentations for 4 epochs) \n* Transfer this information to student (light augmentations for 6-7 epochs)\n* Use these student weights as a starting point for NIH pseudo labeling training (light augmentations for 7-8 epochs)\n* Train on full labeled data (heavy augmentation for 7-8 epochs)\n* Train on full labeled data (no augmentation for 1-2 epochs)\n* Repeat on different image sizes and model architectures and forward select based on validation fold predictions\n\nI had around 10 models validated against fold 1 and 10 validated against fold 2. I forward selected each fold separately and averaged their predictions. Seeing as I never got around to full KFold training, I thought that including public models would add some beneficial diversity, so I added [@sin](https://www.kaggle.com/underwearfitting)’s public 5Fold ResNet200D models to inference. My final submission was a convex combination of my own models weighted `0.85` and [sin](https://www.kaggle.com/underwearfitting)’s 5 models weighted at `0.15`.\n\nI chose my best public leaderboard score for my first submission (seeing as I trusted CV and public correlation) and chose my best CV score for my second submission. They were both `0.971` on the private leaderboard and not surprisingly, my best CV score was also my best private leaderboard score. \n\n**What didn’t really work**\n\n* TTA beyond simple horizontal flipping only showed improvements when the number of steps was fairly high (>5) and this took too long to infer with\n* Models other than ResNet200D and SEResNet152D (ResNeSt200E worked as well but took too long to train)\n* Image resolutions below `512`\n\n**What I wanted to test, but didn’t**\n\n* Experiment with changing image resolutions during training as a means of regularization\n* Using shape priors and a segmentation model to create features (categorize catheter overlaps with HoG descriptors and intensity histograms)\n* Use masks and segmentation model outputs to categorize distance between various anatomical features and the catheters to create more features\n* Upsample images without any catheters present with NIH Chest XRay images and compare performance\n* Experiment with label smoothing / special loss functions\n* Use MixUp on pseudo labels to further reduce confirmation bias\n\n**Acknowledgements**\n\nI would first like to thank [@yasufuminakama](https://www.kaggle.com/yasufuminakama) and [@hengck23](https://www.kaggle.com/hengck23) for sharing their approach on how to use annotated images to increase CV scores. Without their generous contributions in this competition, I would not have made much progress during these past couple months. \n\nI also want to thank [@chrisdeotte](https://www.kaggle.com/cdeotte) for his wonderful notebooks on forward selection, [pseudo-labeling](https://www.kaggle.com/cdeotte/pseudo-labeling-qda-0-969), and [finding duplicate images with RAPIDS](https://www.kaggle.com/cdeotte/rapids-cuml-knn-find-duplicates). Forward selection was very useful for generating high quality pseudo labels and he inspired me to experiment with ensembling models trained on different image sizes. Thank you Chris.\n\nAdditionally, I would like to thank [@raddar](https://www.kaggle.com/raddar) for sharing domain knowledge so openly during this competition and [@sin](https://www.kaggle.com/underwearfitting) for sharing his use of heavy augmentations and his cross-validation strategy. In coupling their ideas, I was able to generate a robust CV strategy that informed all my decisions during this competition.\n\nLastly, I want to thank Z by HP & NVIDIA for providing me with a Z8 workstation and ZBook Studio laptop. Without these GPUs, I would not have been able to experiment with these larger networks and different image resolutions so quickly. I ran smaller experiments / debugged my code on the laptop and ran the main experiments on the workstation. GPU memory was the bottleneck in this competition so having powerful local GPUs was integral to my solution. (RTX Quadro 8000 and RTX Quadro 5000 mobile, for reference). When it comes to deep learning and GPUs, bigger seems to be better. That being said, I wish I found more elegant approaches to circumvent this problem, but I am still learning. Next time.", "1241214": "Congratulations @tuckerarrants and thanks for sharing this detailed writeup :)\n\n>I trained stages 1 and 2 on a lower image resolution and then trained many different stage 3 models of various image resolutions from these weights.\n\nHow low was the image resolution you used in stage 1 and 2? Because with my models, whenever I increase the image size in step 3, both CV and LB decrease.\nDid you tweak some parameters other than batch size when you increase the image size?", "1241219": "Thank you Amin. The lowest resolution I used in stages 1 and 2 was `576`. Sometimes I went as high as `768` for these stages, but only before I started reusing lower resolution weights to save time.\n\nI used slightly heavier augmentations each time I increased the image size.", "1241225": "Great job Tucker. Thanks for sharing. Strong solo finish.", "1241227": "No thank you, I could not have done it without all your contributions. Congratulations on 2nd place, very impressive indeed.", "1241232": "Great write-up. When looking at your ensembling efforts did you assign weights at the macro level across all columns or did you ever try to optimize at the column level? I found that in my experiment when assigning weights my ensembling technique would basically just hard select whichever model had the best performance for a given column and any combination of models didn't really seem to yield very much. \n\nNever got around to applying that knowledge to the leaderboard, but I was surprised to see that behavior. \n\n", "1241247": "Thank you. Sadly, I only got around to assigning them at the macro level. Only at the last minute did I think to do it for each column. I may play around with the column-wise approach now that the competition is over to compare performance.", "1241336": "tuckerarrants Congratulations Tucker Arrants for becoming Competition expert and also for Solo Silver . Great Writeup", "1241355": "At the column level I tried. Will be posting a discussion soon. Somehow at the column level the optimization doesn’t lead to good results.", "1241361": "Interesting...I look forward to reading your discussion post.", "1241363": "Thank you Usha, I really appreciate it.", "1241422": "Great write up, thank you. Can I ask how you were able to get the Z8 and Zbook from HP / NVIDIA 😛 \nTrying to figure out how I can overcome this bottleneck!", "1241444": "It was quite the surprise. They launched an ambassadorship program last year and I was asked to join, presumably based on some of my public Kaggle notebooks. (It was not something I applied for). \n\nMy best advice is to continue your data science journey and to share what you learn: hopefully a similar opportunity will present itself to you. I think that such programs will become more prevalent as more companies recognize the importance of hardware in data science / machine learning / deep learning. \n\nI feel your pain, I have been trying to secure NVIDIA GPUs for my own personal use and they are very illusive these days. Hopefully this shortage is temporary and we can all get our hands on some solid GPUs in the near future.", "1241518": "Congratulations. Well explained Great job ))" }, "source": "meta" }