Congratulations to all the winners and great thanks to discussion participants!
\n\nWe started the competition using Keras because we were familiar with it and then preferred to stick to it.
\n\nAt first, we were just trying different models (Inception, ResNets, Xception) and some small architectures (MobileNet, SqueezeNet) and chose to use pretrained ResNet50 from Keras applications. After long time with it and no result more than 0.47 LB we switched to Inception networks.
\n\nAll models were trained on single fold: 85% train and 15% test.\nWe splitted data using iterative stratification package from discussion.\nFrom external data we used only rare classes and excluded all duplicates from training.
\n\nTop performing solution scored 0.505 on private LB. We didn't choose it because it performed worse both in local validation and on public LB. \nIt was an ensemble of all the best SE-BNInception (6 models) with fixed threshold. 4 of them were RGBY and 2 were RGB.
\n\nWe used primarily the server with one 1080Ti and 64GB of RAM.\nTo reduce disk bottleneck, we put all training data into ramdisk with plenty of swap (~3x load speedup).\nOccasionally we used standard workstations with 1060 and 8-16GB RAM.
", "messages": [ { "id": "454523", "postDate": "01/11/2019 19:28:12", "content": "Congratulations to all the winners and great thanks to discussion participants!
\n\nWe started the competition using Keras because we were familiar with it and then preferred to stick to it.
\n\nAt first, we were just trying different models (Inception, ResNets, Xception) and some small architectures (MobileNet, SqueezeNet) and chose to use pretrained ResNet50 from Keras applications. After long time with it and no result more than 0.47 LB we switched to Inception networks.
\n\nAll models were trained on single fold: 85% train and 15% test.\nWe splitted data using iterative stratification package from discussion.\nFrom external data we used only rare classes and excluded all duplicates from training.
\n\nTop performing solution scored 0.505 on private LB. We didn't choose it because it performed worse both in local validation and on public LB. \nIt was an ensemble of all the best SE-BNInception (6 models) with fixed threshold. 4 of them were RGBY and 2 were RGB.
\n\nWe used primarily the server with one 1080Ti and 64GB of RAM.\nTo reduce disk bottleneck, we put all training data into ramdisk with plenty of swap (~3x load speedup).\nOccasionally we used standard workstations with 1060 and 8-16GB RAM.
", "rawMarkdown": "Congratulations to all the winners and great thanks to discussion participants!\n\nWe started the competition using Keras because we were familiar with it and then preferred to stick to it. \n\nAt first, we were just trying different models (Inception, ResNets, Xception) and some small architectures (MobileNet, SqueezeNet) and chose to use pretrained ResNet50 from Keras applications. After long time with it and no result more than 0.47 LB we switched to Inception networks. \n\nAll models were trained on single fold: 85% train and 15% test.\nWe splitted data using iterative stratification package from discussion.\nFrom external data we used only rare classes and excluded all duplicates from training.\n\n## Things that worked ##\n\n- F1 loss and BCE with weighted positive predictions\n- SGD with warm restarts (cosine learning rate, initial lr=0.1)\n- Log-dampened class weights\n- Oversampling of rare classes by 5x\n- Undersampling classes 0 and 25 by 0.5x\n- \"Standard\" augmentations (flip rl/ud, shift, rotate, shear, brightness)\n- Squeeze-Excitation module\n- Weight decay\n- Randomly dropping green channel and all classes in 1% of images led to +0.04 LB\n- BatchNorm running mean/variance update on test set\n\n## Things that did not work ##\n- Imagenet weights (we still wonder why)\n- Focal loss\n- Threshold fitting\n- GapNet-like things\n- Convolutional block attention module\n- Filling empty predictions with most popular class\n- Mixup, probably because of the specialty of the green channel\n- Heavy augmentations (blur, various types of noise, dropout, contrast, elastic transformations, sharpen/emboss)\n- Small crops (less than 40% of number of pixels)\n- AdamW\n- Global average + global max pooling\n- TTA (but not hurt)\n- BatchNorm in dense layers reduced performance dramatically\n\n## Things that we regret: ##\n- Realized the way we overfit (high recall, low precision) too late (2 days before the end of the competition)\n- Not using cross-validation\n- Not ensembling different architectures\n- Closer to final: using Keras\n\n## Top solution ##\nTop performing solution scored 0.505 on private LB. We didn't choose it because it performed worse both in local validation and on public LB. \nIt was an ensemble of all the best SE-BNInception (6 models) with fixed threshold. 4 of them were RGBY and 2 were RGB.\n\n## Hardware ##\nWe used primarily the server with one 1080Ti and 64GB of RAM.\nTo reduce disk bottleneck, we put all training data into ramdisk with plenty of swap (~3x load speedup).\nOccasionally we used standard workstations with 1060 and 8-16GB RAM.", "votes": null }, { "id": "454651", "postDate": "01/11/2019 23:26:15", "content": "Thanks for sharing. We also tried different things for filling the empty predictions, but found nothing that worked
", "rawMarkdown": "Thanks for sharing. We also tried different things for filling the empty predictions, but found nothing that worked", "votes": null }, { "id": "1079465", "postDate": "11/16/2020 05:12:27", "content": "Hi, your solution was really impressive. I am new to Kaggle and just looking for some competitions for exercise. I wonder how you play with the squeeze-and-excitation structure with pretrained model from Keras, since Keras is well encapsulated and I wonder how to regularly insert layers of SEnet into pretrained networks like ResNet50. Plus if I insert SE layers into ResNet50, is it still appropriate to use the pretrained weights of it?
\nI'll really appreciate it for your reply. Thanks a lot.
Hi, your solution was really impressive. I am new to Kaggle and just looking for some competitions for exercise. I wonder how you play with the squeeze-and-excitation structure with pretrained model from Keras, since Keras is well encapsulated and I wonder how to regularly insert layers of SEnet into pretrained networks like ResNet50. Plus if I insert SE layers into ResNet50, is it still appropriate to use the pretrained weights of it?
\nI'll really appreciate it for your reply. Thanks a lot.
Thanks for sharing. We also tried different things for filling the empty predictions, but found nothing that worked
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "454523": "Congratulations to all the winners and great thanks to discussion participants!\n\nWe started the competition using Keras because we were familiar with it and then preferred to stick to it. \n\nAt first, we were just trying different models (Inception, ResNets, Xception) and some small architectures (MobileNet, SqueezeNet) and chose to use pretrained ResNet50 from Keras applications. After long time with it and no result more than 0.47 LB we switched to Inception networks. \n\nAll models were trained on single fold: 85% train and 15% test.\nWe splitted data using iterative stratification package from discussion.\nFrom external data we used only rare classes and excluded all duplicates from training.\n\n## Things that worked ##\n\n- F1 loss and BCE with weighted positive predictions\n- SGD with warm restarts (cosine learning rate, initial lr=0.1)\n- Log-dampened class weights\n- Oversampling of rare classes by 5x\n- Undersampling classes 0 and 25 by 0.5x\n- \"Standard\" augmentations (flip rl/ud, shift, rotate, shear, brightness)\n- Squeeze-Excitation module\n- Weight decay\n- Randomly dropping green channel and all classes in 1% of images led to +0.04 LB\n- BatchNorm running mean/variance update on test set\n\n## Things that did not work ##\n- Imagenet weights (we still wonder why)\n- Focal loss\n- Threshold fitting\n- GapNet-like things\n- Convolutional block attention module\n- Filling empty predictions with most popular class\n- Mixup, probably because of the specialty of the green channel\n- Heavy augmentations (blur, various types of noise, dropout, contrast, elastic transformations, sharpen/emboss)\n- Small crops (less than 40% of number of pixels)\n- AdamW\n- Global average + global max pooling\n- TTA (but not hurt)\n- BatchNorm in dense layers reduced performance dramatically\n\n## Things that we regret: ##\n- Realized the way we overfit (high recall, low precision) too late (2 days before the end of the competition)\n- Not using cross-validation\n- Not ensembling different architectures\n- Closer to final: using Keras\n\n## Top solution ##\nTop performing solution scored 0.505 on private LB. We didn't choose it because it performed worse both in local validation and on public LB. \nIt was an ensemble of all the best SE-BNInception (6 models) with fixed threshold. 4 of them were RGBY and 2 were RGB.\n\n## Hardware ##\nWe used primarily the server with one 1080Ti and 64GB of RAM.\nTo reduce disk bottleneck, we put all training data into ramdisk with plenty of swap (~3x load speedup).\nOccasionally we used standard workstations with 1060 and 8-16GB RAM.", "454651": "Thanks for sharing. We also tried different things for filling the empty predictions, but found nothing that worked", "1079465": "Hi, your solution was really impressive. I am new to Kaggle and just looking for some competitions for exercise. I wonder how you play with the squeeze-and-excitation structure with pretrained model from Keras, since Keras is well encapsulated and I wonder how to regularly insert layers of SEnet into pretrained networks like ResNet50. Plus if I insert SE layers into ResNet50, is it still appropriate to use the pretrained weights of it?\nI'll really appreciate it for your reply. Thanks a lot." }, "source": "meta" }