![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F362676%2F318492d44eeea2d6e21b5ec53a34ffac%2FScreenshot%20from%202019-09-24%2020-27-00.png?generation=1569525109617780&alt=media\")
Dear all, thank you for excellent spirit of competition and astonishing results! Without such favorable pressure we wouldn’t be able to achieve such scores.
\n\nI will split description into two main parts: pipeline and prediction with post-processing.
\n\nPipeline:\n- 6-channel images.\n- Super Resolution. During exploratory data analysis we suddenly realize that the size of individual cells is rather small and convolutional layer might have a hard time learning all patterns. So we use simple bicubic superresolution technique. That really boost our scores. We checked up to 960 * 960. We have tried to use GAN’s (ESRGAN in particular) for this task but its performance was worse compared to vanilla bicubic interpolation.\n- 3Fold by experiments.\n- Add all controls to train. We tried to add controls only from train experiments, but it didn’t work.\n- Mixed Precision: Due to the large size of images we decided to use only Mixed Precision learning because of inevitable graphics card’s memory shortage.\n- Gradual layer unfreezing: We found out that gradual layer unfreezing was crucial condition for our models not to diverge. First 5 epochs we unfreeze 20% of layers from head each epoch. Presumably it was because of Mixed Precision - some sort of gradient explosion or something similar.\n- Metric learning - CosFace loss: We took my favorite metric learning loss. We also tried ArcLoss and AdaCos but they fail to converge or achieve worse results in Mixed precision mode.\n- Cyclic Linear Lr Annealing.\n- Cyclic Linear Scale annealing: This idea is inspired by AdaCos. https://arxiv.org/abs/1905.00292 where authors decrease scale as model achieve high scores. S_max = 64, S_min = 24.\n- Classes for distinct cell types are distinct as well: So we predict vector of 1108 * 4 + control classes. It improved convergence drastically.\n
Unsupervised Domain Adaptation by Backpropagation: https://arxiv.org/abs/1409.7495. We took plate and cell type as domain label and schedule gradient reversal layer coefficient as lr and Scale for CosFace. \n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F362676%2F81d59df47d3bf1faf271f11d74245223%2FScreenshot%20from%202019-09-24%2020-55-48.png?generation=1569525156026046&alt=media\")
Backbones: seresnet50 (img_size up to 960), senet154 (img_size up to 672), xception (img_size up to 860), inceptionresnetv2 (img_size up to 512), densenet161 (img_size up to 840), densenet201 (img_size up to 720).
Prediction:\n- Prediction balancing. This technique can be used when classes are equally balanced. Excellent implementation: https://www.kaggle.com/c/quickdraw-doodle-recognition/discussion/73803#latest-438270.\n- Progressive prediction. Idea is simple: for every experiment each class can be presented only once. So if we predict some class with high probability there is a little chance that this class can be predicted further with lower probability. We take 50% most probable classes, remove them from consideration and predict the next sample and so on.\n- Leak.
\n\nWhat didn’t work:\n- Mix-up\n- ArcFace\n- AdaCos\n- Deformable ConvNet - https://arxiv.org/pdf/1811.11168.pdf\n- Deep Supervision\n- Weight Standardisation\n- Concatenation of two sites together to make 12-channel image
", "messages": [ { "id": 635088, "postDate": "2019-09-27T06:34:51.840Z", "content": "Dear all, thank you for excellent spirit of competition and astonishing results! Without such favorable pressure we wouldn’t be able to achieve such scores.
\n\nI will split description into two main parts: pipeline and prediction with post-processing.
\n\nPipeline:\n- 6-channel images.\n- Super Resolution. During exploratory data analysis we suddenly realize that the size of individual cells is rather small and convolutional layer might have a hard time learning all patterns. So we use simple bicubic superresolution technique. That really boost our scores. We checked up to 960 * 960. We have tried to use GAN’s (ESRGAN in particular) for this task but its performance was worse compared to vanilla bicubic interpolation.\n- 3Fold by experiments.\n- Add all controls to train. We tried to add controls only from train experiments, but it didn’t work.\n- Mixed Precision: Due to the large size of images we decided to use only Mixed Precision learning because of inevitable graphics card’s memory shortage.\n- Gradual layer unfreezing: We found out that gradual layer unfreezing was crucial condition for our models not to diverge. First 5 epochs we unfreeze 20% of layers from head each epoch. Presumably it was because of Mixed Precision - some sort of gradient explosion or something similar.\n- Metric learning - CosFace loss: We took my favorite metric learning loss. We also tried ArcLoss and AdaCos but they fail to converge or achieve worse results in Mixed precision mode.\n- Cyclic Linear Lr Annealing.\n- Cyclic Linear Scale annealing: This idea is inspired by AdaCos. https://arxiv.org/abs/1905.00292 where authors decrease scale as model achieve high scores. S_max = 64, S_min = 24.\n- Classes for distinct cell types are distinct as well: So we predict vector of 1108 * 4 + control classes. It improved convergence drastically.\n
Unsupervised Domain Adaptation by Backpropagation: https://arxiv.org/abs/1409.7495. We took plate and cell type as domain label and schedule gradient reversal layer coefficient as lr and Scale for CosFace. \n
Backbones: seresnet50 (img_size up to 960), senet154 (img_size up to 672), xception (img_size up to 860), inceptionresnetv2 (img_size up to 512), densenet161 (img_size up to 840), densenet201 (img_size up to 720).
Prediction:\n- Prediction balancing. This technique can be used when classes are equally balanced. Excellent implementation: https://www.kaggle.com/c/quickdraw-doodle-recognition/discussion/73803#latest-438270.\n- Progressive prediction. Idea is simple: for every experiment each class can be presented only once. So if we predict some class with high probability there is a little chance that this class can be predicted further with lower probability. We take 50% most probable classes, remove them from consideration and predict the next sample and so on.\n- Leak.
\n\nWhat didn’t work:\n- Mix-up\n- ArcFace\n- AdaCos\n- Deformable ConvNet - https://arxiv.org/pdf/1811.11168.pdf\n- Deep Supervision\n- Weight Standardisation\n- Concatenation of two sites together to make 12-channel image
", "rawMarkdown": "Dear all, thank you for excellent spirit of competition and astonishing results! Without such favorable pressure we wouldn’t be able to achieve such scores.\n\nI will split description into two main parts: pipeline and prediction with post-processing.\n\n**Pipeline:**\n- **6-channel images.**\n- **Super Resolution.** During exploratory data analysis we suddenly realize that the size of individual cells is rather small and convolutional layer might have a hard time learning all patterns. So we use simple bicubic superresolution technique. That really boost our scores. We checked up to 960 * 960. We have tried to use GAN’s (ESRGAN in particular) for this task but its performance was worse compared to vanilla bicubic interpolation.\n- **3Fold by experiments.**\n- **Add all controls to train.** We tried to add controls only from train experiments, but it didn’t work.\n- **Mixed Precision:** Due to the large size of images we decided to use only Mixed Precision learning because of inevitable graphics card’s memory shortage.\n- **Gradual layer unfreezing:** We found out that gradual layer unfreezing was crucial condition for our models not to diverge. First 5 epochs we unfreeze 20% of layers from head each epoch. Presumably it was because of Mixed Precision - some sort of gradient explosion or something similar.\n- **Metric learning - CosFace loss:** We took my favorite metric learning loss. We also tried ArcLoss and AdaCos but they fail to converge or achieve worse results in Mixed precision mode.\n- **Cyclic Linear Lr Annealing.**\n- **Cyclic Linear Scale annealing:** This idea is inspired by AdaCos. https://arxiv.org/abs/1905.00292 where authors decrease scale as model achieve high scores. S_max = 64, S_min = 24.\n- **Classes for distinct cell types are distinct as well:** So we predict vector of 1108 * 4 + control classes. It improved convergence drastically.\n\n\n- **Unsupervised Domain Adaptation by Backpropagation:** https://arxiv.org/abs/1409.7495. We took plate and cell type as domain label and schedule gradient reversal layer coefficient as lr and Scale for CosFace. \n\n\n- **Backbones:** seresnet50 (img_size up to 960), senet154 (img_size up to 672), xception (img_size up to 860), inceptionresnetv2 (img_size up to 512), densenet161 (img_size up to 840), densenet201 (img_size up to 720). \n- SGD optimizer with momentum. We tried Adam, Lookahead and Ranger but they didn’t improve validation score.\n- Gradient normalization.\n- Batch size = 16 with accumulation up to 32.\n- Focal loss: gamma = 32.\n\n\n**Prediction:**\n- **Prediction balancing.** This technique can be used when classes are equally balanced. Excellent implementation: https://www.kaggle.com/c/quickdraw-doodle-recognition/discussion/73803#latest-438270.\n- **Progressive prediction.** Idea is simple: for every experiment each class can be presented only once. So if we predict some class with high probability there is a little chance that this class can be predicted further with lower probability. We take 50% most probable classes, remove them from consideration and predict the next sample and so on.\n- **Leak.**\n\n\n**What didn’t work:**\n- Mix-up\n- ArcFace\n- AdaCos\n- Deformable ConvNet - https://arxiv.org/pdf/1811.11168.pdf\n- Deep Supervision\n- Weight Standardisation\n- Concatenation of two sites together to make 12-channel image\n\n", "votes": 39 }, { "id": 635102, "postDate": "2019-09-27T06:55:10.760Z", "content": "Thank you so much for your writeup. \nI havent worked with metric learning much. \nSorry if it is a stupid question. How can you have softmax probability from metric learning model so that you can apply balance technique?
", "rawMarkdown": "Thank you so much for your writeup. \nI havent worked with metric learning much. \nSorry if it is a stupid question. How can you have softmax probability from metric learning model so that you can apply balance technique?", "votes": 1, "replies": [ { "id": 635108, "postDate": "2019-09-27T07:02:35.673Z", "content": "@backaggle, [Name]Face block outputs probabilities that further goes to cross-entropy loss. So you can use embeddings or probabilities from [Name]Face block.
", "rawMarkdown": "@backaggle, [Name]Face block outputs probabilities that further goes to cross-entropy loss. So you can use embeddings or probabilities from [Name]Face block. ", "votes": 1 } ] }, { "id": 635123, "postDate": "2019-09-27T07:19:12.960Z", "content": "Congratulations to your team on winning gold medal position and being well stable in LB. Is it possible for you to open-source the code solution, even the baseline structure? I would like to recreate and re-run it. many thanks again!
", "rawMarkdown": "Congratulations to your team on winning gold medal position and being well stable in LB. Is it possible for you to open-source the code solution, even the baseline structure? I would like to recreate and re-run it. many thanks again!" }, { "id": 635147, "postDate": "2019-09-27T07:45:41.667Z", "rawMarkdown": "", "isDeleted": true }, { "id": 635145, "postDate": "2019-09-27T07:44:39.697Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 635102, "author_name": "cab", "author_url": "", "post_date": "2019-09-27T06:55:10.760000", "content": "Thank you so much for your writeup. \nI havent worked with metric learning much. \nSorry if it is a stupid question. How can you have softmax probability from metric learning model so that you can apply balance technique?
", "votes": 1, "replies": [ { "id": 635108, "author_name": "Ivan Sosin", "author_url": "", "post_date": "2019-09-27T07:02:35.673000", "content": "@backaggle, [Name]Face block outputs probabilities that further goes to cross-entropy loss. So you can use embeddings or probabilities from [Name]Face block.
", "votes": 1, "replies": [] } ] }, { "id": 635123, "author_name": "FGPC", "author_url": "", "post_date": "2019-09-27T07:19:12.960000", "content": "Congratulations to your team on winning gold medal position and being well stable in LB. Is it possible for you to open-source the code solution, even the baseline structure? I would like to recreate and re-run it. many thanks again!
", "votes": 0, "replies": [] }, { "id": 635147, "author_name": "", "author_url": "", "post_date": "2019-09-27T07:45:41.667000", "content": "", "votes": 0, "replies": [] }, { "id": 635145, "author_name": "", "author_url": "", "post_date": "2019-09-27T07:44:39.697000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "635088": "Dear all, thank you for excellent spirit of competition and astonishing results! Without such favorable pressure we wouldn’t be able to achieve such scores.\n\nI will split description into two main parts: pipeline and prediction with post-processing.\n\n**Pipeline:**\n- **6-channel images.**\n- **Super Resolution.** During exploratory data analysis we suddenly realize that the size of individual cells is rather small and convolutional layer might have a hard time learning all patterns. So we use simple bicubic superresolution technique. That really boost our scores. We checked up to 960 * 960. We have tried to use GAN’s (ESRGAN in particular) for this task but its performance was worse compared to vanilla bicubic interpolation.\n- **3Fold by experiments.**\n- **Add all controls to train.** We tried to add controls only from train experiments, but it didn’t work.\n- **Mixed Precision:** Due to the large size of images we decided to use only Mixed Precision learning because of inevitable graphics card’s memory shortage.\n- **Gradual layer unfreezing:** We found out that gradual layer unfreezing was crucial condition for our models not to diverge. First 5 epochs we unfreeze 20% of layers from head each epoch. Presumably it was because of Mixed Precision - some sort of gradient explosion or something similar.\n- **Metric learning - CosFace loss:** We took my favorite metric learning loss. We also tried ArcLoss and AdaCos but they fail to converge or achieve worse results in Mixed precision mode.\n- **Cyclic Linear Lr Annealing.**\n- **Cyclic Linear Scale annealing:** This idea is inspired by AdaCos. https://arxiv.org/abs/1905.00292 where authors decrease scale as model achieve high scores. S_max = 64, S_min = 24.\n- **Classes for distinct cell types are distinct as well:** So we predict vector of 1108 * 4 + control classes. It improved convergence drastically.\n\n\n- **Unsupervised Domain Adaptation by Backpropagation:** https://arxiv.org/abs/1409.7495. We took plate and cell type as domain label and schedule gradient reversal layer coefficient as lr and Scale for CosFace. \n\n\n- **Backbones:** seresnet50 (img_size up to 960), senet154 (img_size up to 672), xception (img_size up to 860), inceptionresnetv2 (img_size up to 512), densenet161 (img_size up to 840), densenet201 (img_size up to 720). \n- SGD optimizer with momentum. We tried Adam, Lookahead and Ranger but they didn’t improve validation score.\n- Gradient normalization.\n- Batch size = 16 with accumulation up to 32.\n- Focal loss: gamma = 32.\n\n\n**Prediction:**\n- **Prediction balancing.** This technique can be used when classes are equally balanced. Excellent implementation: https://www.kaggle.com/c/quickdraw-doodle-recognition/discussion/73803#latest-438270.\n- **Progressive prediction.** Idea is simple: for every experiment each class can be presented only once. So if we predict some class with high probability there is a little chance that this class can be predicted further with lower probability. We take 50% most probable classes, remove them from consideration and predict the next sample and so on.\n- **Leak.**\n\n\n**What didn’t work:**\n- Mix-up\n- ArcFace\n- AdaCos\n- Deformable ConvNet - https://arxiv.org/pdf/1811.11168.pdf\n- Deep Supervision\n- Weight Standardisation\n- Concatenation of two sites together to make 12-channel image\n\n", "635102": "Thank you so much for your writeup. \nI havent worked with metric learning much. \nSorry if it is a stupid question. How can you have softmax probability from metric learning model so that you can apply balance technique?", "635123": "Congratulations to your team on winning gold medal position and being well stable in LB. Is it possible for you to open-source the code solution, even the baseline structure? I would like to recreate and re-run it. many thanks again!", "635147": "", "635145": "" } }