Congrats to all the winners and thanks to the host and Kaggle hosted such an interesting competition. Huge thank to my teammates you are the greatest!
\n\nOverview:\nAll our models were trained with 512x512x6 images. We sampled a random site for each sample per epoch. All our were 2-headed.\nFirst head: classification head had an output with 1139 neurons, \nSecond head: embeddings.
\n\nChallenges:\nValidation \nSpecific experiments like U2OS-4 failed
\n\nValidation for CNNs:\nWe noticed that specific experiment types perform very differently from others. We chose the hardest experiment for our model and built the validation based on it.
\n\nTraining Augmentations:\n- flips (horizontal, vertical)\n- rot90 \n- transpose\n- shift (0.25) with 101 reflections \n- rot360 (-180, 180)\n- cutout (32x32, 2 holes)\n- noise (gaussian, localvar, poisson, salt&pepper, speckle)\n- clahe\n- gamma (0.9,1.1)
\n\nData Pre-Processing:\nper channel (img - img.mean()) / (img.std() + 1e-6)\nWe noticed that due to heavy augmentations we had a true_division warning. This warning caused batchnorms to feel bad, so we did this trick to prevent zero division.
\n\nModel training:\nWe train our models with NVidia Apex and Pytorch 1.2 on 8x1080ti GPU server for 110 epochs. The training takes 1-3 days depending on the model.\nWe did not use oversample.\nBecause of the long iteration, we didn’t use any K-Fold training. We wanted to train it closer to the end of the competition, but we didn’t have much time.
\n\nOptimizer: SGD
\n\nScheduler: init LR 0.1, \n5 epochs warmup,\ndrop LR every 40 epochs with factor 0.1
\n\nLoss Functions:\nSmooth CrossEntropy for classification, Center Loss for embeddings.
\n\nModel:\nOur final model is Senet154 trained on train data + pseudo-labels.\nTo produce pseudo-labels were using an ensemble SeNet154, SeResnext50, SeResnext101, Polynet, EfficientNet-b6, ResNeXt101-wsl.
\n\nTest time augmentations:\noutput = (model(img1) + model(img2) + model(img1.flip(2)) + model(img1.flip(3)) + model(img2.flip(2)) + model(img2.flip(3)) / 6.
\n\nPost-processing:\nHungry experiment re-calibration and plate re-calibration\nhttps://www.kaggle.com/c/quickdraw-doodle-recognition/discussion/73803#latest-438270
\n\nWhat didn’t work or had about the same performance:\n- Mix-up, manifold mixup\n- ArcFace\n- Last stride 1\n- Focal Loss\n- XGBoost ensembling \n- Metric search based on the embeddings distances\n- GeM\n- Different optimizers like Adam, Radam, Ranger, etc.\n- ResNeXt101-wsl performed worse than expected.
", "messages": [ { "id": 635137, "postDate": "2019-09-27T07:35:08.843Z", "content": "Congrats to all the winners and thanks to the host and Kaggle hosted such an interesting competition. Huge thank to my teammates you are the greatest!
\n\nOverview:\nAll our models were trained with 512x512x6 images. We sampled a random site for each sample per epoch. All our were 2-headed.\nFirst head: classification head had an output with 1139 neurons, \nSecond head: embeddings.
\n\nChallenges:\nValidation \nSpecific experiments like U2OS-4 failed
\n\nValidation for CNNs:\nWe noticed that specific experiment types perform very differently from others. We chose the hardest experiment for our model and built the validation based on it.
\n\nTraining Augmentations:\n- flips (horizontal, vertical)\n- rot90 \n- transpose\n- shift (0.25) with 101 reflections \n- rot360 (-180, 180)\n- cutout (32x32, 2 holes)\n- noise (gaussian, localvar, poisson, salt&pepper, speckle)\n- clahe\n- gamma (0.9,1.1)
\n\nData Pre-Processing:\nper channel (img - img.mean()) / (img.std() + 1e-6)\nWe noticed that due to heavy augmentations we had a true_division warning. This warning caused batchnorms to feel bad, so we did this trick to prevent zero division.
\n\nModel training:\nWe train our models with NVidia Apex and Pytorch 1.2 on 8x1080ti GPU server for 110 epochs. The training takes 1-3 days depending on the model.\nWe did not use oversample.\nBecause of the long iteration, we didn’t use any K-Fold training. We wanted to train it closer to the end of the competition, but we didn’t have much time.
\n\nOptimizer: SGD
\n\nScheduler: init LR 0.1, \n5 epochs warmup,\ndrop LR every 40 epochs with factor 0.1
\n\nLoss Functions:\nSmooth CrossEntropy for classification, Center Loss for embeddings.
\n\nModel:\nOur final model is Senet154 trained on train data + pseudo-labels.\nTo produce pseudo-labels were using an ensemble SeNet154, SeResnext50, SeResnext101, Polynet, EfficientNet-b6, ResNeXt101-wsl.
\n\nTest time augmentations:\noutput = (model(img1) + model(img2) + model(img1.flip(2)) + model(img1.flip(3)) + model(img2.flip(2)) + model(img2.flip(3)) / 6.
\n\nPost-processing:\nHungry experiment re-calibration and plate re-calibration\nhttps://www.kaggle.com/c/quickdraw-doodle-recognition/discussion/73803#latest-438270
\n\nWhat didn’t work or had about the same performance:\n- Mix-up, manifold mixup\n- ArcFace\n- Last stride 1\n- Focal Loss\n- XGBoost ensembling \n- Metric search based on the embeddings distances\n- GeM\n- Different optimizers like Adam, Radam, Ranger, etc.\n- ResNeXt101-wsl performed worse than expected.
", "rawMarkdown": "Congrats to all the winners and thanks to the host and Kaggle hosted such an interesting competition. Huge thank to my teammates you are the greatest!\n\n**Overview:**\nAll our models were trained with 512x512x6 images. We sampled a random site for each sample per epoch. All our were 2-headed.\nFirst head: classification head had an output with 1139 neurons, \nSecond head: embeddings.\n\n**Challenges:**\nValidation \nSpecific experiments like U2OS-4 failed \n\n**Validation for CNNs:**\nWe noticed that specific experiment types perform very differently from others. We chose the hardest experiment for our model and built the validation based on it. \n\n**Training Augmentations:**\n- flips (horizontal, vertical)\n- rot90 \n- transpose\n- shift (0.25) with 101 reflections \n- rot360 (-180, 180)\n- cutout (32x32, 2 holes)\n- noise (gaussian, localvar, poisson, salt&pepper, speckle)\n- clahe\n- gamma (0.9,1.1)\n\n**Data Pre-Processing:**\nper channel (img - img.mean()) / (img.std() + 1e-6)\nWe noticed that due to heavy augmentations we had a true_division warning. This warning caused batchnorms to feel bad, so we did this trick to prevent zero division.\n\n**Model training:**\nWe train our models with NVidia Apex and Pytorch 1.2 on 8x1080ti GPU server for 110 epochs. The training takes 1-3 days depending on the model.\nWe did not use oversample.\nBecause of the long iteration, we didn’t use any K-Fold training. We wanted to train it closer to the end of the competition, but we didn’t have much time. \n\n**Optimizer:** SGD\n\n**Scheduler:** init LR 0.1, \n5 epochs warmup,\ndrop LR every 40 epochs with factor 0.1\n\n**Loss Functions:**\nSmooth CrossEntropy for classification, Center Loss for embeddings. \n\n**Model:**\nOur final model is Senet154 trained on train data + pseudo-labels.\nTo produce pseudo-labels were using an ensemble SeNet154, SeResnext50, SeResnext101, Polynet, EfficientNet-b6, ResNeXt101-wsl. \n\n**Test time augmentations:**\noutput = (model(img1) + model(img2) + model(img1.flip(2)) + model(img1.flip(3)) + model(img2.flip(2)) + model(img2.flip(3)) / 6.\n\n**Post-processing:**\nHungry experiment re-calibration and plate re-calibration\nhttps://www.kaggle.com/c/quickdraw-doodle-recognition/discussion/73803#latest-438270 \n\n**What didn’t work or had about the same performance:**\n- Mix-up, manifold mixup\n- ArcFace\n- Last stride 1\n- Focal Loss\n- XGBoost ensembling \n- Metric search based on the embeddings distances\n- GeM\n- Different optimizers like Adam, Radam, Ranger, etc.\n- ResNeXt101-wsl performed worse than expected. \n", "votes": 30 }, { "id": 635197, "postDate": "2019-09-27T08:46:49.253Z", "content": "Smooth CrossEntropy for classification, Center Loss for embeddings.\nSo how do you infer predictions? How do you use those embeddings?
And how do you select pseudo-labels?
", "rawMarkdown": "`Smooth CrossEntropy for classification, Center Loss for embeddings.`\nSo how do you infer predictions? How do you use those embeddings?\n\nAnd how do you select pseudo-labels?", "votes": 1, "replies": [ { "id": 635227, "postDate": "2019-09-27T09:23:21.570Z", "content": "We wanted to find the nearest neighbor based on these embedding (and maybe to combine with classification part), but we didn’t succeed with re-calibration (1 class per experiment) based on the distances.
\n\nSo center loss was just a regularizer for the model’a embeddings.
\n\nWe used only logits from model’s linear output.
", "rawMarkdown": "We wanted to find the nearest neighbor based on these embedding (and maybe to combine with classification part), but we didn’t succeed with re-calibration (1 class per experiment) based on the distances. \n\nSo center loss was just a regularizer for the model’a embeddings. \n\nWe used only logits from model’s linear output. ", "votes": 2 }, { "id": 635301, "postDate": "2019-09-27T10:48:13.597Z", "content": "Regarding pseudo-labels, we didn't apply any methods to select them. \nWe added all of them after re-calibration to our train set and trained with Smooth-CE loss
", "rawMarkdown": "Regarding pseudo-labels, we didn't apply any methods to select them. \nWe added all of them after re-calibration to our train set and trained with Smooth-CE loss", "votes": 1 } ] }, { "id": 635146, "postDate": "2019-09-27T07:44:41.583Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 635197, "author_name": "Artyom Palvelev", "author_url": "", "post_date": "2019-09-27T08:46:49.253000", "content": "Smooth CrossEntropy for classification, Center Loss for embeddings.\nSo how do you infer predictions? How do you use those embeddings?
And how do you select pseudo-labels?
", "votes": 1, "replies": [ { "id": 635227, "author_name": "Igor Krashenyi", "author_url": "", "post_date": "2019-09-27T09:23:21.570000", "content": "We wanted to find the nearest neighbor based on these embedding (and maybe to combine with classification part), but we didn’t succeed with re-calibration (1 class per experiment) based on the distances.
\n\nSo center loss was just a regularizer for the model’a embeddings.
\n\nWe used only logits from model’s linear output.
", "votes": 2, "replies": [] }, { "id": 635301, "author_name": "Igor Krashenyi", "author_url": "", "post_date": "2019-09-27T10:48:13.597000", "content": "Regarding pseudo-labels, we didn't apply any methods to select them. \nWe added all of them after re-calibration to our train set and trained with Smooth-CE loss
", "votes": 1, "replies": [] } ] }, { "id": 635146, "author_name": "", "author_url": "", "post_date": "2019-09-27T07:44:41.583000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "635137": "Congrats to all the winners and thanks to the host and Kaggle hosted such an interesting competition. Huge thank to my teammates you are the greatest!\n\n**Overview:**\nAll our models were trained with 512x512x6 images. We sampled a random site for each sample per epoch. All our were 2-headed.\nFirst head: classification head had an output with 1139 neurons, \nSecond head: embeddings.\n\n**Challenges:**\nValidation \nSpecific experiments like U2OS-4 failed \n\n**Validation for CNNs:**\nWe noticed that specific experiment types perform very differently from others. We chose the hardest experiment for our model and built the validation based on it. \n\n**Training Augmentations:**\n- flips (horizontal, vertical)\n- rot90 \n- transpose\n- shift (0.25) with 101 reflections \n- rot360 (-180, 180)\n- cutout (32x32, 2 holes)\n- noise (gaussian, localvar, poisson, salt&pepper, speckle)\n- clahe\n- gamma (0.9,1.1)\n\n**Data Pre-Processing:**\nper channel (img - img.mean()) / (img.std() + 1e-6)\nWe noticed that due to heavy augmentations we had a true_division warning. This warning caused batchnorms to feel bad, so we did this trick to prevent zero division.\n\n**Model training:**\nWe train our models with NVidia Apex and Pytorch 1.2 on 8x1080ti GPU server for 110 epochs. The training takes 1-3 days depending on the model.\nWe did not use oversample.\nBecause of the long iteration, we didn’t use any K-Fold training. We wanted to train it closer to the end of the competition, but we didn’t have much time. \n\n**Optimizer:** SGD\n\n**Scheduler:** init LR 0.1, \n5 epochs warmup,\ndrop LR every 40 epochs with factor 0.1\n\n**Loss Functions:**\nSmooth CrossEntropy for classification, Center Loss for embeddings. \n\n**Model:**\nOur final model is Senet154 trained on train data + pseudo-labels.\nTo produce pseudo-labels were using an ensemble SeNet154, SeResnext50, SeResnext101, Polynet, EfficientNet-b6, ResNeXt101-wsl. \n\n**Test time augmentations:**\noutput = (model(img1) + model(img2) + model(img1.flip(2)) + model(img1.flip(3)) + model(img2.flip(2)) + model(img2.flip(3)) / 6.\n\n**Post-processing:**\nHungry experiment re-calibration and plate re-calibration\nhttps://www.kaggle.com/c/quickdraw-doodle-recognition/discussion/73803#latest-438270 \n\n**What didn’t work or had about the same performance:**\n- Mix-up, manifold mixup\n- ArcFace\n- Last stride 1\n- Focal Loss\n- XGBoost ensembling \n- Metric search based on the embeddings distances\n- GeM\n- Different optimizers like Adam, Radam, Ranger, etc.\n- ResNeXt101-wsl performed worse than expected. \n", "635197": "`Smooth CrossEntropy for classification, Center Loss for embeddings.`\nSo how do you infer predictions? How do you use those embeddings?\n\nAnd how do you select pseudo-labels?", "635146": "" } }