{
"id": 110998,
"title": "117th place solution [0.718] using PyTorch & fastai",
"url": "/competitions/recursion-cellular-image-classification/writeups/first-team-117th-place-solution-0-718-using-pytorc",
"author_name": "",
"post_date": "2019-10-18T19:15:35.120Z",
"votes": 1,
"comment_count": 2,
"views": 0,
"content": "A big thank you goes out to the competition organizers and they great community!
\n\nThis was my first kaggle competition I joined from the beginning and I learned a lot! :-D
\n\nFinal setup for highest score:\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1686473%2F0b3b7757f33b40bd54f75953d3d49659%2Frcic_network.png?generation=1570045807295295&alt=media\")
\n- ImageNet-pretained EfficientNet-B3 with 6D input stage (2x 3D weights copied to get 6D input stage at the beginning of training).\n- AdaCos metric learning approach was implemented, which should be more stable without hyperparameter tuning when compared to other cosine-based softmax losses.\n- Standard data augmentation (flip vertical/horizontal, rotation, zoom, lighting, warp) with custom implementation of „Color Augmentation for histopathology images”. Data visualization function displaying the 6D images as 3D+3D image:\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1686473%2F83c27c07dc7a61366fe8bc1a9f3bfbbe%2Frcic_data.png?generation=1570045826942943&alt=media\")
\n- Embedding layers for meta data to let the model learn from celltype, plategroup, and experiment data (i.e., data leak). (However, there was no time in the end to optimize the embeddings used for the full training and test predictions. Something like a similarity search for the test experiments would have been an interesting approach.)\n- CutMix was used for the image data for some parts of the training. In addition, a linear interpolation of the embeddings was used during CutMix training.\n- Predictions were carried out with 3x TTA to obtain the feature embeddings of the image and meta data input.\n- Finally, FAISS was used for cosine similarity search of the extracted feature embeddings.
\n\nGeneral training setup:\n1. Train new network output stage/head with frozen pretrained feature extractor network.\n1. Retrain pretrained extractor network with new head (approx. 60 epochs).\n1. Add meta data embeddings, freeze feature extractor network and train embeddings and output stage (approx. 50 epochs).\n1. Train entire network (approx. 150 epochs).
\n\nFor all training stages the 1cycle policy was used for several cycles. EfficientNet training was carried out with AdamW. CutMix was used for the late training stages.
\n\nNotebooks from the GitHub repo:\n- EfficientNet-B3 training:\n - Pretraining\n - Complex network\n - Complex network 12D (2x 6 channel images)\n- EfficientNet-B4 (training not finished):\n - Pretraining\n - Complex network 12D (2x 6 channel images)
\n\nWhat didn’t work out or had no impact over a more basic setup:\n- Separate classifiers for celltypes and/or plategroups (i.e., siRNA groups of 277).\n- Some domain adaption techniques.\n- Mean or k-means clustering of siRNA feature embeddings for cosine similarity search.
\n\nLearnings:\n- A lot about PyTorch, fastai, and adjacent DL/ML topics.\n- Stay longer on the smaller model side, do more A/B or CV testing of improvements, then first really over-fit, and then try to reduce it.\n- EfficientNet-B3 retraining took forever and training of EfficientNet-B4 was not finished at the competition deadline (however, this looked very promising).
\n\nIf you have any questions or are interested in further information, please feel free to contact me! :-)
\n\nThank you very much!
",
"messages": [
{
"id": "639139",
"postDate": "10/02/2019 19:59:04",
"content": "A big thank you goes out to the competition organizers and they great community!
\n\nThis was my first kaggle competition I joined from the beginning and I learned a lot! :-D
\n\nFinal setup for highest score:\n\n- ImageNet-pretained EfficientNet-B3 with 6D input stage (2x 3D weights copied to get 6D input stage at the beginning of training).\n- AdaCos metric learning approach was implemented, which should be more stable without hyperparameter tuning when compared to other cosine-based softmax losses.\n- Standard data augmentation (flip vertical/horizontal, rotation, zoom, lighting, warp) with custom implementation of „Color Augmentation for histopathology images”. Data visualization function displaying the 6D images as 3D+3D image:\n\n- Embedding layers for meta data to let the model learn from celltype, plategroup, and experiment data (i.e., data leak). (However, there was no time in the end to optimize the embeddings used for the full training and test predictions. Something like a similarity search for the test experiments would have been an interesting approach.)\n- CutMix was used for the image data for some parts of the training. In addition, a linear interpolation of the embeddings was used during CutMix training.\n- Predictions were carried out with 3x TTA to obtain the feature embeddings of the image and meta data input.\n- Finally, FAISS was used for cosine similarity search of the extracted feature embeddings.
\n\nGeneral training setup:\n1. Train new network output stage/head with frozen pretrained feature extractor network.\n1. Retrain pretrained extractor network with new head (approx. 60 epochs).\n1. Add meta data embeddings, freeze feature extractor network and train embeddings and output stage (approx. 50 epochs).\n1. Train entire network (approx. 150 epochs).
\n\nFor all training stages the 1cycle policy was used for several cycles. EfficientNet training was carried out with AdamW. CutMix was used for the late training stages.
\n\nNotebooks from the GitHub repo:\n- EfficientNet-B3 training:\n - Pretraining\n - Complex network\n - Complex network 12D (2x 6 channel images)\n- EfficientNet-B4 (training not finished):\n - Pretraining\n - Complex network 12D (2x 6 channel images)
\n\nWhat didn’t work out or had no impact over a more basic setup:\n- Separate classifiers for celltypes and/or plategroups (i.e., siRNA groups of 277).\n- Some domain adaption techniques.\n- Mean or k-means clustering of siRNA feature embeddings for cosine similarity search.
\n\nLearnings:\n- A lot about PyTorch, fastai, and adjacent DL/ML topics.\n- Stay longer on the smaller model side, do more A/B or CV testing of improvements, then first really over-fit, and then try to reduce it.\n- EfficientNet-B3 retraining took forever and training of EfficientNet-B4 was not finished at the competition deadline (however, this looked very promising).
\n\nIf you have any questions or are interested in further information, please feel free to contact me! :-)
\n\nThank you very much!
",
"rawMarkdown": "*A big thank you goes out to the competition organizers and they great community!*\n\n*This was my first kaggle competition I joined from the beginning and I learned a lot! :-D*\n\n**Final setup for highest score:**\n\n- [ImageNet-pretained EfficientNet-B3](https://github.com/lukemelas/EfficientNet-PyTorch) with 6D input stage (2x 3D weights copied to get 6D input stage at the beginning of training).\n- [AdaCos metric learning approach](https://arxiv.org/abs/1905.00292) was implemented, which should be more stable without hyperparameter tuning when compared to other cosine-based softmax losses.\n- Standard data augmentation (flip vertical/horizontal, rotation, zoom, lighting, warp) with custom implementation of [„Color Augmentation for histopathology images”](https://arxiv.org/abs/1707.06183). Data visualization function displaying the 6D images as 3D+3D image:\n\n- Embedding layers for meta data to let the model learn from celltype, plategroup, and experiment data (i.e., data leak). (However, there was no time in the end to optimize the embeddings used for the full training and test predictions. Something like a similarity search for the test experiments would have been an interesting approach.)\n- [CutMix]( https://github.com/oguiza/fastai_extensions/tree/master/shared/0_image_data_augmentation) was used for the image data for some parts of the training. In addition, a linear interpolation of the embeddings was used during CutMix training.\n- Predictions were carried out with 3x TTA to obtain the feature embeddings of the image and meta data input.\n- Finally, [FAISS]( https://github.com/facebookresearch/faiss) was used for cosine similarity search of the extracted feature embeddings.\n\n**General training setup:**\n1. Train new network output stage/head with frozen pretrained feature extractor network.\n1. Retrain pretrained extractor network with new head (approx. 60 epochs).\n1. Add meta data embeddings, freeze feature extractor network and train embeddings and output stage (approx. 50 epochs).\n1. Train entire network (approx. 150 epochs).\n\nFor all training stages the [1cycle policy](https://sgugger.github.io/the-1cycle-policy.html) was used for several cycles. EfficientNet training was carried out with AdamW. CutMix was used for the late training stages.\n\n**[Notebooks from the GitHub repo](https://github.com/MicPie/recursion-cellular-image-classification/blob/master/README.md):**\n- EfficientNet-B3 training:\n - [Pretraining](https://github.com/MicPie/recursion-cellular-image-classification/blob/master/rcic_v20_efficientnet-b3.ipynb)\n - [Complex network](https://github.com/MicPie/recursion-cellular-image-classification/blob/master/rcic_v32_efficientnet-b3_celltype_plate-group-exp_Full.ipynb)\n - [Complex network 12D](https://github.com/MicPie/recursion-cellular-image-classification/blob/master/rcic_v41_efficientnet-b3_celltype_plate-group-exp_12D.ipynb) (2x 6 channel images)\n- EfficientNet-B4 (training not finished):\n - [Pretraining](https://github.com/MicPie/recursion-cellular-image-classification/blob/master/rcic_v43_efficientnet-b4_Mish_Ranger_LabelSmoothingLoss.ipynb)\n - [Complex network 12D](https://github.com/MicPie/recursion-cellular-image-classification/blob/master/rcic_v44_efficientnet-b4_celltype_plate-group-exp_12D_model-exchange.ipynb) (2x 6 channel images)\n\n**What didn’t work out or had no impact over a more basic setup:**\n- Separate classifiers for celltypes and/or plategroups (i.e., siRNA groups of 277).\n- Some domain adaption techniques.\n- Mean or k-means clustering of siRNA feature embeddings for cosine similarity search.\n\n**Learnings:**\n- A lot about PyTorch, fastai, and adjacent DL/ML topics.\n- Stay longer on the smaller model side, do more A/B or CV testing of improvements, then first really over-fit, and then try to reduce it.\n- EfficientNet-B3 retraining took forever and training of EfficientNet-B4 was not finished at the competition deadline (however, this looked very promising).\n\nIf you have any questions or are interested in further information, please feel free to contact me! :-)\n\nThank you very much!",
"votes": null
},
{
"id": "639341",
"postDate": "10/03/2019 05:11:16",
"content": "Congrats.....\nThank you for Sharing your Approach & Insights.....!! @micpie
",
"rawMarkdown": "Congrats.....\nThank you for Sharing your Approach & Insights.....!! @micpie",
"votes": null
},
{
"id": "786131",
"postDate": "03/25/2020 16:42:10",
"content": "I hadn't thought of using the data from both sites to classify or embedding the additional information on the images. Thanks for sharing your approach!
",
"rawMarkdown": "I hadn't thought of using the data from both sites to classify or embedding the additional information on the images. Thanks for sharing your approach!",
"votes": null
}
],
"comments": [
{
"id": 639341,
"author_name": "veeralakrishna",
"author_url": "",
"post_date": "10/03/2019 05:11:16",
"content": "Congrats.....\nThank you for Sharing your Approach & Insights.....!! @micpie
",
"votes": null,
"replies": []
},
{
"id": 786131,
"author_name": "skylerg022",
"author_url": "",
"post_date": "03/25/2020 16:42:10",
"content": "I hadn't thought of using the data from both sites to classify or embedding the additional information on the images. Thanks for sharing your approach!
",
"votes": null,
"replies": []
}
],
"raw_markdown_by_id": {
"639139": "*A big thank you goes out to the competition organizers and they great community!*\n\n*This was my first kaggle competition I joined from the beginning and I learned a lot! :-D*\n\n**Final setup for highest score:**\n\n- [ImageNet-pretained EfficientNet-B3](https://github.com/lukemelas/EfficientNet-PyTorch) with 6D input stage (2x 3D weights copied to get 6D input stage at the beginning of training).\n- [AdaCos metric learning approach](https://arxiv.org/abs/1905.00292) was implemented, which should be more stable without hyperparameter tuning when compared to other cosine-based softmax losses.\n- Standard data augmentation (flip vertical/horizontal, rotation, zoom, lighting, warp) with custom implementation of [„Color Augmentation for histopathology images”](https://arxiv.org/abs/1707.06183). Data visualization function displaying the 6D images as 3D+3D image:\n\n- Embedding layers for meta data to let the model learn from celltype, plategroup, and experiment data (i.e., data leak). (However, there was no time in the end to optimize the embeddings used for the full training and test predictions. Something like a similarity search for the test experiments would have been an interesting approach.)\n- [CutMix]( https://github.com/oguiza/fastai_extensions/tree/master/shared/0_image_data_augmentation) was used for the image data for some parts of the training. In addition, a linear interpolation of the embeddings was used during CutMix training.\n- Predictions were carried out with 3x TTA to obtain the feature embeddings of the image and meta data input.\n- Finally, [FAISS]( https://github.com/facebookresearch/faiss) was used for cosine similarity search of the extracted feature embeddings.\n\n**General training setup:**\n1. Train new network output stage/head with frozen pretrained feature extractor network.\n1. Retrain pretrained extractor network with new head (approx. 60 epochs).\n1. Add meta data embeddings, freeze feature extractor network and train embeddings and output stage (approx. 50 epochs).\n1. Train entire network (approx. 150 epochs).\n\nFor all training stages the [1cycle policy](https://sgugger.github.io/the-1cycle-policy.html) was used for several cycles. EfficientNet training was carried out with AdamW. CutMix was used for the late training stages.\n\n**[Notebooks from the GitHub repo](https://github.com/MicPie/recursion-cellular-image-classification/blob/master/README.md):**\n- EfficientNet-B3 training:\n - [Pretraining](https://github.com/MicPie/recursion-cellular-image-classification/blob/master/rcic_v20_efficientnet-b3.ipynb)\n - [Complex network](https://github.com/MicPie/recursion-cellular-image-classification/blob/master/rcic_v32_efficientnet-b3_celltype_plate-group-exp_Full.ipynb)\n - [Complex network 12D](https://github.com/MicPie/recursion-cellular-image-classification/blob/master/rcic_v41_efficientnet-b3_celltype_plate-group-exp_12D.ipynb) (2x 6 channel images)\n- EfficientNet-B4 (training not finished):\n - [Pretraining](https://github.com/MicPie/recursion-cellular-image-classification/blob/master/rcic_v43_efficientnet-b4_Mish_Ranger_LabelSmoothingLoss.ipynb)\n - [Complex network 12D](https://github.com/MicPie/recursion-cellular-image-classification/blob/master/rcic_v44_efficientnet-b4_celltype_plate-group-exp_12D_model-exchange.ipynb) (2x 6 channel images)\n\n**What didn’t work out or had no impact over a more basic setup:**\n- Separate classifiers for celltypes and/or plategroups (i.e., siRNA groups of 277).\n- Some domain adaption techniques.\n- Mean or k-means clustering of siRNA feature embeddings for cosine similarity search.\n\n**Learnings:**\n- A lot about PyTorch, fastai, and adjacent DL/ML topics.\n- Stay longer on the smaller model side, do more A/B or CV testing of improvements, then first really over-fit, and then try to reduce it.\n- EfficientNet-B3 retraining took forever and training of EfficientNet-B4 was not finished at the competition deadline (however, this looked very promising).\n\nIf you have any questions or are interested in further information, please feel free to contact me! :-)\n\nThank you very much!",
"639341": "Congrats.....\nThank you for Sharing your Approach & Insights.....!! @micpie",
"786131": "I hadn't thought of using the data from both sites to classify or embedding the additional information on the images. Thanks for sharing your approach!"
},
"source": "meta"
}