I don’t know about other people, but when I saw $1,000 in my inbox I had Christmas Morning moment. Free GCP credits are extremely important for our community. Many thanks to Google & Kaggle!
\n\nEnsemble of 3 Deep CNNs: ResNet50, InceptionV3, and InceptionResNetV2.
\n\nWe expected test images to come from different regions. So there were no doubts that augmentation and ensembling would be the most important directions to make robust models. But later when detailed EDA showed substantial difference in train/test label distributions it became clear that tuning models using tuning set would also play very important role.
\n\nTo get more diversity for ensemble I used following approaches:
\n\nOf course all models were trained from scratch according to the rules. I think it was nice because there was no those usual pretrained model search rush :) Scores below are from Stage 1 LB.
\n\nI trained 3 models described above, tuned them, predicted original test images, predicted mirrored test images, and computed weighted average.
\n\nrotate(np.random.randint(-30, 30)), then apply hard augmentation (short range or even two possible values) during another several epochs: e.g. rotate(np.random.choice([-30, 30])).I don’t know about other people, but when I saw $1,000 in my inbox I had Christmas Morning moment. Free GCP credits are extremely important for our community. Many thanks to Google & Kaggle!
\n\nEnsemble of 3 Deep CNNs: ResNet50, InceptionV3, and InceptionResNetV2.
\n\nWe expected test images to come from different regions. So there were no doubts that augmentation and ensembling would be the most important directions to make robust models. But later when detailed EDA showed substantial difference in train/test label distributions it became clear that tuning models using tuning set would also play very important role.
\n\nTo get more diversity for ensemble I used following approaches:
\n\nOf course all models were trained from scratch according to the rules. I think it was nice because there was no those usual pretrained model search rush :) Scores below are from Stage 1 LB.
\n\nI trained 3 models described above, tuned them, predicted original test images, predicted mirrored test images, and computed weighted average.
\n\nrotate(np.random.randint(-30, 30)), then apply hard augmentation (short range or even two possible values) during another several epochs: e.g. rotate(np.random.choice([-30, 30])).Congrats and thanks for sharing.
", "rawMarkdown": "Congrats and thanks for sharing.", "votes": null }, { "id": "433005", "postDate": "12/04/2018 14:58:37", "content": "Congratulations and thank you for sharing this overview
", "rawMarkdown": "Congratulations and thank you for sharing this overview", "votes": null } ], "comments": [ { "id": 426533, "author_name": "sheriytm", "author_url": "", "post_date": "11/23/2018 12:01:15", "content": "Congrats and thanks for sharing.
", "votes": null, "replies": [] }, { "id": 433005, "author_name": "ashishvarier", "author_url": "", "post_date": "12/04/2018 14:58:37", "content": "Congratulations and thank you for sharing this overview
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "426507": "I don’t know about other people, but when I saw $1,000 in my inbox I had Christmas Morning moment. Free GCP credits are extremely important for our community. Many thanks to Google & Kaggle!\n\n## Solution\nEnsemble of 3 Deep CNNs: ResNet50, InceptionV3, and InceptionResNetV2.\n\n## Concept\nWe expected test images to come from different regions. So there were no doubts that augmentation and ensembling would be the most important directions to make robust models. But later when detailed EDA showed substantial difference in train/test label distributions it became clear that tuning models using tuning set would also play very important role. \n\n## Diversity\nTo get more diversity for ensemble I used following approaches: \n\n1. Different CNN architectures\n2. Different image sizes\n3. Train with/without augmentation\n4. Different augmentation schedules\n\n## Models\nOf course all models were trained from scratch according to the rules. I think it was nice because there was no those usual pretrained model search rush :) Scores below are from Stage 1 LB.\n\n1. ResNet50, 57 epochs, 224x224, adam 0.001, batch 64, increasing augmentation schedule, 4 hours per epoch on 1x P100, 0.311 without tuning, 0.510 with tuning\n2. InceptionV3, 25 epochs, 400x400, adam 0.001, batch 32, increasing augmentation schedule, 9 hours per epoch on 1x P100, 0.327 without tuning, 0.545 with tuning\n3. InceptionResNetV2, 8 epochs, 598x598, adam 0.001, batch 12, no augmentation, 38 hours per epoch on 1x P100, 0.318 without tuning, 0.550 with tuning\n\n## Final submission\nI trained 3 models described above, tuned them, predicted original test images, predicted mirrored test images, and computed weighted average.\n\n## Notes\n1. By increasing augmentation schedule I mean the following: train without augmentation until plateau (or until you feel like enough), then apply soft augmentation (wide range of values) during several epochs: e.g. `rotate(np.random.randint(-30, 30))`, then apply hard augmentation (short range or even two possible values) during another several epochs: e.g. `rotate(np.random.choice([-30, 30]))`.\n2. About training times. I used batch generator derived from [Sequence](https://keras.io/utils/#sequence) and multiprocessing with 8 processes per GPU. My typical hardware setup on GCP looks like: 1x P100, 8x CPU, 15 GB RAM, SSD or 2x P100, 16x CPU, 30 GB RAM, SSD, etc.\n\n## Side Notes\n1. Time after time I had been running 8x V100. It definitely feels like a ride on racing motorbike. I mean not only extreme speed of computation. I also mean adrenaline related to the speed of money flow.\n2. Looking at some images I was curious about one thing. How often do you experience something like nostalgia about the country you’ve never been?\n3. And by the way have you seen [convolved foxes](https://www.kaggle.com/c/inclusive-images-challenge/discussion/69789)?", "426533": "Congrats and thanks for sharing.", "433005": "Congratulations and thank you for sharing this overview" }, "source": "meta" }