For the two submissions for the private leaderboard, we decided to use the best model on the local validation and the best on the public leaderboard.
\n\nThe best model on local validation was a finetuned ResNeXt 101 WSL (32x16d) pre-trained on weak labels from Instagram images. We finetuned it on 90 % of the dataset with mild augmentation. This model solely gave us 92.75 on local validation (10 % of the dataset) and 90.8 on public lb (92.1 private (5th place, the same)). We decided not to use this one for the final prediction as we were afraid of the \"lb shake-up\" as well as this 90/10 validation scheme wasn't tested properly and we didn't want to have an overfitted model in our final set of submissions for private lb since 10 % was still a quite small portion of data even though the split was stratified. So, instead, we used the same model but which was trained on 75/25 split (local: 92.3, public: 91.0, private: 91.45).
\n\nThe final submission (91.88 private, 92.22 public) which you see on the private leaderboard is the majority vote of many models from previous submissions. We believe, the models we chose for the blend were orthogonal enough to each other. Specifically, it was a blend of three: another result of ResNeXt 101 WSL (32x16d) (local: 92.65); DenseNet (local: 91.45); and fused prediction of other 5 models (sklearn classifiers on ResNet features, Inception v3, Resnext 101 WSL, Efficientnet, Resnet 101).
\n\nValidation: stratified split 75/25 (we didn't do K-Fold CV to save time but you can add one loop to the code easily).
\n\nCode and more details: https://github.com/schatt89/VehicleRecognition
\n\nWhat didn't work:\n1. Weighted sampler nor loss\n2. Two-stage training: when the model weights are frozen and only the last layer is trained and, after, the whole model is finetuned with a lower lr\n3. AdamW optimizer (the same results)
", "messages": [ { "id": "696138", "postDate": "12/16/2019 07:06:03", "content": "For the two submissions for the private leaderboard, we decided to use the best model on the local validation and the best on the public leaderboard.
\n\nThe best model on local validation was a finetuned ResNeXt 101 WSL (32x16d) pre-trained on weak labels from Instagram images. We finetuned it on 90 % of the dataset with mild augmentation. This model solely gave us 92.75 on local validation (10 % of the dataset) and 90.8 on public lb (92.1 private (5th place, the same)). We decided not to use this one for the final prediction as we were afraid of the \"lb shake-up\" as well as this 90/10 validation scheme wasn't tested properly and we didn't want to have an overfitted model in our final set of submissions for private lb since 10 % was still a quite small portion of data even though the split was stratified. So, instead, we used the same model but which was trained on 75/25 split (local: 92.3, public: 91.0, private: 91.45).
\n\nThe final submission (91.88 private, 92.22 public) which you see on the private leaderboard is the majority vote of many models from previous submissions. We believe, the models we chose for the blend were orthogonal enough to each other. Specifically, it was a blend of three: another result of ResNeXt 101 WSL (32x16d) (local: 92.65); DenseNet (local: 91.45); and fused prediction of other 5 models (sklearn classifiers on ResNet features, Inception v3, Resnext 101 WSL, Efficientnet, Resnet 101).
\n\nValidation: stratified split 75/25 (we didn't do K-Fold CV to save time but you can add one loop to the code easily).
\n\nCode and more details: https://github.com/schatt89/VehicleRecognition
\n\nWhat didn't work:\n1. Weighted sampler nor loss\n2. Two-stage training: when the model weights are frozen and only the last layer is trained and, after, the whole model is finetuned with a lower lr\n3. AdamW optimizer (the same results)
", "rawMarkdown": "For the two submissions for the private leaderboard, we decided to use the best model on the local validation and the best on the public leaderboard.\n\nThe best model on local validation was a finetuned [ResNeXt 101 WSL (32x16d)](https://arxiv.org/abs/1805.00932) pre-trained on weak labels from Instagram images. We finetuned it on 90 % of the dataset with mild augmentation. This model solely gave us 92.75 on local validation (10 % of the dataset) and 90.8 on public lb (92.1 private (5th place, the same)). We decided not to use this one for the final prediction as we were afraid of the \"lb shake-up\" as well as this 90/10 validation scheme wasn't tested properly and we didn't want to have an overfitted model in our final set of submissions for private lb since 10 % was still a quite small portion of data even though the split was stratified. So, instead, we used the same model but which was trained on 75/25 split (local: 92.3, public: 91.0, private: 91.45).\n\n\nThe final submission (91.88 private, 92.22 public) which you see on the private leaderboard is the majority vote of many models from previous submissions. We believe, the models we chose for the blend were orthogonal enough to each other. Specifically, it was a blend of three: another result of ResNeXt 101 WSL (32x16d) (local: 92.65); DenseNet (local: 91.45); and fused prediction of other 5 models (sklearn classifiers on ResNet features, Inception v3, Resnext 101 WSL, Efficientnet, Resnet 101).\n\nValidation: stratified split 75/25 (we didn't do K-Fold CV to save time but you can add one loop to the code easily).\n\nCode and more details: [https://github.com/schatt89/VehicleRecognition](https://github.com/schatt89/VehicleRecognition)\n\nWhat didn't work:\n1. Weighted sampler nor loss\n2. Two-stage training: when the model weights are frozen and only the last layer is trained and, after, the whole model is finetuned with a lower lr\n3. AdamW optimizer (the same results)", "votes": null }, { "id": "696330", "postDate": "12/16/2019 13:38:17", "content": "Thanks for the nice description.
\n\nInteresting choice to pretrain with Instagram data with weak labels. Did it help compared to just using the usual Imagenet pretraining? How much?
\n\nFor those not familiar with this trick, here we learn to predict instagram hashtags from the images. Of course this does not give any good accuracy, but turns often out to be helpful when the trained network is used as a basis for the actual task.
", "rawMarkdown": "Thanks for the nice description.\n\nInteresting choice to pretrain with Instagram data with weak labels. Did it help compared to just using the usual Imagenet pretraining? How much?\n\nFor those not familiar with this trick, here we learn to predict instagram hashtags from the images. Of course this does not give any good accuracy, but turns often out to be helpful when the trained network is used as a basis for the actual task.", "votes": null }, { "id": "696368", "postDate": "12/16/2019 14:22:51", "content": "Well, we cannot really track the performance gain between the two since we had ImageNet weights for a lower capacity ResNeXt than the Instagram one. So, it is hard to guess whether the gain would be significant for the same number of parameters. What we can say is that compared to ResNeXt 101 32x8 (ImageNet weights), ResNeXt101 32x32 (Instagram weights) had additional 2 percentage points (88.4 vs 90.4). To be honest, we were just flattered by the gain presented in the tech report (Fig. 5) and the subsequent validation score improvement.
", "rawMarkdown": "Well, we cannot really track the performance gain between the two since we had ImageNet weights for a lower capacity ResNeXt than the Instagram one. So, it is hard to guess whether the gain would be significant for the same number of parameters. What we can say is that compared to ResNeXt 101 32x8 (ImageNet weights), ResNeXt101 32x32 (Instagram weights) had additional 2 percentage points (88.4 vs 90.4). To be honest, we were just flattered by the gain presented in the [tech report](https://arxiv.org/abs/1805.00932) (Fig. 5) and the subsequent validation score improvement.", "votes": null } ], "comments": [ { "id": 696330, "author_name": "mahehu", "author_url": "", "post_date": "12/16/2019 13:38:17", "content": "Thanks for the nice description.
\n\nInteresting choice to pretrain with Instagram data with weak labels. Did it help compared to just using the usual Imagenet pretraining? How much?
\n\nFor those not familiar with this trick, here we learn to predict instagram hashtags from the images. Of course this does not give any good accuracy, but turns often out to be helpful when the trained network is used as a basis for the actual task.
", "votes": null, "replies": [ { "id": 696368, "author_name": "vdyashin", "author_url": "", "post_date": "12/16/2019 14:22:51", "content": "Well, we cannot really track the performance gain between the two since we had ImageNet weights for a lower capacity ResNeXt than the Instagram one. So, it is hard to guess whether the gain would be significant for the same number of parameters. What we can say is that compared to ResNeXt 101 32x8 (ImageNet weights), ResNeXt101 32x32 (Instagram weights) had additional 2 percentage points (88.4 vs 90.4). To be honest, we were just flattered by the gain presented in the tech report (Fig. 5) and the subsequent validation score improvement.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "696138": "For the two submissions for the private leaderboard, we decided to use the best model on the local validation and the best on the public leaderboard.\n\nThe best model on local validation was a finetuned [ResNeXt 101 WSL (32x16d)](https://arxiv.org/abs/1805.00932) pre-trained on weak labels from Instagram images. We finetuned it on 90 % of the dataset with mild augmentation. This model solely gave us 92.75 on local validation (10 % of the dataset) and 90.8 on public lb (92.1 private (5th place, the same)). We decided not to use this one for the final prediction as we were afraid of the \"lb shake-up\" as well as this 90/10 validation scheme wasn't tested properly and we didn't want to have an overfitted model in our final set of submissions for private lb since 10 % was still a quite small portion of data even though the split was stratified. So, instead, we used the same model but which was trained on 75/25 split (local: 92.3, public: 91.0, private: 91.45).\n\n\nThe final submission (91.88 private, 92.22 public) which you see on the private leaderboard is the majority vote of many models from previous submissions. We believe, the models we chose for the blend were orthogonal enough to each other. Specifically, it was a blend of three: another result of ResNeXt 101 WSL (32x16d) (local: 92.65); DenseNet (local: 91.45); and fused prediction of other 5 models (sklearn classifiers on ResNet features, Inception v3, Resnext 101 WSL, Efficientnet, Resnet 101).\n\nValidation: stratified split 75/25 (we didn't do K-Fold CV to save time but you can add one loop to the code easily).\n\nCode and more details: [https://github.com/schatt89/VehicleRecognition](https://github.com/schatt89/VehicleRecognition)\n\nWhat didn't work:\n1. Weighted sampler nor loss\n2. Two-stage training: when the model weights are frozen and only the last layer is trained and, after, the whole model is finetuned with a lower lr\n3. AdamW optimizer (the same results)", "696330": "Thanks for the nice description.\n\nInteresting choice to pretrain with Instagram data with weak labels. Did it help compared to just using the usual Imagenet pretraining? How much?\n\nFor those not familiar with this trick, here we learn to predict instagram hashtags from the images. Of course this does not give any good accuracy, but turns often out to be helpful when the trained network is used as a basis for the actual task.", "696368": "Well, we cannot really track the performance gain between the two since we had ImageNet weights for a lower capacity ResNeXt than the Instagram one. So, it is hard to guess whether the gain would be significant for the same number of parameters. What we can say is that compared to ResNeXt 101 32x8 (ImageNet weights), ResNeXt101 32x32 (Instagram weights) had additional 2 percentage points (88.4 vs 90.4). To be honest, we were just flattered by the gain presented in the [tech report](https://arxiv.org/abs/1805.00932) (Fig. 5) and the subsequent validation score improvement." }, "source": "meta" }