![\"\"](\"https://user-images.githubusercontent.com/27487010/69476665-0858c880-0e20-11ea-9fb4-5292f61e9c12.png\")
https://www.kaggle.com/c/landmark-recognition-2019
\n\nPublished two papers regarding our solution. You can check from:
\n\nYou can reproduce our environments using Dockerfile provided here https://github.com/lyakaap/Landmark2019-1st-and-3rd-Place-Solution/blob/master/docker/Dockerfile
\n\n| Dataset (train split) | # Samples |# Labels |\n| --- | --- | --- | \n| GLD-v1 | 1,225,029 | 14,951 | \n| GLD-v2 | 4,132,914 | 203,094 |\n| GLD-v2 (clean)| 1,580,470|81,313|
\n\nPrepare cleaned subset\n\n(You can skip this procedure to generate a cleaned subset. Pre-computed files are available on kaggle dataset.)
\n\nRun scripts/prepare_cleaned_subset.sh for cleaning the GLD-v2 dataset. The cleaning code requires DELF library (install instructions)
\n\nModel training and inference are done in exp/ directory.
\n\n#train models by various parameter settings with 4 gpus (each training is done with 2 gpus).\npython vX.py tuning -d 0,1,2,3 --n-gpu 2
# predict\npython vX.py predict -m vX/epX.pth -d 0'
# predict with multiple gpus\npython vX.py multigpu-predict -m vX/epX.pth --scale L2 --ms -b 32 -d 0,1
| Place| Team |Private | Public |\n| --- | --- | --- | \n| 1st | smlyaka |37.23 | 35.69 | \n| 2nd | imagesearch |34.75 | 32.25 | \n| 3rd | Layer 6 AI |32.18 | 29.85 |
\n\nhttps://github.com/filipradenovic/cnnimageretrieval-pytorch/tree/master/cirtorch\nhttps://github.com/ronghuaiyang/arcface-pytorch/blob/master/models/metrics.py\nhttps://github.com/4uiiurz1/pytorch-adacos/blob/master/metrics.py\nhttps://github.com/kevin-ssy/FishNet\nhttps://github.com/facebookresearch/maskrcnn-benchmark/blob/master/maskrcnn_benchmark/data/samplers/grouped_batch_sampler.py
\n\n
https://www.kaggle.com/c/landmark-recognition-2019
\n\nPublished two papers regarding our solution. You can check from:
\n\nYou can reproduce our environments using Dockerfile provided here https://github.com/lyakaap/Landmark2019-1st-and-3rd-Place-Solution/blob/master/docker/Dockerfile
\n\n| Dataset (train split) | # Samples |# Labels |\n| --- | --- | --- | \n| GLD-v1 | 1,225,029 | 14,951 | \n| GLD-v2 | 4,132,914 | 203,094 |\n| GLD-v2 (clean)| 1,580,470|81,313|
\n\nPrepare cleaned subset\n\n(You can skip this procedure to generate a cleaned subset. Pre-computed files are available on kaggle dataset.)
\n\nRun scripts/prepare_cleaned_subset.sh for cleaning the GLD-v2 dataset. The cleaning code requires DELF library (install instructions)
\n\nModel training and inference are done in exp/ directory.
\n\n#train models by various parameter settings with 4 gpus (each training is done with 2 gpus).\npython vX.py tuning -d 0,1,2,3 --n-gpu 2
# predict\npython vX.py predict -m vX/epX.pth -d 0'
# predict with multiple gpus\npython vX.py multigpu-predict -m vX/epX.pth --scale L2 --ms -b 32 -d 0,1
| Place| Team |Private | Public |\n| --- | --- | --- | \n| 1st | smlyaka |37.23 | 35.69 | \n| 2nd | imagesearch |34.75 | 32.25 | \n| 3rd | Layer 6 AI |32.18 | 29.85 |
\n\nhttps://github.com/filipradenovic/cnnimageretrieval-pytorch/tree/master/cirtorch\nhttps://github.com/ronghuaiyang/arcface-pytorch/blob/master/models/metrics.py\nhttps://github.com/4uiiurz1/pytorch-adacos/blob/master/metrics.py\nhttps://github.com/kevin-ssy/FishNet\nhttps://github.com/facebookresearch/maskrcnn-benchmark/blob/master/maskrcnn_benchmark/data/samplers/grouped_batch_sampler.py
\n\n\n", "rawMarkdown": "> #8th place solution\n> \n> # Summary\n> * **Training set**: Sub-set of 2018 Google Landmark Recognition\n> * **Descriptor**: Ensemble of DenseNet121-GeM-512FC (512x512, ArcFace) and ResNet101-GeM-2048FC (768x~, ContrastiveLoss)\n> * **Landmark classifier**: ResNet50 with T-scalling trained on OpenImages and Places. Remove images with less than 0.99 probability from train-set (4M->2.6M)\n> * **Inference**: Extract features for train and test set and compute the cosine similarity matrix between test and class centers. Then, apply K-NN to retrieve the 12 nearest neighbours and apply softmax on them.\n> * **Re-rank 1**: Multiply confidence score from inference by classification's score on test.\n> * **Re-rank 2**: Perform K-NN search (K=1750) of test set on itself and apply softmax. Multiply last score by this one.\n> \n> # Long version\n> I believe two issues must be addressed to tackle this competition. The first is a similarity search problem, where we want to create a good descriptor that would allow us to correctly classify an image based on a K-NN search in a gallery (trainset).\n> The second part is a detection/re-rank of non-landmarks in both train (gallery) and test (query) sets. This is paramount since the GAP metric used in this competition penalize us if we make a prediction for a non-landmark with higher confidence score than any other landmark images.\n> \n> ## Similarity search\n> ### Descriptor\n> I reckon a metric learning approach is much more natural to this task than a pure classification one. This is because we have a huge number of classes (> 200k) and some of them have just a few examples. Furthermore, in practice we would not be able to re-train our classifier every time we need to add a new landmark to the gallery.\n> \n> With this mindeset, I spent most of the time (until the last 2 weeks) training several models with ArcFace loss. I tried several types of pooling schemes such as SPoC, GeM and REMAP (my own re-implementation). Also tried several types of backbones like DenseNet121, ResNet101 and SEResNeXt101.\n> \n> The ones that worked best for me were DenseNet121-GeM-512FC (512x512) and SEResNeXt101-REMAP-2048FC (384x384). However, SEResNeXt101-REMAP was really slow so I only used DenseNet for the second stage.\n> \n> I trained on a sub-set of last years' competition, it was around 80k images of 15k classes. A margin of 0.8 for [ArcFace](https://arxiv.org/abs/1801.07698) loss was used and images were resized to the specified size. Weird as it may sound, I found it was better to squash it into a square than keeping the original aspect ratio with zero padding.\n> \n> I (unfortunately) spent most of my time here and the result were not so great. My best model only score around 0.33 on the image **retrieval** public LB (no QE, DBA or any re-rank). At this time I decided to give up ArcFace and go for the siamese approach.\n> \n> In the last weeks I trained a ResNet101-GeM-2048FC (768x~) following Radenovic's [paper](https://arxiv.org/abs/1711.02512) and despite I didn't submit to the retrieval competition (deadline was near) I believe it was considerably better than the other models I had trained so far.\n> \n> My final descriptor was an 2560-dim. ensemble of both models. I L2-normalized both and concatenated as follows: [1.5 * ResNet, 1 * DenseNet]. I didn't have enough time nor RAM to apply PCA on the resultant descriptor.\n> \n> ### Landmark classifier\n> Since there were a lot of non-landmark images both in test and train sets I trained a landmark classifier using the descriptor's train set as landmark examples. For the non-landmarks, I used images from OpenImages and Places datasets. I used classes such as: sky, forest, bathroom, bedroom, restaurant, Human faces, food, etc...\n> It was important to use outdoor classes as negative examples, otherwise the model would be biased to predict any outdoor image as landmark. Finally, I used [T-scalling](https://arxiv.org/pdf/1706.04599.pdf) to scale the predictions.\n> \n> For the train set I removed all the images with classification score less than 0.99, which resulted in around 2.6M images.\n> \n> For test set, on the other hand, I did not remove any images but used the scores to re-rank the final submission's confidence scores. I did this because, if the conf. score is well ranked, there is no penalty for making predictions on non-landmark images.\n> \n> ### Out-of-distribution images\n> Since the descriptor was only training on landmark images it is very bad in differentiating between images that are non-landmarks, e.g. human faces or types of food. As consequence of this fact, non-landmark images are matched with very high confidence scores, decreasing the GAP metric. In order to overcome this issue, three re-rank steps were used (will be explained later).\n> \n> ### Extract features\n> The CNNs previously mentioned are used to extract features (descriptors) for both train and test sets. Then for the train set, the class centers are computed by averaging all the image descriptors that belong to the same class. Finally, both test and class centers descriptors are L2 normalized.\n> \n> ### Search\n> The K-NN search was performed on the cosine distance matrix using [Faiss](https://github.com/facebookresearch/faiss) library. The top 12 NN were kept and softmax was applied on them.\n> \n> Even though we only need the first nearest neighbour, it was important to keep more of them to apply softmax and convert the cos. similarities into a probability distribution. By doing this we filter out some non-landmarks that have high similarity with a lot of classes in the gallery (train set). Without this or any other re-ranking steps, the score is likely to be close to zero even for a good descriptor.\n> \n> ### Re-rank:\n> The second re-rank was very straightforward. The original confidence score was multiplied by the probability of being a landmark from the classifier. With this step many non-landmark images got their conf. score reduced.\n> \n> ### Re-rank based on out-of-distribution similarity\n> For this step, the K-NN (K=1750) of the test set with itself was computed. The intuition here is that since our model is known to produce high similarity between out-of-distribution images, if we compute the similarity between the test set it is likely that non-landmark images will have much more similar neighbours than landmark ones. Thus, I apply the softmax and multiply by the confidence score from the last step.\n> \n> Hence, the final score is given by: `score = metric_learning_score * classifier_score * ood_score`\n> \n> # Leaderboard progression (stage 2)\n> * **DenseNet121-GeM-512FC (cls re-rank)**: 0.22379\n> * **DenseNet121-GeM-512FC (cls and ood re-rank)**: 0.23862\n> * **ResNet101-GeM-2048FC (cls and ood re-rank**): 0.26994\n> * **Ensemble (cls and ood re-rank)**: 0.28792\n> # Resources\n> * **GPU**: 1x Nvidia 1080ti (11GB)\n> * **RAM**: 32GB\n> * **HD**: 1TB SSD\n> * **Processor**: Intel i7 6800k\n> \n> #Acknowledgments\n> Thanks Kaggle and Google for hosting this competition. Thanks as well to all the community and @filipradenovic for opening his 2018 Google-Landmark-Retrieval solution.", "votes": null }, { "id": "960261", "postDate": "08/06/2020 09:18:56", "content": "8th place solution
\n \nSummary
\n \n\n
\n \n- Training set: Sub-set of 2018 Google Landmark Recognition
\n- Descriptor: Ensemble of DenseNet121-GeM-512FC (512x512, ArcFace) and ResNet101-GeM-2048FC (768x~, ContrastiveLoss)
\n- Landmark classifier: ResNet50 with T-scalling trained on OpenImages and Places. Remove images with less than 0.99 probability from train-set (4M->2.6M)
\n- Inference: Extract features for train and test set and compute the cosine similarity matrix between test and class centers. Then, apply K-NN to retrieve the 12 nearest neighbours and apply softmax on them.
\n- Re-rank 1: Multiply confidence score from inference by classification's score on test.
\n- Re-rank 2: Perform K-NN search (K=1750) of test set on itself and apply softmax. Multiply last score by this one.
\nLong version
\n \nI believe two issues must be addressed to tackle this competition. The first is a similarity search problem, where we want to create a good descriptor that would allow us to correctly classify an image based on a K-NN search in a gallery (trainset).\n The second part is a detection/re-rank of non-landmarks in both train (gallery) and test (query) sets. This is paramount since the GAP metric used in this competition penalize us if we make a prediction for a non-landmark with higher confidence score than any other landmark images.
\n \nSimilarity search
\n \nDescriptor
\n \nI reckon a metric learning approach is much more natural to this task than a pure classification one. This is because we have a huge number of classes (> 200k) and some of them have just a few examples. Furthermore, in practice we would not be able to re-train our classifier every time we need to add a new landmark to the gallery.
\n \nWith this mindeset, I spent most of the time (until the last 2 weeks) training several models with ArcFace loss. I tried several types of pooling schemes such as SPoC, GeM and REMAP (my own re-implementation). Also tried several types of backbones like DenseNet121, ResNet101 and SEResNeXt101.
\n \nThe ones that worked best for me were DenseNet121-GeM-512FC (512x512) and SEResNeXt101-REMAP-2048FC (384x384). However, SEResNeXt101-REMAP was really slow so I only used DenseNet for the second stage.
\n \nI trained on a sub-set of last years' competition, it was around 80k images of 15k classes. A margin of 0.8 for ArcFace loss was used and images were resized to the specified size. Weird as it may sound, I found it was better to squash it into a square than keeping the original aspect ratio with zero padding.
\n \nI (unfortunately) spent most of my time here and the result were not so great. My best model only score around 0.33 on the image retrieval public LB (no QE, DBA or any re-rank). At this time I decided to give up ArcFace and go for the siamese approach.
\n \nIn the last weeks I trained a ResNet101-GeM-2048FC (768x~) following Radenovic's paper and despite I didn't submit to the retrieval competition (deadline was near) I believe it was considerably better than the other models I had trained so far.
\n \nMy final descriptor was an 2560-dim. ensemble of both models. I L2-normalized both and concatenated as follows: [1.5 * ResNet, 1 * DenseNet]. I didn't have enough time nor RAM to apply PCA on the resultant descriptor.
\n \nLandmark classifier
\n \nSince there were a lot of non-landmark images both in test and train sets I trained a landmark classifier using the descriptor's train set as landmark examples. For the non-landmarks, I used images from OpenImages and Places datasets. I used classes such as: sky, forest, bathroom, bedroom, restaurant, Human faces, food, etc...\n It was important to use outdoor classes as negative examples, otherwise the model would be biased to predict any outdoor image as landmark. Finally, I used T-scalling to scale the predictions.
\n \nFor the train set I removed all the images with classification score less than 0.99, which resulted in around 2.6M images.
\n \nFor test set, on the other hand, I did not remove any images but used the scores to re-rank the final submission's confidence scores. I did this because, if the conf. score is well ranked, there is no penalty for making predictions on non-landmark images.
\n \nOut-of-distribution images
\n \nSince the descriptor was only training on landmark images it is very bad in differentiating between images that are non-landmarks, e.g. human faces or types of food. As consequence of this fact, non-landmark images are matched with very high confidence scores, decreasing the GAP metric. In order to overcome this issue, three re-rank steps were used (will be explained later).
\n \nExtract features
\n \nThe CNNs previously mentioned are used to extract features (descriptors) for both train and test sets. Then for the train set, the class centers are computed by averaging all the image descriptors that belong to the same class. Finally, both test and class centers descriptors are L2 normalized.
\n \nSearch
\n \nThe K-NN search was performed on the cosine distance matrix using Faiss library. The top 12 NN were kept and softmax was applied on them.
\n \nEven though we only need the first nearest neighbour, it was important to keep more of them to apply softmax and convert the cos. similarities into a probability distribution. By doing this we filter out some non-landmarks that have high similarity with a lot of classes in the gallery (train set). Without this or any other re-ranking steps, the score is likely to be close to zero even for a good descriptor.
\n \nRe-rank:
\n \nThe second re-rank was very straightforward. The original confidence score was multiplied by the probability of being a landmark from the classifier. With this step many non-landmark images got their conf. score reduced.
\n \nRe-rank based on out-of-distribution similarity
\n \nFor this step, the K-NN (K=1750) of the test set with itself was computed. The intuition here is that since our model is known to produce high similarity between out-of-distribution images, if we compute the similarity between the test set it is likely that non-landmark images will have much more similar neighbours than landmark ones. Thus, I apply the softmax and multiply by the confidence score from the last step.
\n \nHence, the final score is given by:
\n \nscore = metric_learning_score * classifier_score * ood_scoreLeaderboard progression (stage 2)
\n \n\n
\n \n- DenseNet121-GeM-512FC (cls re-rank): 0.22379
\n- DenseNet121-GeM-512FC (cls and ood re-rank): 0.23862
\n- ResNet101-GeM-2048FC (cls and ood re-rank): 0.26994
\n- \n
Ensemble (cls and ood re-rank): 0.28792
\n \nResources
- \n
GPU: 1x Nvidia 1080ti (11GB)
- RAM: 32GB
\n- HD: 1TB SSD
\n- Processor: Intel i7 6800k
\nAcknowledgments
\n \nThanks Kaggle and Google for hosting this competition. Thanks as well to all the community and @filipradenovic for opening his 2018 Google-Landmark-Retrieval solution.
\n
\n", "rawMarkdown": "# 2nd Place solution to Kaggle Landmark Recognition and Retrieval Competition 2019\n\n> arxiv link: https://arxiv.org/pdf/1906.03990.pdf\n> source code: https://github.com/PaddlePaddle/models/tree/develop/PaddleCV/Research/landmark", "votes": null }, { "id": "965640", "postDate": "08/10/2020 18:31:12", "content": "arxiv link: https://arxiv.org/pdf/1906.03990.pdf\n source code: https://github.com/PaddlePaddle/models/tree/develop/PaddleCV/Research/landmark
\n
Thanks for sharing @arachauhan. However, instead of copy-pasting from the source, you could have linked the original repo: https://github.com/lyakaap/Landmark2019-1st-and-3rd-Place-Solution. \nDoing so will get you more upvotes next time!
", "rawMarkdown": "Thanks for sharing @arachauhan. However, instead of copy-pasting from the source, you could have linked the original repo: https://github.com/lyakaap/Landmark2019-1st-and-3rd-Place-Solution. \nDoing so will get you more upvotes next time!", "votes": null } ], "comments": [ { "id": 960258, "author_name": "arachauhan", "author_url": "", "post_date": "08/06/2020 09:17:14", "content": "\n", "votes": null, "replies": [] }, { "id": 960261, "author_name": "arachauhan", "author_url": "", "post_date": "08/06/2020 09:18:56", "content": "8th place solution
\n \nSummary
\n \n\n
\n \n- Training set: Sub-set of 2018 Google Landmark Recognition
\n- Descriptor: Ensemble of DenseNet121-GeM-512FC (512x512, ArcFace) and ResNet101-GeM-2048FC (768x~, ContrastiveLoss)
\n- Landmark classifier: ResNet50 with T-scalling trained on OpenImages and Places. Remove images with less than 0.99 probability from train-set (4M->2.6M)
\n- Inference: Extract features for train and test set and compute the cosine similarity matrix between test and class centers. Then, apply K-NN to retrieve the 12 nearest neighbours and apply softmax on them.
\n- Re-rank 1: Multiply confidence score from inference by classification's score on test.
\n- Re-rank 2: Perform K-NN search (K=1750) of test set on itself and apply softmax. Multiply last score by this one.
\nLong version
\n \nI believe two issues must be addressed to tackle this competition. The first is a similarity search problem, where we want to create a good descriptor that would allow us to correctly classify an image based on a K-NN search in a gallery (trainset).\n The second part is a detection/re-rank of non-landmarks in both train (gallery) and test (query) sets. This is paramount since the GAP metric used in this competition penalize us if we make a prediction for a non-landmark with higher confidence score than any other landmark images.
\n \nSimilarity search
\n \nDescriptor
\n \nI reckon a metric learning approach is much more natural to this task than a pure classification one. This is because we have a huge number of classes (> 200k) and some of them have just a few examples. Furthermore, in practice we would not be able to re-train our classifier every time we need to add a new landmark to the gallery.
\n \nWith this mindeset, I spent most of the time (until the last 2 weeks) training several models with ArcFace loss. I tried several types of pooling schemes such as SPoC, GeM and REMAP (my own re-implementation). Also tried several types of backbones like DenseNet121, ResNet101 and SEResNeXt101.
\n \nThe ones that worked best for me were DenseNet121-GeM-512FC (512x512) and SEResNeXt101-REMAP-2048FC (384x384). However, SEResNeXt101-REMAP was really slow so I only used DenseNet for the second stage.
\n \nI trained on a sub-set of last years' competition, it was around 80k images of 15k classes. A margin of 0.8 for ArcFace loss was used and images were resized to the specified size. Weird as it may sound, I found it was better to squash it into a square than keeping the original aspect ratio with zero padding.
\n \nI (unfortunately) spent most of my time here and the result were not so great. My best model only score around 0.33 on the image retrieval public LB (no QE, DBA or any re-rank). At this time I decided to give up ArcFace and go for the siamese approach.
\n \nIn the last weeks I trained a ResNet101-GeM-2048FC (768x~) following Radenovic's paper and despite I didn't submit to the retrieval competition (deadline was near) I believe it was considerably better than the other models I had trained so far.
\n \nMy final descriptor was an 2560-dim. ensemble of both models. I L2-normalized both and concatenated as follows: [1.5 * ResNet, 1 * DenseNet]. I didn't have enough time nor RAM to apply PCA on the resultant descriptor.
\n \nLandmark classifier
\n \nSince there were a lot of non-landmark images both in test and train sets I trained a landmark classifier using the descriptor's train set as landmark examples. For the non-landmarks, I used images from OpenImages and Places datasets. I used classes such as: sky, forest, bathroom, bedroom, restaurant, Human faces, food, etc...\n It was important to use outdoor classes as negative examples, otherwise the model would be biased to predict any outdoor image as landmark. Finally, I used T-scalling to scale the predictions.
\n \nFor the train set I removed all the images with classification score less than 0.99, which resulted in around 2.6M images.
\n \nFor test set, on the other hand, I did not remove any images but used the scores to re-rank the final submission's confidence scores. I did this because, if the conf. score is well ranked, there is no penalty for making predictions on non-landmark images.
\n \nOut-of-distribution images
\n \nSince the descriptor was only training on landmark images it is very bad in differentiating between images that are non-landmarks, e.g. human faces or types of food. As consequence of this fact, non-landmark images are matched with very high confidence scores, decreasing the GAP metric. In order to overcome this issue, three re-rank steps were used (will be explained later).
\n \nExtract features
\n \nThe CNNs previously mentioned are used to extract features (descriptors) for both train and test sets. Then for the train set, the class centers are computed by averaging all the image descriptors that belong to the same class. Finally, both test and class centers descriptors are L2 normalized.
\n \nSearch
\n \nThe K-NN search was performed on the cosine distance matrix using Faiss library. The top 12 NN were kept and softmax was applied on them.
\n \nEven though we only need the first nearest neighbour, it was important to keep more of them to apply softmax and convert the cos. similarities into a probability distribution. By doing this we filter out some non-landmarks that have high similarity with a lot of classes in the gallery (train set). Without this or any other re-ranking steps, the score is likely to be close to zero even for a good descriptor.
\n \nRe-rank:
\n \nThe second re-rank was very straightforward. The original confidence score was multiplied by the probability of being a landmark from the classifier. With this step many non-landmark images got their conf. score reduced.
\n \nRe-rank based on out-of-distribution similarity
\n \nFor this step, the K-NN (K=1750) of the test set with itself was computed. The intuition here is that since our model is known to produce high similarity between out-of-distribution images, if we compute the similarity between the test set it is likely that non-landmark images will have much more similar neighbours than landmark ones. Thus, I apply the softmax and multiply by the confidence score from the last step.
\n \nHence, the final score is given by:
\n \nscore = metric_learning_score * classifier_score * ood_scoreLeaderboard progression (stage 2)
\n \n\n
\n \n- DenseNet121-GeM-512FC (cls re-rank): 0.22379
\n- DenseNet121-GeM-512FC (cls and ood re-rank): 0.23862
\n- ResNet101-GeM-2048FC (cls and ood re-rank): 0.26994
\n- \n
Ensemble (cls and ood re-rank): 0.28792
\n \nResources
- \n
GPU: 1x Nvidia 1080ti (11GB)
- RAM: 32GB
\n- HD: 1TB SSD
\n- Processor: Intel i7 6800k
\nAcknowledgments
\n \nThanks Kaggle and Google for hosting this competition. Thanks as well to all the community and @filipradenovic for opening his 2018 Google-Landmark-Retrieval solution.
\n
\n", "votes": null, "replies": [] }, { "id": 965640, "author_name": "yassinealouini", "author_url": "", "post_date": "08/10/2020 18:31:12", "content": "arxiv link: https://arxiv.org/pdf/1906.03990.pdf\n source code: https://github.com/PaddlePaddle/models/tree/develop/PaddleCV/Research/landmark
\n
Thanks for sharing @arachauhan. However, instead of copy-pasting from the source, you could have linked the original repo: https://github.com/lyakaap/Landmark2019-1st-and-3rd-Place-Solution. \nDoing so will get you more upvotes next time!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "960257": "\n\n\n\nhttps://www.kaggle.com/c/landmark-recognition-2019\n\nPublished two papers regarding our solution. You can check from:\n\n1. [Large-scale Landmark Retrieval/Recognition under a Noisy and Diverse Dataset](https://arxiv.org/abs/1906.04087)\nTech report describing our solution (4-pages)\n2. [Two-stage Discriminative Re-ranking for Large-scale Landmark Retrieval](https://arxiv.org/abs/2003.11211)\nExtended version of above tech report, including more detailed explanation and additional experimental results (10-pages, CVPR Workshop 2020)\n\n### Environments \n\nYou can reproduce our environments using Dockerfile provided here [https://github.com/lyakaap/Landmark2019-1st-and-3rd-Place-Solution/blob/master/docker/Dockerfile](https://github.com/lyakaap/Landmark2019-1st-and-3rd-Place-Solution/blob/master/docker/Dockerfile)\n\n### Data\n\n- Google Landmark Dataset v1 (GLD-v1): [https://www.kaggle.com/google/google-landmarks-dataset](https://www.kaggle.com/google/google-landmarks-dataset)\n- Google Landmark Dataset v2 (GLD-v2): [https://github.com/cvdfoundation/google-landmark](https://github.com/cvdfoundation/google-landmark)\n- Clean version of the v2 (Newly Released!): [https://www.kaggle.com/confirm/cleaned-subsets-of-google-landmarks-v2/kernels](https://www.kaggle.com/confirm/cleaned-subsets-of-google-landmarks-v2/kernels)\n\n### Dataset statistics:\n\n| Dataset (train split) | # Samples |# Labels |\n| --- | --- | --- | \n| GLD-v1 | 1,225,029 | 14,951 | \n| GLD-v2 | 4,132,914 | 203,094 |\n| GLD-v2 (clean)| 1,580,470|81,313|\n\n#### Prepare cleaned subset\n\n(You can skip this procedure to generate a cleaned subset. Pre-computed files are available on [kaggle dataset.](https://www.kaggle.com/confirm/cleaned-subsets-of-google-landmarks-v2))\n\nRun scripts/prepare_cleaned_subset.sh for cleaning the GLD-v2 dataset. The cleaning code requires DELF library ([install instructions](https://github.com/tensorflow/models/blob/master/research/delf/INSTALL_INSTRUCTIONS.md))\n\n### exp\n\nModel training and inference are done in exp/ directory.\n\n` #train models by various parameter settings with 4 gpus (each training is done with 2 gpus).\npython vX.py tuning -d 0,1,2,3 --n-gpu 2 `\n\n`# predict`\n`python vX.py predict -m vX/epX.pth -d 0'`\n\n`# predict with multiple gpus`\n`python vX.py multigpu-predict -m vX/epX.pth --scale L2 --ms -b 32 -d 0,1`\n\n## Results (retrieval challenge)\n\n| Place| Team |Private | Public |\n| --- | --- | --- | \n| 1st | smlyaka |37.23 | 35.69 | \n| 2nd | imagesearch |34.75 | 32.25 | \n| 3rd | Layer 6 AI |32.18 | 29.85 | \n\n## Reference\n\nhttps://github.com/filipradenovic/cnnimageretrieval-pytorch/tree/master/cirtorch\nhttps://github.com/ronghuaiyang/arcface-pytorch/blob/master/models/metrics.py\nhttps://github.com/4uiiurz1/pytorch-adacos/blob/master/metrics.py\nhttps://github.com/kevin-ssy/FishNet\nhttps://github.com/facebookresearch/maskrcnn-benchmark/blob/master/maskrcnn_benchmark/data/samplers/grouped_batch_sampler.py\n\n\n### UPVOTE if it helps you 👍", "960258": "> #8th place solution\n> \n> # Summary\n> * **Training set**: Sub-set of 2018 Google Landmark Recognition\n> * **Descriptor**: Ensemble of DenseNet121-GeM-512FC (512x512, ArcFace) and ResNet101-GeM-2048FC (768x~, ContrastiveLoss)\n> * **Landmark classifier**: ResNet50 with T-scalling trained on OpenImages and Places. Remove images with less than 0.99 probability from train-set (4M->2.6M)\n> * **Inference**: Extract features for train and test set and compute the cosine similarity matrix between test and class centers. Then, apply K-NN to retrieve the 12 nearest neighbours and apply softmax on them.\n> * **Re-rank 1**: Multiply confidence score from inference by classification's score on test.\n> * **Re-rank 2**: Perform K-NN search (K=1750) of test set on itself and apply softmax. Multiply last score by this one.\n> \n> # Long version\n> I believe two issues must be addressed to tackle this competition. The first is a similarity search problem, where we want to create a good descriptor that would allow us to correctly classify an image based on a K-NN search in a gallery (trainset).\n> The second part is a detection/re-rank of non-landmarks in both train (gallery) and test (query) sets. This is paramount since the GAP metric used in this competition penalize us if we make a prediction for a non-landmark with higher confidence score than any other landmark images.\n> \n> ## Similarity search\n> ### Descriptor\n> I reckon a metric learning approach is much more natural to this task than a pure classification one. This is because we have a huge number of classes (> 200k) and some of them have just a few examples. Furthermore, in practice we would not be able to re-train our classifier every time we need to add a new landmark to the gallery.\n> \n> With this mindeset, I spent most of the time (until the last 2 weeks) training several models with ArcFace loss. I tried several types of pooling schemes such as SPoC, GeM and REMAP (my own re-implementation). Also tried several types of backbones like DenseNet121, ResNet101 and SEResNeXt101.\n> \n> The ones that worked best for me were DenseNet121-GeM-512FC (512x512) and SEResNeXt101-REMAP-2048FC (384x384). However, SEResNeXt101-REMAP was really slow so I only used DenseNet for the second stage.\n> \n> I trained on a sub-set of last years' competition, it was around 80k images of 15k classes. A margin of 0.8 for [ArcFace](https://arxiv.org/abs/1801.07698) loss was used and images were resized to the specified size. Weird as it may sound, I found it was better to squash it into a square than keeping the original aspect ratio with zero padding.\n> \n> I (unfortunately) spent most of my time here and the result were not so great. My best model only score around 0.33 on the image **retrieval** public LB (no QE, DBA or any re-rank). At this time I decided to give up ArcFace and go for the siamese approach.\n> \n> In the last weeks I trained a ResNet101-GeM-2048FC (768x~) following Radenovic's [paper](https://arxiv.org/abs/1711.02512) and despite I didn't submit to the retrieval competition (deadline was near) I believe it was considerably better than the other models I had trained so far.\n> \n> My final descriptor was an 2560-dim. ensemble of both models. I L2-normalized both and concatenated as follows: [1.5 * ResNet, 1 * DenseNet]. I didn't have enough time nor RAM to apply PCA on the resultant descriptor.\n> \n> ### Landmark classifier\n> Since there were a lot of non-landmark images both in test and train sets I trained a landmark classifier using the descriptor's train set as landmark examples. For the non-landmarks, I used images from OpenImages and Places datasets. I used classes such as: sky, forest, bathroom, bedroom, restaurant, Human faces, food, etc...\n> It was important to use outdoor classes as negative examples, otherwise the model would be biased to predict any outdoor image as landmark. Finally, I used [T-scalling](https://arxiv.org/pdf/1706.04599.pdf) to scale the predictions.\n> \n> For the train set I removed all the images with classification score less than 0.99, which resulted in around 2.6M images.\n> \n> For test set, on the other hand, I did not remove any images but used the scores to re-rank the final submission's confidence scores. I did this because, if the conf. score is well ranked, there is no penalty for making predictions on non-landmark images.\n> \n> ### Out-of-distribution images\n> Since the descriptor was only training on landmark images it is very bad in differentiating between images that are non-landmarks, e.g. human faces or types of food. As consequence of this fact, non-landmark images are matched with very high confidence scores, decreasing the GAP metric. In order to overcome this issue, three re-rank steps were used (will be explained later).\n> \n> ### Extract features\n> The CNNs previously mentioned are used to extract features (descriptors) for both train and test sets. Then for the train set, the class centers are computed by averaging all the image descriptors that belong to the same class. Finally, both test and class centers descriptors are L2 normalized.\n> \n> ### Search\n> The K-NN search was performed on the cosine distance matrix using [Faiss](https://github.com/facebookresearch/faiss) library. The top 12 NN were kept and softmax was applied on them.\n> \n> Even though we only need the first nearest neighbour, it was important to keep more of them to apply softmax and convert the cos. similarities into a probability distribution. By doing this we filter out some non-landmarks that have high similarity with a lot of classes in the gallery (train set). Without this or any other re-ranking steps, the score is likely to be close to zero even for a good descriptor.\n> \n> ### Re-rank:\n> The second re-rank was very straightforward. The original confidence score was multiplied by the probability of being a landmark from the classifier. With this step many non-landmark images got their conf. score reduced.\n> \n> ### Re-rank based on out-of-distribution similarity\n> For this step, the K-NN (K=1750) of the test set with itself was computed. The intuition here is that since our model is known to produce high similarity between out-of-distribution images, if we compute the similarity between the test set it is likely that non-landmark images will have much more similar neighbours than landmark ones. Thus, I apply the softmax and multiply by the confidence score from the last step.\n> \n> Hence, the final score is given by: `score = metric_learning_score * classifier_score * ood_score`\n> \n> # Leaderboard progression (stage 2)\n> * **DenseNet121-GeM-512FC (cls re-rank)**: 0.22379\n> * **DenseNet121-GeM-512FC (cls and ood re-rank)**: 0.23862\n> * **ResNet101-GeM-2048FC (cls and ood re-rank**): 0.26994\n> * **Ensemble (cls and ood re-rank)**: 0.28792\n> # Resources\n> * **GPU**: 1x Nvidia 1080ti (11GB)\n> * **RAM**: 32GB\n> * **HD**: 1TB SSD\n> * **Processor**: Intel i7 6800k\n> \n> #Acknowledgments\n> Thanks Kaggle and Google for hosting this competition. Thanks as well to all the community and @filipradenovic for opening his 2018 Google-Landmark-Retrieval solution.", "960261": "# 2nd Place solution to Kaggle Landmark Recognition and Retrieval Competition 2019\n\n> arxiv link: https://arxiv.org/pdf/1906.03990.pdf\n> source code: https://github.com/PaddlePaddle/models/tree/develop/PaddleCV/Research/landmark", "965640": "Thanks for sharing @arachauhan. However, instead of copy-pasting from the source, you could have linked the original repo: https://github.com/lyakaap/Landmark2019-1st-and-3rd-Place-Solution. \nDoing so will get you more upvotes next time!" }, "source": "meta" }