{ "id": 275924, "title": "13th Place - Soft Landmark ID Assignment", "url": "/competitions/landmark-retrieval-2021/writeups/prateek-agnihotri-13th-place-soft-landmark-id-assi", "author_name": "", "post_date": "2021-10-02T09:57:27.713Z", "votes": 27, "comment_count": 5, "views": 0, "content": "

Thanks to google and kaggle for hosting this competition and clairvoyant.ai for supporting this work. It was a lot of fun and a full of learning experience.

\n

Certainly, due to limited hardware, extremely large dataset, and time constraints, my models are not that powerful like top teams, but I would love to discuss post-processing, which uses soft landmark id assignment for index and test images, and gave +0.075 boost on private LB.

\n

But before, to give a glimpse of models.

\n

Models

\n

I was using CurricularFace loss along with Adam Optimizer and Gradient Accumulation to train my models. I was using an ensemble of 5 models. All models were trained on the full GLDV2 dataset.

\n

Their scores without any post-processing and using simple KNN on model embeddings are -

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelImage SizePublic LBPrivate LB
Effnet B45120.3220.342
Effnet B63840.3310.349
Effnet v2 - s5120.3250.351
Effnet v2 - m3840.3570.373
Effnet v2 - xl2560.3180.319
Concat all embbedings0.3790.389
Concat all + Post Processing0.4480.464
\n

Post-Processing

\n

Soft Landmark ID assignment

\n

Step 1: Landmark assignment - Test Images

\n

For the landmark id assignment, I calculated the cosine similarity between the test image embeddings and the embeddings of images in the GLDV2 dataset. Next, I picked the top 'n' most similar GLDV2 images for every image in test data and assigned labels and label similarity scores to that image. An example label and label score calculation for an image A in test data is mentioned below.

\n

For a test image A, pick the top 'n' (in my case, it was 5) most similar images from GLDV2-

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
GLDV2 Img 1GLDV2 Img 2GLDV2 Img 3GLDV2 Img 4GLDV2 Img 5
Landmark ID1211012110199
Similarity Score0.90.70.50.470.45
\n

Then Image Landmark ID score can be calculated as

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Landmark IDImagesLandmark ID score
121GLDV2 Img 1, GLDV2 Img 30.9+0.5 = 1.4
10GLDV2 Img 2, GLDV2 Img 30.7+0.47 = 1.17
199GLDV2 Img 50.45
\n

Consider landmark id score as zero for landmark ids that are not there in the top 5.

\n

Step 2: Landmark assignment - Index Images

\n

A similar procedure was followed for index images, and landmark ids and landmark id scores were assigned to them as well.

\n

Step 3: Test-Index similarity score calculation

\n

Finally, calculated similarity between Image A from test and Image B from Index as -

\n

Similarity (A, B) = Landmark_ID_1_score(A) x Landmark_ID_1_score(B) +
\n                               Landmark_ID_2_score(A) x Landmark_ID_2_score(B) +
\n                               Landmark_ID_3_score(A) x Landmark_ID_3_score(B) +
\n                               ………..

\n

Then for every image in test data, arrange all similar images from index data in descending order based on their similarity score.

\n

As the next step, At the end of this list, I concatenated the knn-neighbours not found in the above search.

\n

This post-processing gave about +0.069 boost on public LB and +0.075 on private LB, with 0.448 and 0.464 as final Public and Private scores.

\n

Additional

\n

Additionally, I am attaching my gradient accumulation code for TPUs and a TensorFlow implementation of CurricularFace. I hope you will find them useful as well.

", "messages": [ { "id": "1531720", "postDate": "10/02/2021 09:49:16", "content": "

Thanks to google and kaggle for hosting this competition and clairvoyant.ai for supporting this work. It was a lot of fun and a full of learning experience.

\n

Certainly, due to limited hardware, extremely large dataset, and time constraints, my models are not that powerful like top teams, but I would love to discuss post-processing, which uses soft landmark id assignment for index and test images, and gave +0.075 boost on private LB.

\n

But before, to give a glimpse of models.

\n

Models

\n

I was using CurricularFace loss along with Adam Optimizer and Gradient Accumulation to train my models. I was using an ensemble of 5 models. All models were trained on the full GLDV2 dataset.

\n

Their scores without any post-processing and using simple KNN on model embeddings are -

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelImage SizePublic LBPrivate LB
Effnet B45120.3220.342
Effnet B63840.3310.349
Effnet v2 - s5120.3250.351
Effnet v2 - m3840.3570.373
Effnet v2 - xl2560.3180.319
Concat all embbedings0.3790.389
Concat all + Post Processing0.4480.464
\n

Post-Processing

\n

Soft Landmark ID assignment

\n

Step 1: Landmark assignment - Test Images

\n

For the landmark id assignment, I calculated the cosine similarity between the test image embeddings and the embeddings of images in the GLDV2 dataset. Next, I picked the top 'n' most similar GLDV2 images for every image in test data and assigned labels and label similarity scores to that image. An example label and label score calculation for an image A in test data is mentioned below.

\n

For a test image A, pick the top 'n' (in my case, it was 5) most similar images from GLDV2-

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
GLDV2 Img 1GLDV2 Img 2GLDV2 Img 3GLDV2 Img 4GLDV2 Img 5
Landmark ID1211012110199
Similarity Score0.90.70.50.470.45
\n

Then Image Landmark ID score can be calculated as

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Landmark IDImagesLandmark ID score
121GLDV2 Img 1, GLDV2 Img 30.9+0.5 = 1.4
10GLDV2 Img 2, GLDV2 Img 30.7+0.47 = 1.17
199GLDV2 Img 50.45
\n

Consider landmark id score as zero for landmark ids that are not there in the top 5.

\n

Step 2: Landmark assignment - Index Images

\n

A similar procedure was followed for index images, and landmark ids and landmark id scores were assigned to them as well.

\n

Step 3: Test-Index similarity score calculation

\n

Finally, calculated similarity between Image A from test and Image B from Index as -

\n

Similarity (A, B) = Landmark_ID_1_score(A) x Landmark_ID_1_score(B) +
\n                               Landmark_ID_2_score(A) x Landmark_ID_2_score(B) +
\n                               Landmark_ID_3_score(A) x Landmark_ID_3_score(B) +
\n                               ………..

\n

Then for every image in test data, arrange all similar images from index data in descending order based on their similarity score.

\n

As the next step, At the end of this list, I concatenated the knn-neighbours not found in the above search.

\n

This post-processing gave about +0.069 boost on public LB and +0.075 on private LB, with 0.448 and 0.464 as final Public and Private scores.

\n

Additional

\n

Additionally, I am attaching my gradient accumulation code for TPUs and a TensorFlow implementation of CurricularFace. I hope you will find them useful as well.

", "rawMarkdown": "Thanks to google and kaggle for hosting this competition and clairvoyant.ai for supporting this work. It was a lot of fun and a full of learning experience.\n\nCertainly, due to limited hardware, extremely large dataset, and time constraints, my models are not that powerful like top teams, but I would love to discuss post-processing, which uses soft landmark id assignment for index and test images, and gave **+0.075** boost on private LB.\n\nBut before, to give a glimpse of models.\n\n## Models\nI was using CurricularFace loss along with Adam Optimizer and Gradient Accumulation to train my models. I was using an ensemble of 5 models. All models were trained on the full GLDV2 dataset.\n\nTheir scores without any post-processing and using simple KNN on model embeddings are - \n| Model | Image Size | Public LB | Private LB |\n| --- | --- |\n| Effnet B4 | 512 | 0.322 | 0.342 |\n| Effnet B6 | 384 | 0.331 | 0.349 |\n| Effnet v2 - s | 512 | 0.325 | 0.351 |\n| Effnet v2 - m | 384 | 0.357 | 0.373 |\n| Effnet v2 - xl | 256 | 0.318 | 0.319 |\n| Concat all embbedings | | 0.379 | 0.389 |\n| Concat all + Post Processing | | 0.448 | 0.464 |\n\n## Post-Processing\n###Soft Landmark ID assignment\n**Step 1:** Landmark assignment - Test Images\n\nFor the landmark id assignment, I calculated the cosine similarity between the test image embeddings and the embeddings of images in the GLDV2 dataset. Next, I picked the top 'n' most similar GLDV2 images for every image in test data and assigned labels and label similarity scores to that image. An example label and label score calculation for an image A in test data is mentioned below.\n\nFor a test image A, pick the top 'n' (in my case, it was 5) most similar images from GLDV2-\n| | GLDV2 Img 1 | GLDV2 Img 2 | GLDV2 Img 3 | GLDV2 Img 4 | GLDV2 Img 5 |\n| --- | --- |\n| Landmark ID | 121 | 10 | 121 | 10 | 199 |\n| Similarity Score | 0.9 | 0.7 | 0.5 | 0.47 | 0.45 |\n\nThen Image Landmark ID score can be calculated as\n| Landmark ID | Images | Landmark ID score |\n| --- | --- |\n| 121 | GLDV2 Img 1, GLDV2 Img 3 | 0.9+0.5 = 1.4 |\n| 10 | GLDV2 Img 2, GLDV2 Img 3 | 0.7+0.47 = 1.17 |\n| 199 | GLDV2 Img 5 | 0.45 |\n\nConsider landmark id score as zero for landmark ids that are not there in the top 5.\n\n**Step 2:** Landmark assignment - Index Images\n\nA similar procedure was followed for index images, and landmark ids and landmark id scores were assigned to them as well.\n\n**Step 3:** Test-Index similarity score calculation\n\nFinally, calculated similarity between Image A from test and Image B from Index as - \n\nSimilarity (A, B) = Landmark_ID_1_score(A) x Landmark_ID_1_score(B) +\n                               Landmark_ID_2_score(A) x Landmark_ID_2_score(B) +\n                               Landmark_ID_3_score(A) x Landmark_ID_3_score(B) + \n                               ...........\n\nThen for every image in test data, arrange all similar images from index data in descending order based on their similarity score.\n\nAs the next step, At the end of this list, I concatenated the knn-neighbours not found in the above search.\n\nThis post-processing gave about +0.069 boost on public LB and +0.075 on private LB, with 0.448 and 0.464 as final Public and Private scores.\n\n## Additional\nAdditionally, I am [attaching](https://www.kaggle.com/prateekagnihotri/curricularface-and-gradient-accumulation) my gradient accumulation code for TPUs and a TensorFlow implementation of CurricularFace. I hope you will find them useful as well.", "votes": null }, { "id": "1531801", "postDate": "10/02/2021 11:41:04", "content": "

Congrats!

\n

One question, the imgs size that you share with us is the train one or the inference one?

\n

Thx

", "rawMarkdown": "Congrats! \n\nOne question, the imgs size that you share with us is the train one or the inference one?\n\nThx", "votes": null }, { "id": "1531815", "postDate": "10/02/2021 12:02:13", "content": "

Thanks, Enric!
\nImg sizes mentioned above are the train img sizes. During inference, like my last time solution, I did inference at train time img size x sqrt ( 2 )

", "rawMarkdown": "Thanks, Enric!\nImg sizes mentioned above are the train img sizes. During inference, like my last time [solution](https://www.kaggle.com/prateekagnihotri/change-image-size-to-get-0-281-lb), I did inference at train time img size x sqrt ( 2 )", "votes": null }, { "id": "1531993", "postDate": "10/02/2021 14:17:50", "content": "

Thanks for sharing this @prateekagnihotri … really insightful especially the post processing bit … and congratulations on you solo silver on both the competitions 👏👏

", "rawMarkdown": "Thanks for sharing this @prateekagnihotri ... really insightful especially the post processing bit ... and congratulations on you solo silver on both the competitions 👏👏", "votes": null }, { "id": "1532038", "postDate": "10/02/2021 15:02:48", "content": "

I am happy that you find it insightful. But it is more painful not to win gold in any😉. Anyway, you have to do everything you can and have to be fortunate. I am doing my part. I hope fortune will smile.

", "rawMarkdown": "I am happy that you find it insightful. But it is more painful not to win gold in any😉. Anyway, you have to do everything you can and have to be fortunate. I am doing my part. I hope fortune will smile.", "votes": null }, { "id": "1532232", "postDate": "10/02/2021 18:30:39", "content": "

Congrats @prateekagnihotri , awesome result!

", "rawMarkdown": "Congrats @prateekagnihotri , awesome result!", "votes": null } ], "comments": [ { "id": 1531801, "author_name": "enric1296", "author_url": "", "post_date": "10/02/2021 11:41:04", "content": "

Congrats!

\n

One question, the imgs size that you share with us is the train one or the inference one?

\n

Thx

", "votes": null, "replies": [ { "id": 1531815, "author_name": "prateekagnihotri", "author_url": "", "post_date": "10/02/2021 12:02:13", "content": "

Thanks, Enric!
\nImg sizes mentioned above are the train img sizes. During inference, like my last time solution, I did inference at train time img size x sqrt ( 2 )

", "votes": null, "replies": [] } ] }, { "id": 1531993, "author_name": "sandy1112", "author_url": "", "post_date": "10/02/2021 14:17:50", "content": "

Thanks for sharing this @prateekagnihotri … really insightful especially the post processing bit … and congratulations on you solo silver on both the competitions 👏👏

", "votes": null, "replies": [ { "id": 1532038, "author_name": "prateekagnihotri", "author_url": "", "post_date": "10/02/2021 15:02:48", "content": "

I am happy that you find it insightful. But it is more painful not to win gold in any😉. Anyway, you have to do everything you can and have to be fortunate. I am doing my part. I hope fortune will smile.

", "votes": null, "replies": [] } ] }, { "id": 1532232, "author_name": "saurabhbagchi", "author_url": "", "post_date": "10/02/2021 18:30:39", "content": "

Congrats @prateekagnihotri , awesome result!

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1531720": "Thanks to google and kaggle for hosting this competition and clairvoyant.ai for supporting this work. It was a lot of fun and a full of learning experience.\n\nCertainly, due to limited hardware, extremely large dataset, and time constraints, my models are not that powerful like top teams, but I would love to discuss post-processing, which uses soft landmark id assignment for index and test images, and gave **+0.075** boost on private LB.\n\nBut before, to give a glimpse of models.\n\n## Models\nI was using CurricularFace loss along with Adam Optimizer and Gradient Accumulation to train my models. I was using an ensemble of 5 models. All models were trained on the full GLDV2 dataset.\n\nTheir scores without any post-processing and using simple KNN on model embeddings are - \n| Model | Image Size | Public LB | Private LB |\n| --- | --- |\n| Effnet B4 | 512 | 0.322 | 0.342 |\n| Effnet B6 | 384 | 0.331 | 0.349 |\n| Effnet v2 - s | 512 | 0.325 | 0.351 |\n| Effnet v2 - m | 384 | 0.357 | 0.373 |\n| Effnet v2 - xl | 256 | 0.318 | 0.319 |\n| Concat all embbedings | | 0.379 | 0.389 |\n| Concat all + Post Processing | | 0.448 | 0.464 |\n\n## Post-Processing\n###Soft Landmark ID assignment\n**Step 1:** Landmark assignment - Test Images\n\nFor the landmark id assignment, I calculated the cosine similarity between the test image embeddings and the embeddings of images in the GLDV2 dataset. Next, I picked the top 'n' most similar GLDV2 images for every image in test data and assigned labels and label similarity scores to that image. An example label and label score calculation for an image A in test data is mentioned below.\n\nFor a test image A, pick the top 'n' (in my case, it was 5) most similar images from GLDV2-\n| | GLDV2 Img 1 | GLDV2 Img 2 | GLDV2 Img 3 | GLDV2 Img 4 | GLDV2 Img 5 |\n| --- | --- |\n| Landmark ID | 121 | 10 | 121 | 10 | 199 |\n| Similarity Score | 0.9 | 0.7 | 0.5 | 0.47 | 0.45 |\n\nThen Image Landmark ID score can be calculated as\n| Landmark ID | Images | Landmark ID score |\n| --- | --- |\n| 121 | GLDV2 Img 1, GLDV2 Img 3 | 0.9+0.5 = 1.4 |\n| 10 | GLDV2 Img 2, GLDV2 Img 3 | 0.7+0.47 = 1.17 |\n| 199 | GLDV2 Img 5 | 0.45 |\n\nConsider landmark id score as zero for landmark ids that are not there in the top 5.\n\n**Step 2:** Landmark assignment - Index Images\n\nA similar procedure was followed for index images, and landmark ids and landmark id scores were assigned to them as well.\n\n**Step 3:** Test-Index similarity score calculation\n\nFinally, calculated similarity between Image A from test and Image B from Index as - \n\nSimilarity (A, B) = Landmark_ID_1_score(A) x Landmark_ID_1_score(B) +\n                               Landmark_ID_2_score(A) x Landmark_ID_2_score(B) +\n                               Landmark_ID_3_score(A) x Landmark_ID_3_score(B) + \n                               ...........\n\nThen for every image in test data, arrange all similar images from index data in descending order based on their similarity score.\n\nAs the next step, At the end of this list, I concatenated the knn-neighbours not found in the above search.\n\nThis post-processing gave about +0.069 boost on public LB and +0.075 on private LB, with 0.448 and 0.464 as final Public and Private scores.\n\n## Additional\nAdditionally, I am [attaching](https://www.kaggle.com/prateekagnihotri/curricularface-and-gradient-accumulation) my gradient accumulation code for TPUs and a TensorFlow implementation of CurricularFace. I hope you will find them useful as well.", "1531801": "Congrats! \n\nOne question, the imgs size that you share with us is the train one or the inference one?\n\nThx", "1531815": "Thanks, Enric!\nImg sizes mentioned above are the train img sizes. During inference, like my last time [solution](https://www.kaggle.com/prateekagnihotri/change-image-size-to-get-0-281-lb), I did inference at train time img size x sqrt ( 2 )", "1531993": "Thanks for sharing this @prateekagnihotri ... really insightful especially the post processing bit ... and congratulations on you solo silver on both the competitions 👏👏", "1532038": "I am happy that you find it insightful. But it is more painful not to win gold in any😉. Anyway, you have to do everything you can and have to be fortunate. I am doing my part. I hope fortune will smile.", "1532232": "Congrats @prateekagnihotri , awesome result!" }, "source": "meta" }