{ "id": 275876, "title": "5th Place Solution Sharing: A Learnable Pooling Approach", "url": "/competitions/landmark-recognition-2021/writeups/notenoughfitting-5th-place-solution-sharing-a-lear", "author_name": "", "post_date": "2021-10-02T03:34:23.637Z", "votes": 33, "comment_count": 8, "views": 0, "content": "

Congratulations to the winners and thanks google and kaggle for hosting such an interesting competition!

\n

This is my first time to participate in the landmark recognition/retrieval competition. After reading the previous winning solutions, I also felt the anxiety with the limited computation resource I have: 1x3090 at the very beginning. So I decided to find a better(affordable) model architecture for the task, instead of following the common network+gem+arc_face approach.

\n

Based on the observation that larger input images would lead to better accuracy, I tried to use the intermediate feature maps from the backbones. With my experience in the Yourtube8M video classification, I then find that NeXtVLAD(https://arxiv.org/pdf/1811.05014.pdf) perform pretty good in aggregating and merging features from these different feature maps.

\n

Model Architecture

\n

\n

As you can see, each feature map is aggregated to one high-dimensional 1D feature with NeXtVLAD, then those features are concatenated and fed into a simple linear projection layer with output_dim=512. In my implementation, I also add more non-linearity with another SE-gating layer(which was actually found to be not really helpful). At last, we use the arc face product and dynamic margin for the loss.

\n

Training setup:
\nV2X: 3.2M images with 81313 landmark
\nV2C: 1.6M images with 81313 landmark
\nV2Full: 4.1M images with 203094 landmarks+ nonlandmark

\n

1) fix the weight in backbone and only train the learnable pooling layer with V2X for 10 epochs, which surprisingly have very high local validation(around 90%). [AdamW, 0.001 initial, 0.8 decay)

\n

2) finetune the whole model with V2C for 10 epochs(around 94% local validation accuracy)[AdamW, 0.0001 initial, 0.9 decay)

\n

3) change only the arc_face_product layer with 203095 classes(use the nonlandmark from 2019 testset is really helpful). Again, fix the weight of backbone and only tune the learnable pooling layers with V2Full. (around 95-96% accuracy after 10 epochs in local validation)[AdamW, 0.001 initial, 0.8 decay]

\n

The advantage of the learnable pooling layer is:

\n
    \n
  1. The training is super fast as you only need to train the learnable pooling layers at step 1 and step 3.
    2. \n
  2. I use the image input size of 256x256 at step 1 and step 2. Then increase the image size only at step 3, when the backbone is fixed(so I never need to train an efficientnet with large input images).
    3. \n
\n

For instance, training a swin-base-224 only takes around 3 days using single 3090 GPU with the setup.

\n

I achieved top10 around early Sep with just one 3090. And then my teammate @marbury provided me with another 2xV100 so that I can just scale to different backbones. My final solution is an ensemble of

\n\n

The final LB score is highly correlated with the accuracy of the backbone models in imagenet, which I suppose show me that this learnable pooling approach have good ability to do the transfer learning.

\n

It would be interesting to see what the model arch can achieve with more complex backbones, including efficientb5-b7, efficientv2-large and swin-large-384. There is still plenty of room to improve and hopefully to inspire more ideas & works using the learnable pooling module for the competition in next year. Will share my code once it is cleaned up.

\n

Other Insights

\n", "messages": [ { "id": "1531430", "postDate": "10/02/2021 02:13:58", "content": "

Congratulations to the winners and thanks google and kaggle for hosting such an interesting competition!

\n

This is my first time to participate in the landmark recognition/retrieval competition. After reading the previous winning solutions, I also felt the anxiety with the limited computation resource I have: 1x3090 at the very beginning. So I decided to find a better(affordable) model architecture for the task, instead of following the common network+gem+arc_face approach.

\n

Based on the observation that larger input images would lead to better accuracy, I tried to use the intermediate feature maps from the backbones. With my experience in the Yourtube8M video classification, I then find that NeXtVLAD(https://arxiv.org/pdf/1811.05014.pdf) perform pretty good in aggregating and merging features from these different feature maps.

\n

Model Architecture

\n

\n

As you can see, each feature map is aggregated to one high-dimensional 1D feature with NeXtVLAD, then those features are concatenated and fed into a simple linear projection layer with output_dim=512. In my implementation, I also add more non-linearity with another SE-gating layer(which was actually found to be not really helpful). At last, we use the arc face product and dynamic margin for the loss.

\n

Training setup:
\nV2X: 3.2M images with 81313 landmark
\nV2C: 1.6M images with 81313 landmark
\nV2Full: 4.1M images with 203094 landmarks+ nonlandmark

\n

1) fix the weight in backbone and only train the learnable pooling layer with V2X for 10 epochs, which surprisingly have very high local validation(around 90%). [AdamW, 0.001 initial, 0.8 decay)

\n

2) finetune the whole model with V2C for 10 epochs(around 94% local validation accuracy)[AdamW, 0.0001 initial, 0.9 decay)

\n

3) change only the arc_face_product layer with 203095 classes(use the nonlandmark from 2019 testset is really helpful). Again, fix the weight of backbone and only tune the learnable pooling layers with V2Full. (around 95-96% accuracy after 10 epochs in local validation)[AdamW, 0.001 initial, 0.8 decay]

\n

The advantage of the learnable pooling layer is:

\n
    \n
  1. The training is super fast as you only need to train the learnable pooling layers at step 1 and step 3.
    2. \n
  2. I use the image input size of 256x256 at step 1 and step 2. Then increase the image size only at step 3, when the backbone is fixed(so I never need to train an efficientnet with large input images).
    3. \n
\n

For instance, training a swin-base-224 only takes around 3 days using single 3090 GPU with the setup.

\n

I achieved top10 around early Sep with just one 3090. And then my teammate @marbury provided me with another 2xV100 so that I can just scale to different backbones. My final solution is an ensemble of

\n\n

The final LB score is highly correlated with the accuracy of the backbone models in imagenet, which I suppose show me that this learnable pooling approach have good ability to do the transfer learning.

\n

It would be interesting to see what the model arch can achieve with more complex backbones, including efficientb5-b7, efficientv2-large and swin-large-384. There is still plenty of room to improve and hopefully to inspire more ideas & works using the learnable pooling module for the competition in next year. Will share my code once it is cleaned up.

\n

Other Insights

\n", "rawMarkdown": "Congratulations to the winners and thanks google and kaggle for hosting such an interesting competition! \n\nThis is my first time to participate in the landmark recognition/retrieval competition. After reading the previous winning solutions, I also felt the anxiety with the limited computation resource I have: 1x3090 at the very beginning. So I decided to find a better(affordable) model architecture for the task, instead of following the common network+gem+arc_face approach.\n\nBased on the observation that larger input images would lead to better accuracy, I tried to use the intermediate feature maps from the backbones. With my experience in the Yourtube8M video classification, I then find that NeXtVLAD(https://arxiv.org/pdf/1811.05014.pdf) perform pretty good in aggregating and merging features from these different feature maps. \n\n\n### Model Architecture \n\n\n\nAs you can see, each feature map is aggregated to one high-dimensional 1D feature with NeXtVLAD, then those features are concatenated and fed into a simple linear projection layer with output_dim=512. In my implementation, I also add more non-linearity with another SE-gating layer(which was actually found to be not really helpful). At last, we use the arc face product and dynamic margin for the loss.\n\nTraining setup:\nV2X: 3.2M images with 81313 landmark\nV2C: 1.6M images with 81313 landmark\nV2Full: 4.1M images with 203094 landmarks+ nonlandmark\n \n1) fix the weight in backbone and only train the learnable pooling layer with V2X for 10 epochs, which surprisingly have very high local validation(around 90%). [AdamW, 0.001 initial, 0.8 decay)\n\n2) finetune the whole model with V2C for 10 epochs(around 94% local validation accuracy)[AdamW, 0.0001 initial, 0.9 decay)\n\n3) change only the arc_face_product layer with 203095 classes(use the nonlandmark from 2019 testset is really helpful). Again, fix the weight of backbone and only tune the learnable pooling layers with V2Full. (around 95-96% accuracy after 10 epochs in local validation)[AdamW, 0.001 initial, 0.8 decay]\n\nThe advantage of the learnable pooling layer is: \n\n1. The training is super fast as you only need to train the learnable pooling layers at step 1 and step 3.\n2. I use the image input size of 256x256 at step 1 and step 2. Then increase the image size only at step 3, when the backbone is fixed(so I never need to train an efficientnet with large input images).\n\nFor instance, training a swin-base-224 only takes around 3 days using single 3090 GPU with the setup. \n\nI achieved top10 around early Sep with just one 3090. And then my teammate @marbury provided me with another 2xV100 so that I can just scale to different backbones. My final solution is an ensemble of\n- swin-base-224\n- swin-base-384\n- swin-large-224\n- resnest101 (512 input at the step 3)\n- efficientv2-m (512 input at step 3)\n\nThe final LB score is highly correlated with the accuracy of the backbone models in imagenet, which I suppose show me that this learnable pooling approach have good ability to do the transfer learning. \n\nIt would be interesting to see what the model arch can achieve with more complex backbones, including efficientb5-b7, efficientv2-large and swin-large-384. There is still plenty of room to improve and hopefully to inspire more ideas & works using the learnable pooling module for the competition in next year. Will share my code once it is cleaned up.\n\n### Other Insights\n- using the whole v2full as the index set during the inference is important to stay at top 10.", "votes": null }, { "id": "1531485", "postDate": "10/02/2021 03:49:34", "content": "

Congratulations and thank you for sharing! 🎉

\n

I'm curious about the inference process,

\n
\n

using the whole v2full as the index set during the inference is important to stay at top 10.

\n
\n

Since the private training set is a 100k subset of the V2C, will this result in a recognition result outside the 81313 classes?

", "rawMarkdown": "Congratulations and thank you for sharing! 🎉\n\nI'm curious about the inference process, \n> using the whole v2full as the index set during the inference is important to stay at top 10.\n\nSince the private training set is a 100k subset of the V2C, will this result in a recognition result outside the 81313 classes?", "votes": null }, { "id": "1531491", "postDate": "10/02/2021 03:58:17", "content": "

@apretrue the test set is not a subset of V2C but V2Full. I encountered the same problem at the beginning(you can refer to my previous post in this competition).

\n

To be more precise, using a subset of the V2full during inference is important. During the inference, I upload the pre-calculated embeddings for V2Full and simply filter out the 'invalid' landmarks which didn't appear in the private index(training) set.

", "rawMarkdown": "apretrue the test set is not a subset of V2C but V2Full. I encountered the same problem at the beginning(you can refer to my previous post in this competition).\n\nTo be more precise, using a subset of the V2full during inference is important. During the inference, I upload the pre-calculated embeddings for V2Full and simply filter out the 'invalid' landmarks which didn't appear in the private index(training) set.", "votes": null }, { "id": "1531548", "postDate": "10/02/2021 05:52:00", "content": "

Congrats on your amazingly efficient solution! Any details about inference or is it just retrieval based top1?

", "rawMarkdown": "Congrats on your amazingly efficient solution! Any details about inference or is it just retrieval based top1?", "votes": null }, { "id": "1531638", "postDate": "10/02/2021 07:56:15", "content": "

A very interesting idea with the merge of models before the final embedding vector. Thanks for the write up

", "rawMarkdown": "A very interesting idea with the merge of models before the final embedding vector. Thanks for the write up", "votes": null }, { "id": "1531840", "postDate": "10/02/2021 12:32:10", "content": "

@keremt

\n

(not sure why the reply is not working)

\n

mainly use the same inference process from last year's winning solution. Find the 3 nearest neighbors from filtered V2Full pretrained embedding and penalize by the similarity to nonlandmarks(A_ij - B_j as mentioned in the post). Finally downrank the nonlandmarks with scores > 0.3. Nothing really new and interesting. I am sure you can learn more from other top solutions about the postprocessing part.

", "rawMarkdown": "keremt \n\n(not sure why the reply is not working)\n\nmainly use the same inference process from last year's winning solution. Find the 3 nearest neighbors from filtered V2Full pretrained embedding and penalize by the similarity to nonlandmarks(A_ij - B_j as mentioned in the post). Finally downrank the nonlandmarks with scores > 0.3. Nothing really new and interesting. I am sure you can learn more from other top solutions about the postprocessing part.", "votes": null }, { "id": "1531952", "postDate": "10/02/2021 13:43:03", "content": "

Hi congratz on the win and sharing your elegant solution here!

\n

I would like to also clarify something about the private train and test sets here.

\n

From the description in the competition page:
\nthe private training set contains only a 100k subset of the total public training set. This 100k subset contains all of the training set images associated with the landmarks in the private test set

\n

But in reality, the 100k images in the private training set is not a subset of the total public training set and both the private train and private test sets contain images with landmark ids that are outside of the 81313 landmark ids present in public train. May I ask if my understanding is correct and just curious on how did you find out about this?

", "rawMarkdown": "Hi congratz on the win and sharing your elegant solution here!\n\nI would like to also clarify something about the private train and test sets here.\n\nFrom the description in the competition [page](https://www.kaggle.com/c/landmark-recognition-2021/data):\n```the private training set contains only a 100k subset of the total public training set. This 100k subset contains all of the training set images associated with the landmarks in the private test set```\n\nBut in reality, the 100k images in the private training set is **not a subset** of the total public training set and both the private train and private test sets contain images with landmark ids that are outside of the 81313 landmark ids present in public train. May I ask if my understanding is correct and just curious on how did you find out about this?", "votes": null }, { "id": "1531960", "postDate": "10/02/2021 13:48:37", "content": "

@tmxxuan Yes, the 100k images in private training set should come from the 200k landmarks instead of 81313 ones. I got notebook exception threw error when applied a landmark_id mask to only 81313 classes. While debugging the issue, I think \"the total public training set\" actually means the V2Full. The description is unfortunately not very clear.

", "rawMarkdown": "tmxxuan Yes, the 100k images in private training set should come from the 200k landmarks instead of 81313 ones. I got notebook exception threw error when applied a landmark_id mask to only 81313 classes. While debugging the issue, I think \"the total public training set\" actually means the V2Full. The description is unfortunately not very clear.", "votes": null }, { "id": "1531984", "postDate": "10/02/2021 14:08:12", "content": "

I agree that the description was not very clear. I had so many Notebook Threw Exception too. Fortunately, some people shared their investigation about the private dataset, for example, here and here. Such discussion helped me a lot to improve the public score.

", "rawMarkdown": "I agree that the description was not very clear. I had so many `Notebook Threw Exception` too. Fortunately, some people shared their investigation about the private dataset, for example, [here](https://www.kaggle.com/c/landmark-recognition-2021/discussion/270020#1501830) and [here](https://www.kaggle.com/c/landmark-recognition-2021/discussion/273849#1521980). Such discussion helped me a lot to improve the public score.", "votes": null } ], "comments": [ { "id": 1531485, "author_name": "apretrue", "author_url": "", "post_date": "10/02/2021 03:49:34", "content": "

Congratulations and thank you for sharing! 🎉

\n

I'm curious about the inference process,

\n
\n

using the whole v2full as the index set during the inference is important to stay at top 10.

\n
\n

Since the private training set is a 100k subset of the V2C, will this result in a recognition result outside the 81313 classes?

", "votes": null, "replies": [ { "id": 1531491, "author_name": "phoenixl", "author_url": "", "post_date": "10/02/2021 03:58:17", "content": "

@apretrue the test set is not a subset of V2C but V2Full. I encountered the same problem at the beginning(you can refer to my previous post in this competition).

\n

To be more precise, using a subset of the V2full during inference is important. During the inference, I upload the pre-calculated embeddings for V2Full and simply filter out the 'invalid' landmarks which didn't appear in the private index(training) set.

", "votes": null, "replies": [] }, { "id": 1531952, "author_name": "tmxxuan", "author_url": "", "post_date": "10/02/2021 13:43:03", "content": "

Hi congratz on the win and sharing your elegant solution here!

\n

I would like to also clarify something about the private train and test sets here.

\n

From the description in the competition page:
\nthe private training set contains only a 100k subset of the total public training set. This 100k subset contains all of the training set images associated with the landmarks in the private test set

\n

But in reality, the 100k images in the private training set is not a subset of the total public training set and both the private train and private test sets contain images with landmark ids that are outside of the 81313 landmark ids present in public train. May I ask if my understanding is correct and just curious on how did you find out about this?

", "votes": null, "replies": [ { "id": 1531960, "author_name": "phoenixl", "author_url": "", "post_date": "10/02/2021 13:48:37", "content": "

@tmxxuan Yes, the 100k images in private training set should come from the 200k landmarks instead of 81313 ones. I got notebook exception threw error when applied a landmark_id mask to only 81313 classes. While debugging the issue, I think \"the total public training set\" actually means the V2Full. The description is unfortunately not very clear.

", "votes": null, "replies": [] } ] }, { "id": 1531984, "author_name": "hdsk38", "author_url": "", "post_date": "10/02/2021 14:08:12", "content": "

I agree that the description was not very clear. I had so many Notebook Threw Exception too. Fortunately, some people shared their investigation about the private dataset, for example, here and here. Such discussion helped me a lot to improve the public score.

", "votes": null, "replies": [] } ] }, { "id": 1531548, "author_name": "keremt", "author_url": "", "post_date": "10/02/2021 05:52:00", "content": "

Congrats on your amazingly efficient solution! Any details about inference or is it just retrieval based top1?

", "votes": null, "replies": [ { "id": 1531840, "author_name": "phoenixl", "author_url": "", "post_date": "10/02/2021 12:32:10", "content": "

@keremt

\n

(not sure why the reply is not working)

\n

mainly use the same inference process from last year's winning solution. Find the 3 nearest neighbors from filtered V2Full pretrained embedding and penalize by the similarity to nonlandmarks(A_ij - B_j as mentioned in the post). Finally downrank the nonlandmarks with scores > 0.3. Nothing really new and interesting. I am sure you can learn more from other top solutions about the postprocessing part.

", "votes": null, "replies": [] } ] }, { "id": 1531638, "author_name": "narsil", "author_url": "", "post_date": "10/02/2021 07:56:15", "content": "

A very interesting idea with the merge of models before the final embedding vector. Thanks for the write up

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1531430": "Congratulations to the winners and thanks google and kaggle for hosting such an interesting competition! \n\nThis is my first time to participate in the landmark recognition/retrieval competition. After reading the previous winning solutions, I also felt the anxiety with the limited computation resource I have: 1x3090 at the very beginning. So I decided to find a better(affordable) model architecture for the task, instead of following the common network+gem+arc_face approach.\n\nBased on the observation that larger input images would lead to better accuracy, I tried to use the intermediate feature maps from the backbones. With my experience in the Yourtube8M video classification, I then find that NeXtVLAD(https://arxiv.org/pdf/1811.05014.pdf) perform pretty good in aggregating and merging features from these different feature maps. \n\n\n### Model Architecture \n\n\n\nAs you can see, each feature map is aggregated to one high-dimensional 1D feature with NeXtVLAD, then those features are concatenated and fed into a simple linear projection layer with output_dim=512. In my implementation, I also add more non-linearity with another SE-gating layer(which was actually found to be not really helpful). At last, we use the arc face product and dynamic margin for the loss.\n\nTraining setup:\nV2X: 3.2M images with 81313 landmark\nV2C: 1.6M images with 81313 landmark\nV2Full: 4.1M images with 203094 landmarks+ nonlandmark\n \n1) fix the weight in backbone and only train the learnable pooling layer with V2X for 10 epochs, which surprisingly have very high local validation(around 90%). [AdamW, 0.001 initial, 0.8 decay)\n\n2) finetune the whole model with V2C for 10 epochs(around 94% local validation accuracy)[AdamW, 0.0001 initial, 0.9 decay)\n\n3) change only the arc_face_product layer with 203095 classes(use the nonlandmark from 2019 testset is really helpful). Again, fix the weight of backbone and only tune the learnable pooling layers with V2Full. (around 95-96% accuracy after 10 epochs in local validation)[AdamW, 0.001 initial, 0.8 decay]\n\nThe advantage of the learnable pooling layer is: \n\n1. The training is super fast as you only need to train the learnable pooling layers at step 1 and step 3.\n2. I use the image input size of 256x256 at step 1 and step 2. Then increase the image size only at step 3, when the backbone is fixed(so I never need to train an efficientnet with large input images).\n\nFor instance, training a swin-base-224 only takes around 3 days using single 3090 GPU with the setup. \n\nI achieved top10 around early Sep with just one 3090. And then my teammate @marbury provided me with another 2xV100 so that I can just scale to different backbones. My final solution is an ensemble of\n- swin-base-224\n- swin-base-384\n- swin-large-224\n- resnest101 (512 input at the step 3)\n- efficientv2-m (512 input at step 3)\n\nThe final LB score is highly correlated with the accuracy of the backbone models in imagenet, which I suppose show me that this learnable pooling approach have good ability to do the transfer learning. \n\nIt would be interesting to see what the model arch can achieve with more complex backbones, including efficientb5-b7, efficientv2-large and swin-large-384. There is still plenty of room to improve and hopefully to inspire more ideas & works using the learnable pooling module for the competition in next year. Will share my code once it is cleaned up.\n\n### Other Insights\n- using the whole v2full as the index set during the inference is important to stay at top 10.", "1531485": "Congratulations and thank you for sharing! 🎉\n\nI'm curious about the inference process, \n> using the whole v2full as the index set during the inference is important to stay at top 10.\n\nSince the private training set is a 100k subset of the V2C, will this result in a recognition result outside the 81313 classes?", "1531491": "apretrue the test set is not a subset of V2C but V2Full. I encountered the same problem at the beginning(you can refer to my previous post in this competition).\n\nTo be more precise, using a subset of the V2full during inference is important. During the inference, I upload the pre-calculated embeddings for V2Full and simply filter out the 'invalid' landmarks which didn't appear in the private index(training) set.", "1531548": "Congrats on your amazingly efficient solution! Any details about inference or is it just retrieval based top1?", "1531638": "A very interesting idea with the merge of models before the final embedding vector. Thanks for the write up", "1531840": "keremt \n\n(not sure why the reply is not working)\n\nmainly use the same inference process from last year's winning solution. Find the 3 nearest neighbors from filtered V2Full pretrained embedding and penalize by the similarity to nonlandmarks(A_ij - B_j as mentioned in the post). Finally downrank the nonlandmarks with scores > 0.3. Nothing really new and interesting. I am sure you can learn more from other top solutions about the postprocessing part.", "1531952": "Hi congratz on the win and sharing your elegant solution here!\n\nI would like to also clarify something about the private train and test sets here.\n\nFrom the description in the competition [page](https://www.kaggle.com/c/landmark-recognition-2021/data):\n```the private training set contains only a 100k subset of the total public training set. This 100k subset contains all of the training set images associated with the landmarks in the private test set```\n\nBut in reality, the 100k images in the private training set is **not a subset** of the total public training set and both the private train and private test sets contain images with landmark ids that are outside of the 81313 landmark ids present in public train. May I ask if my understanding is correct and just curious on how did you find out about this?", "1531960": "tmxxuan Yes, the 100k images in private training set should come from the 200k landmarks instead of 81313 ones. I got notebook exception threw error when applied a landmark_id mask to only 81313 classes. While debugging the issue, I think \"the total public training set\" actually means the V2Full. The description is unfortunately not very clear.", "1531984": "I agree that the description was not very clear. I had so many `Notebook Threw Exception` too. Fortunately, some people shared their investigation about the private dataset, for example, [here](https://www.kaggle.com/c/landmark-recognition-2021/discussion/270020#1501830) and [here](https://www.kaggle.com/c/landmark-recognition-2021/discussion/273849#1521980). Such discussion helped me a lot to improve the public score." }, "source": "meta" }