\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F701191%2Fb1ac1bca6f58600f19ccbb31a2496682%2Fmodels_jpg.jpg?generation=1597910875460797&alt=media\")
[Update] solution arxiv link : https://arxiv.org/abs/2009.05132
\n
Great thanks to google and kaggle team for hosting this competition, and congrats to all participants who finished successfully. I really learned a lot during the competition through reading articles, analysing codes, and doing experiments.
\nI'd like to share my solution, and detailed solution will be uploaded to arxiv in a few days.
\nModel structure is as below.
\n
Validation set : 1 sample per class which has >=4 samples in GLD v2 clean dataset(72322/81313 classes)
\nCosine softmax : s=determined by adacos, m=0
\nWeighted cross entropy : proportional to 1/log(class cnt)
\nAugmentation : left-right flip
\nOptimizer : SGD(1e-3, momentum=0.9, decay=1e-5)
\nEmbedding Dimension : 512 for every model
\nHardware : Colab TPUs
1.Use GLD v2 clean dataset to train model to classify 81313 classes
\nefn7 512x512 priv.LB:0.30264, pub.LB:0.33907
\n2.Take efficientnet backbone from step 1, use GLD v2 total dataset to train model to classify 203094 classes
\n efn7 512x512 priv.LB:0.33749, pub.LB:0.36576
3.Take whole model from step 2, give increasingly bigger images to the model
\n efn7 640x640 priv.LB:0.35389, pub.LB:0.39121
\n efn7 736x736 priv.LB:0.36364, pub.LB:0.40174
\n4.Take whole model from step 3, set twice loss weight for GLD v2 clean samples
\n efn7 640x640 priv.LB:0.35932, pub.LB:0.39881
\n efn7 736x736 priv.LB:0.36569, pub.LB:0.40215
1.736x736 efn7+efn6+efn5+efn5 weighted concat
\n(all train step3, weight : efn7=1.0, efn6=0.8, efn5=0.5)
\npriv.LB:0.38366, pub.LB: 0.41986
\n2.Same Config, with train step 4 for efn7
\npriv.LB:0.38677, pub.LB: 0.42328
If you have any questions, feel free to ask.
\nThank you.
[Update] solution arxiv link : https://arxiv.org/abs/2009.05132
\n
Great thanks to google and kaggle team for hosting this competition, and congrats to all participants who finished successfully. I really learned a lot during the competition through reading articles, analysing codes, and doing experiments.
\nI'd like to share my solution, and detailed solution will be uploaded to arxiv in a few days.
\nModel structure is as below.
\n
Validation set : 1 sample per class which has >=4 samples in GLD v2 clean dataset(72322/81313 classes)
\nCosine softmax : s=determined by adacos, m=0
\nWeighted cross entropy : proportional to 1/log(class cnt)
\nAugmentation : left-right flip
\nOptimizer : SGD(1e-3, momentum=0.9, decay=1e-5)
\nEmbedding Dimension : 512 for every model
\nHardware : Colab TPUs
1.Use GLD v2 clean dataset to train model to classify 81313 classes
\nefn7 512x512 priv.LB:0.30264, pub.LB:0.33907
\n2.Take efficientnet backbone from step 1, use GLD v2 total dataset to train model to classify 203094 classes
\n efn7 512x512 priv.LB:0.33749, pub.LB:0.36576
3.Take whole model from step 2, give increasingly bigger images to the model
\n efn7 640x640 priv.LB:0.35389, pub.LB:0.39121
\n efn7 736x736 priv.LB:0.36364, pub.LB:0.40174
\n4.Take whole model from step 3, set twice loss weight for GLD v2 clean samples
\n efn7 640x640 priv.LB:0.35932, pub.LB:0.39881
\n efn7 736x736 priv.LB:0.36569, pub.LB:0.40215
1.736x736 efn7+efn6+efn5+efn5 weighted concat
\n(all train step3, weight : efn7=1.0, efn6=0.8, efn5=0.5)
\npriv.LB:0.38366, pub.LB: 0.41986
\n2.Same Config, with train step 4 for efn7
\npriv.LB:0.38677, pub.LB: 0.42328
If you have any questions, feel free to ask.
\nThank you.
Congratulations! Can't wait for your ArXiv paper 😊
\nDid I understood correctly? You trained only on Colab TPUs? 😲
", "rawMarkdown": "Congratulations! Can't wait for your ArXiv paper 😊\n\nDid I understood correctly? You **trained only on Colab TPUs?** 😲", "votes": 5, "replies": [ { "id": 979371, "postDate": "2020-08-20T19:31:51.997Z", "content": "Thank you! It's too hard to write a paper..(my first time) yes, colab tpus are quite fast!
", "rawMarkdown": "Thank you! It's too hard to write a paper..(my first time) yes, colab tpus are quite fast!", "votes": 4 }, { "id": 979386, "postDate": "2020-08-20T19:42:23.623Z", "content": "Wow, that's empowering 😲 You just gave a lot of folks hope that they too can get good results in Computer Vision competitions without having a DL rig or spending to much money on cloud GPUs 😲
", "rawMarkdown": "Wow, that's empowering 😲 You just gave a lot of folks hope that they too can get good results in Computer Vision competitions without having a DL rig or spending to much money on cloud GPUs 😲", "votes": 3 } ] }, { "id": 989187, "postDate": "2020-08-28T15:38:41.300Z", "content": "This Solution is Absolutely Amazing @keetar Thanks a lot for sharing 👍✔️💯
", "rawMarkdown": "This Solution is Absolutely Amazing @keetar Thanks a lot for sharing 👍✔️💯", "votes": 3, "replies": [ { "id": 990103, "postDate": "2020-08-29T10:42:49.523Z", "content": "Thank you! :)
", "rawMarkdown": "Thank you! :)" } ] }, { "id": 978720, "postDate": "2020-08-20T10:48:01.200Z", "content": "Congratulations! Can I ask you a question ,How you concat efn7+efn6+efn5+efn5 ?
\nafter GAP
\nI use : output=tf.concat([efnb6_224x224_model_0.output,efnb6_256x256_model_1.output],axis=-1)
\nOutput dimension：（2304x2,)
\nThe submitted result shows：Notebook Exceeded Allowed Compute
Thanks! It seems output dimension is wrong.
\nMaybe you shoud use output[0] instead of output to correct it.
\nIn my case, I concatenated in this way.
\noutputs = model((image[tf.newaxis], image[tf.newaxis], image[tf.newaxis], image[tf.newaxis]))
\noutput1 = tf.math.l2_normalize(outputs[0][0])
output2 = 0.8*tf.math.l2_normalize(outputs[1][0])
\noutput3 = 0.5*tf.math.l2_normalize(outputs[2][0])
\noutput4 = 0.5*tf.math.l2_normalize(outputs[3][0])
\nfeatures = tf.concat([output1,output2,output3,output4],axis=-1)
", "rawMarkdown": "Thanks! It seems output dimension is wrong. \nMaybe you shoud use output[0] instead of output to correct it.\nIn my case, I concatenated in this way.\noutputs = model((image[tf.newaxis], image[tf.newaxis], image[tf.newaxis], image[tf.newaxis]))\noutput1 = tf.math.l2_normalize(outputs[0][0])\n\noutput2 = 0.8*tf.math.l2_normalize(outputs[1][0])\n\noutput3 = 0.5*tf.math.l2_normalize(outputs[2][0])\n\noutput4 = 0.5*tf.math.l2_normalize(outputs[3][0])\n\nfeatures = tf.concat([output1,output2,output3,output4],axis=-1)", "votes": 6 }, { "id": 978753, "postDate": "2020-08-20T11:09:25.507Z", "content": "thank you, very much!
", "rawMarkdown": "thank you, very much!" } ] }, { "id": 979542, "postDate": "2020-08-20T23:50:35.930Z", "content": "Congrats and thank you for sharing. it's a very great solution!
\nHow did you store a big dataset for Colab TPU? In this competition dataset is very big(1TB). It's difficult to handle.
Thank you! I used private GCP bucket to store data
", "rawMarkdown": "Thank you! I used private GCP bucket to store data", "votes": 3 } ] }, { "id": 978974, "postDate": "2020-08-20T14:47:29.333Z", "content": "Huge props for achieving this awesome result with just Colab TPUs, proving that this competition was pretty accessible after all.
\nI was a bit depressed after reading about \"21 P100 GPUs\" in original DELG paper or \"32 GPU cluster\" in 4th place solution but it turns out you could win by a big margin without those crazy resources after all…
", "rawMarkdown": "Huge props for achieving this awesome result with just Colab TPUs, proving that this competition was pretty accessible after all.\n\nI was a bit depressed after reading about \"21 P100 GPUs\" in original DELG paper or \"32 GPU cluster\" in 4th place solution but it turns out you could win by a big margin without those crazy resources after all...", "votes": 2, "replies": [ { "id": 979382, "postDate": "2020-08-20T19:39:25.193Z", "content": "Thank you! I was suprised during the process, too!
", "rawMarkdown": "Thank you! I was suprised during the process, too!" }, { "id": 985641, "postDate": "2020-08-25T22:33:55.890Z", "content": "TPU is a 32 GPU cluster tho
", "rawMarkdown": "TPU is a 32 GPU cluster tho", "votes": 2 } ] }, { "id": 995545, "postDate": "2020-09-02T14:45:29.890Z", "content": "Congratulations and thank you for sharing your solution 🙏
", "rawMarkdown": "Congratulations and thank you for sharing your solution 🙏", "votes": 1 }, { "id": 988109, "postDate": "2020-08-27T19:27:31.060Z", "content": "Congratulations!
\nI have a question:
\n\ns=determined by adacos
\n
Have you used static or dynamic s parameter from adacos? If static, have you calculated it using sqrt(2)*(log(C)-1) formula?
\nAnd why did you use m = 0, but not some very small value for example 1e-4?
Thank you!
\n1) I used fixed adacos. formula is right.
\n2) Because dataset is noisy, I thought trying to cluster more between same class samples could make training more hard.
Awesome! I really learned a lot.
\nCongratulations!! I
Thank you! :)
", "rawMarkdown": "Thank you! :)" } ] }, { "id": 984280, "postDate": "2020-08-25T02:37:10.993Z", "content": "Congratulation @keetar! And thanks for sharing.
\nDid you used Dropout or Batchnormalization? Did you tried smaller image sizes like 256x256 or 384x384 or EfficientNet B0-B6 ?
Thank you! I did not use dropout nor batchnormalization. I started with 256x256 at first, and dropped those models after seeing 512x512 scores much better. Similary, smaller EfficientNets had lower performance so dropped them. As you can see in my final ensemble, I used 2 efn5s, 1 efn6, 1 efn7.
", "rawMarkdown": "Thank you! I did not use dropout nor batchnormalization. I started with 256x256 at first, and dropped those models after seeing 512x512 scores much better. Similary, smaller EfficientNets had lower performance so dropped them. As you can see in my final ensemble, I used 2 efn5s, 1 efn6, 1 efn7. ", "votes": 1 } ] }, { "id": 984258, "postDate": "2020-08-25T02:02:18.967Z", "content": "Congratulations!
", "rawMarkdown": "Congratulations!", "votes": 1, "replies": [ { "id": 984475, "postDate": "2020-08-25T05:36:35.127Z", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": 1 } ] }, { "id": 983598, "postDate": "2020-08-24T12:40:19.033Z", "content": "Congratulations!
", "rawMarkdown": "Congratulations!", "votes": 1, "replies": [ { "id": 984484, "postDate": "2020-08-25T05:42:57.543Z", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": 1 } ] }, { "id": 983317, "postDate": "2020-08-24T07:11:29.890Z", "content": "Great work thanks!
", "rawMarkdown": "Great work thanks!", "votes": 1, "replies": [ { "id": 984483, "postDate": "2020-08-25T05:42:50.260Z", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": 1 } ] }, { "id": 983126, "postDate": "2020-08-24T04:29:56.773Z", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": 1, "replies": [ { "id": 984486, "postDate": "2020-08-25T05:44:07.613Z", "content": "Thanks to you!
", "rawMarkdown": "Thanks to you!", "votes": 1 } ] }, { "id": 982700, "postDate": "2020-08-23T15:38:31.993Z", "content": "Congratulations
", "rawMarkdown": "Congratulations", "votes": 1, "replies": [ { "id": 984474, "postDate": "2020-08-25T05:36:19.693Z", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": 1 } ] }, { "id": 981032, "postDate": "2020-08-22T05:49:22.733Z", "content": "Congratulations and thanks a lot for sharing so much detailed information👍.
", "rawMarkdown": "Congratulations and thanks a lot for sharing so much detailed information👍.", "votes": 1, "replies": [ { "id": 982100, "postDate": "2020-08-23T04:13:44.363Z", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": 1 } ] }, { "id": 981014, "postDate": "2020-08-22T05:28:30.963Z", "content": "@keetar Embedding Dimension : 512 for every model are you talking about Dense(512) or you use Embedding Layer
Hi, Dense(512) is what I used.
", "rawMarkdown": "Hi, Dense(512) is what I used.", "votes": 1 } ] }, { "id": 980578, "postDate": "2020-08-21T17:43:01.337Z", "content": "Congrats
", "rawMarkdown": "Congrats", "votes": 1, "replies": [ { "id": 982098, "postDate": "2020-08-23T04:13:23.917Z", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": 1 } ] }, { "id": 980354, "postDate": "2020-08-21T14:32:41.400Z", "content": "Congrats! Could you please share your training pipeline. I'm new to TPU and want to learn how it works :)
", "rawMarkdown": "Congrats! Could you please share your training pipeline. I'm new to TPU and want to learn how it works :)", "votes": 1, "replies": [ { "id": 982105, "postDate": "2020-08-23T04:21:05.850Z", "content": "Thank you! About TPU pipelines, you can refer chris deotte's notebooks in melanoma competition. https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords
", "rawMarkdown": "Thank you! About TPU pipelines, you can refer chris deotte's notebooks in melanoma competition. [https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords](url)", "votes": 1 }, { "id": 982564, "postDate": "2020-08-23T13:28:21.707Z", "content": "@keetar Thanks for the reply! One more newbie question, how many images did you put in each .tfrecords file? I generated a 40GB train.tfrecords on my custom dataset, should I split it into smaller ones?
", "rawMarkdown": "@keetar Thanks for the reply! One more newbie question, how many images did you put in each .tfrecords file? I generated a 40GB train.tfrecords on my custom dataset, should I split it into smaller ones?" }, { "id": 982625, "postDate": "2020-08-23T14:30:58.420Z", "content": "I used 128fold for GLDv2 total, and 16fold for GLDv2 clean. I don't know what size is optimal, too :)
", "rawMarkdown": "I used 128fold for GLDv2 total, and 16fold for GLDv2 clean. I don't know what size is optimal, too :)", "votes": 1 } ] }, { "id": 979749, "postDate": "2020-08-21T05:19:26.470Z", "content": "Congrats! @keetar I have a question - did you only use one type of augmentation (left-right flip) for all the training steps?
", "rawMarkdown": "Congrats! @keetar I have a question - did you only use one type of augmentation (left-right flip) for all the training steps?", "votes": 1, "replies": [ { "id": 982096, "postDate": "2020-08-23T04:12:00.030Z", "content": "Thank you, congratulations! Yes, only left-right flip was used. I thought train datasets were too large that It's hard to overfits. And another intention was not to disturb image distribution, since I don't use TTA in submissions.
", "rawMarkdown": "Thank you, congratulations! Yes, only left-right flip was used. I thought train datasets were too large that It's hard to overfits. And another intention was not to disturb image distribution, since I don't use TTA in submissions. " } ] }, { "id": 979279, "postDate": "2020-08-20T18:25:27.360Z", "content": "Hi @keetar can you please direct me to some articles or anything that i can find useful to learn more about image retrieval. Believe me, I tried many different things github repos, articles, previous year solution, videos but all of them was either a failure or not complete enough to give me a proper understanding. I have fair idea of the Image Retrieval problem, but still don't have a clue as to what is the use of index images. I read the discussions and one of them host explained but then I was confused why there are no index images for training then. I maybe totally wrong with my understanding.
\nI would be extremely thankful to you if you could list down some resources for helping me get started. Please if you can then do release your code. Thank and congratulations for your victory. 👍
Thank you! I'm new to this field so I was confused about those concepts as well. You can think test images as query, and index images as database. So, for this competition, if we submit our model(trained using train sets) that extract embeddings from images, scoring system uses it to 1) Extract embeddings for the private test and index sets 2) Create a kNN(k=100) lookup for each test sample, using the Euclidean distance between test and index embeddings 3) Score the quality of the lookups using the competition metric.
\nIn summary, this retrieval competition evaluates our model's performance by measuring 'quality of the lookups(from index sets) for each test image'.
\nYou can refer this paper for more details. https://arxiv.org/pdf/2004.01804.pdf
Thanks a ton @keetar. Just one more thing, are there any simple implementation of this task that you can direct me to.? It will extremely helpful.
", "rawMarkdown": "Thanks a ton @keetar. Just one more thing, are there any simple implementation of this task that you can direct me to.? It will extremely helpful. ", "votes": 1 }, { "id": 983455, "postDate": "2020-08-24T10:15:10.380Z", "content": "@keetar, it seems like paper link is not valid.
", "rawMarkdown": "@keetar, it seems like paper link is not valid. " }, { "id": 983792, "postDate": "2020-08-24T15:44:31.410Z", "content": "Copy and paste that link in another tab. I had the same issue.
", "rawMarkdown": "Copy and paste that link in another tab. I had the same issue. ", "votes": 1 } ] }, { "id": 979273, "postDate": "2020-08-20T18:22:51.440Z", "content": "Congratulations on first!
", "rawMarkdown": "Congratulations on first!", "votes": 1, "replies": [ { "id": 979365, "postDate": "2020-08-20T19:28:42.330Z", "content": "Thank you!
", "rawMarkdown": "Thank you!" } ] }, { "id": 979223, "postDate": "2020-08-20T17:44:09.283Z", "content": "Congrats and thanks for sharing!
", "rawMarkdown": "Congrats and thanks for sharing! ", "votes": 1, "replies": [ { "id": 979366, "postDate": "2020-08-20T19:28:57.527Z", "content": "Thank you!
", "rawMarkdown": "Thank you!" } ] }, { "id": 979159, "postDate": "2020-08-20T16:59:58.160Z", "content": "Thanks for sharing. I am interested in the arXiv upload. May I ask, is this generally all done in PyTorch?
", "rawMarkdown": "Thanks for sharing. I am interested in the arXiv upload. May I ask, is this generally all done in PyTorch?", "votes": 1, "replies": [ { "id": 979370, "postDate": "2020-08-20T19:29:46.447Z", "content": "Thanks! All works are done in tensorflow 2
", "rawMarkdown": "Thanks! All works are done in tensorflow 2", "votes": 1 } ] }, { "id": 979136, "postDate": "2020-08-20T16:42:03.347Z", "content": "Congrats man, good job
", "rawMarkdown": "Congrats man, good job", "votes": 1, "replies": [ { "id": 979383, "postDate": "2020-08-20T19:39:43.300Z", "content": "Thank you!
", "rawMarkdown": "\nThank you!" } ] }, { "id": 979039, "postDate": "2020-08-20T15:23:33.837Z", "content": "@keetar, congratulations for winning. Just to know , how much time needed to train the model ?
", "rawMarkdown": "@keetar, congratulations for winning. Just to know , how much time needed to train the model ?", "votes": 1, "replies": [ { "id": 979373, "postDate": "2020-08-20T19:35:13.357Z", "content": "Thank you! It's hard to say strictly because of too many expriment failures and so on, but if things were going smoothly It will take about 3 weeks to get my score.
", "rawMarkdown": "Thank you! It's hard to say strictly because of too many expriment failures and so on, but if things were going smoothly It will take about 3 weeks to get my score.", "votes": 1 } ] }, { "id": 978984, "postDate": "2020-08-20T14:51:53.630Z", "content": "Congrats!!! well deserved, any preprocessing or postprocessing techniques used to extract the local features?
", "rawMarkdown": "Congrats!!! well deserved, any preprocessing or postprocessing techniques used to extract the local features?", "votes": 1, "replies": [ { "id": 979376, "postDate": "2020-08-20T19:36:46.553Z", "content": "Thank you! No, I only relied on 512 global features for each model.
", "rawMarkdown": "Thank you! No, I only relied on 512 global features for each model.", "votes": 1 } ] }, { "id": 978894, "postDate": "2020-08-20T13:30:30.580Z", "content": "Congratulations! Well deserved win! :)
", "rawMarkdown": "Congratulations! Well deserved win! :)", "votes": 1, "replies": [ { "id": 978896, "postDate": "2020-08-20T13:34:47.170Z", "content": "Thank you! Congratulations to you!
", "rawMarkdown": "Thank you! Congratulations to you! ", "votes": 2 }, { "id": 980437, "postDate": "2020-08-21T15:35:52.777Z", "content": "\n\nCosine softmax : s=determined by adacos, m=0
\n
@keetar , did you really use margin=0 or is it a typo?
\nThanks
I used margin to be zero, because train datasets are quite noisy so I thought trying to cluster more between same classes could make training more hard. It's not experimentally proved.
", "rawMarkdown": " I used margin to be zero, because train datasets are quite noisy so I thought trying to cluster more between same classes could make training more hard. It's not experimentally proved.", "votes": 1 } ] }, { "id": 978847, "postDate": "2020-08-20T12:55:48.563Z", "content": "Thanks for sharing great work man!!!
", "rawMarkdown": "Thanks for sharing great work man!!! ", "votes": 1, "replies": [ { "id": 978900, "postDate": "2020-08-20T13:35:17.580Z", "content": "Thank you!!
", "rawMarkdown": "Thank you!!", "votes": 1 } ] }, { "id": 978729, "postDate": "2020-08-20T10:57:38.430Z", "content": "Thanks for sharing your strategy and congratulations on achieving the first place with a great margin. 👍
", "rawMarkdown": "Thanks for sharing your strategy and congratulations on achieving the first place with a great margin. 👍 \n\n", "votes": 1, "replies": [ { "id": 978746, "postDate": "2020-08-20T11:06:03.083Z", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": 2 } ] }, { "id": 978687, "postDate": "2020-08-20T10:27:16.353Z", "content": "Congratulations! May I know were you using TPU for this? What learning rate / learning rate schedule u used in all the steps?
", "rawMarkdown": "Congratulations! May I know were you using TPU for this? What learning rate / learning rate schedule u used in all the steps?", "votes": 1, "replies": [ { "id": 978701, "postDate": "2020-08-20T10:35:02.390Z", "content": "Thanks! I used colab tpus. I forgot to mention optimizer! SGD(lr=1e-3, momentum=0.9, decay=1e-5) and no scheduling was used. updated
", "rawMarkdown": "Thanks! I used colab tpus. I forgot to mention optimizer! SGD(lr=1e-3, momentum=0.9, decay=1e-5) and no scheduling was used. updated", "votes": 4 }, { "id": 978735, "postDate": "2020-08-20T11:02:11.353Z", "content": "I used efnb4 to train on colab TPUsV2, and after running for several epochs， the program crashed.
\nhow big is you batch_size on colab tpus?
I experienced the crash, too. After experiments, batchsize 4 or 8 for efn7, and 8 or 16 for efn 6,5 were used depending on image sizes.(per TPU)
", "rawMarkdown": " I experienced the crash, too. After experiments, batchsize 4 or 8 for efn7, and 8 or 16 for efn 6,5 were used depending on image sizes.(per TPU)", "votes": 6 }, { "id": 983656, "postDate": "2020-08-24T13:30:20.313Z", "content": "Hey Keetar, would u mind telling us also the individual model performance of efnb6 and efnb5?
", "rawMarkdown": "Hey Keetar, would u mind telling us also the individual model performance of efnb6 and efnb5?", "votes": 1 } ] }, { "id": 978653, "postDate": "2020-08-20T09:57:44.310Z", "content": "Hi Keetar, Congratulation on achieving 1st place with a huge margin.
\nI used a similar structure and setup as your first step. But with limited time and TPU quota, I only finished training 4 epochs of EFN-B6 and and 2 epochs of EFN-B7. They both only obtained about .27 in pub.LB.
\nHow many epochs of training set have you fitted to achieve each milestone?
\nThanks.
", "rawMarkdown": "Hi Keetar, Congratulation on achieving 1st place with a huge margin.\n\nI used a similar structure and setup as your first step. But with limited time and TPU quota, I only finished training 4 epochs of EFN-B6 and and 2 epochs of EFN-B7. They both only obtained about .27 in pub.LB.\n\nHow many epochs of training set have you fitted to achieve each milestone?\n\nThanks.", "votes": 1, "replies": [ { "id": 978723, "postDate": "2020-08-20T10:48:56.450Z", "content": "Thank you!
\n[Updated]
\nIt took 35epochs for 1st step, 13epochs for 2nd step, 5epochs for 3rd step, and 4 epochs for 4th step.
Congrats on achieving such an insanely high score!
\nWhat was the hardware config/training times for each step?
", "rawMarkdown": "Congrats on achieving such an insanely high score!\n\nWhat was the hardware config/training times for each step?", "votes": 1, "replies": [ { "id": 978683, "postDate": "2020-08-20T10:22:04.803Z", "content": "Thank you! I used colab TPUs for this work.
\n[Updated]
\nIt took 149hours for 1st step, 150hours for 2nd step, 80 hours for 3rd, 64 hours for 4th.
Thanks for the extremely detailed write-up 🙏🙏🙏. It feels like a tutorial \"here is how to pwn this competition\", not a formal paper \"hey look how awesome I am but I won't give you details\" that we usually see.
\nI can see a a bloodbath coming in Recognition challenge, where everyone will copy this strategy for global feature network :))
\nBtw turns out that a year ago we were colleagues (but in different countries) xD
", "rawMarkdown": "Thanks for the extremely detailed write-up 🙏🙏🙏. It feels like a tutorial \"here is how to pwn this competition\", not a formal paper \"hey look how awesome I am but I won't give you details\" that we usually see.\n\nI can see a a bloodbath coming in Recognition challenge, where everyone will copy this strategy for global feature network :)) \n\nBtw turns out that a year ago we were colleagues (but in different countries) xD", "votes": 2, "replies": [ { "id": 985666, "postDate": "2020-08-25T23:18:41.957Z", "content": "Thanks for your review! Yes, It's going to be a fierce competition. Maybe we can learn more by that :)
\nIndeed we were colleagues! It's nice to see you 🙂
Btw, during training - did you just resize the images to 512x512 (without preserving aspect ratio), or you did croppings around the center as in the DELG paper? During inference, you also resized images without preserving aspect ratio?
", "rawMarkdown": "Btw, during training - did you just resize the images to 512x512 (without preserving aspect ratio), or you did croppings around the center as in the DELG paper? During inference, you also resized images without preserving aspect ratio?", "votes": 1 }, { "id": 985685, "postDate": "2020-08-25T23:50:12.087Z", "content": "Yes, I did not preserve any aspect ratio. And also cropping was not used at all.
", "rawMarkdown": "Yes, I did not preserve any aspect ratio. And also cropping was not used at all.", "votes": 1 } ] }, { "id": 984393, "postDate": "2020-08-25T04:40:39.743Z", "content": "Congratulations Master :)
", "rawMarkdown": "Congratulations Master :)", "votes": 2, "replies": [ { "id": 984488, "postDate": "2020-08-25T05:44:31.803Z", "content": "Thank you! :)
", "rawMarkdown": "Thank you! :)", "votes": 1 } ] }, { "id": 980517, "postDate": "2020-08-21T16:53:32.960Z", "content": "Good Job on first @keetar
", "rawMarkdown": "Good Job on first @keetar ", "votes": 2, "replies": [ { "id": 982097, "postDate": "2020-08-23T04:12:55.177Z", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": 1 } ] }, { "id": 979855, "postDate": "2020-08-21T06:50:35.557Z", "content": "I congratulate @keetar for sharing the details and for his excellent achievement. I do not have any doubt, a lot of hard work and discipline has gone in to achieve that score.
\nHowever, I have a small point to make to the organizers. Will it not be better to make a 'research' competiton at least 3-4 months long? I say this, so that competitiors can come up with some new techniques, that can make improvements over the benchmark scores, without using traditional methods used for better scores. This is in no way to criticize the traditional methods. But to encourage a practice that can make this community better , in terms of paper reading, contemporary technique implementation and perhaps, to innovate a new technique.
\nOtherwise, I feel, kaggle is increasingly becoming a battleground for GPU/TPU based execution of pretrained models, using traditional methods.
\n@andrefaraujo @tobiasweyand @maggie
", "rawMarkdown": "I congratulate @keetar for sharing the details and for his excellent achievement. I do not have any doubt, a lot of hard work and discipline has gone in to achieve that score.\n\nHowever, I have a small point to make to the organizers. Will it not be better to make a 'research' competiton at least 3-4 months long? I say this, so that competitiors can come up with some new techniques, that can make improvements over the benchmark scores, without using traditional methods used for better scores. This is in no way to criticize the traditional methods. But to encourage a practice that can make this community better , in terms of paper reading, contemporary technique implementation and perhaps, to innovate a new technique. \n\nOtherwise, I feel, kaggle is increasingly becoming a battleground for GPU/TPU based execution of pretrained models, using traditional methods. \n\n@andrefaraujo @tobiasweyand @maggie ", "votes": 2, "replies": [ { "id": 979890, "postDate": "2020-08-21T07:24:18.520Z", "content": "I started way too late with this competition, 11 days before the deadline and I was working on creating a new loss function and reducing the computation requirements. I succeeded because it scored .076 with only 3 hours of GPU training, but I didn't had the time to train it to convergence, thanks to GCP's new no GPU quota for you policy.
\nMaybe if i started sooner I could have trained my model, but such a computation intensive competitions should be at least 3 months long.
", "rawMarkdown": "I started way too late with this competition, 11 days before the deadline and I was working on creating a new loss function and reducing the computation requirements. I succeeded because it scored .076 with only 3 hours of GPU training, but I didn't had the time to train it to convergence, thanks to GCP's new no GPU quota for you policy.\n\nMaybe if i started sooner I could have trained my model, but such a computation intensive competitions should be at least 3 months long.\n\n", "votes": 1 }, { "id": 979898, "postDate": "2020-08-21T07:30:16.193Z", "content": "I think more than 300 competitiors joined , while 3-4 weeks were remaining, which includes me as well. I too spent a lot of time on reading papers and to create a new loss function. I learned a lot. But on leaderboard , I agree with you, you can hardly make a mark, with less GPU memory.
", "rawMarkdown": "I think more than 300 competitiors joined , while 3-4 weeks were remaining, which includes me as well. I too spent a lot of time on reading papers and to create a new loss function. I learned a lot. But on leaderboard , I agree with you, you can hardly make a mark, with less GPU memory. " } ] }, { "id": 979644, "postDate": "2020-08-21T02:59:24.157Z", "content": "Congrats on 1st place and thanks for sharing details solution @keetar
", "rawMarkdown": "Congrats on 1st place and thanks for sharing details solution @keetar ", "votes": 2, "replies": [ { "id": 979708, "postDate": "2020-08-21T04:29:57.103Z", "content": "Thank you!
", "rawMarkdown": "Thank you!" } ] }, { "id": 979602, "postDate": "2020-08-21T01:40:12.007Z", "content": "Congrats and thanks for sharing!
\nHave you try to train with index set GLD-v2?
\nDoes \"use GLD v2 total dataset to train model to classify 203094 classes\" mean using train.csv to train?
Thank you! Congratulations to you!
\nYes I trained using both GLD-v2 clean and GLD-v2 total dataset.
\nGLD-v2 total dataset has 203094 classes. You can find link in the external data thread.
Congratulation 👏
", "rawMarkdown": "Congratulation 👏", "replies": [ { "id": 990104, "postDate": "2020-08-29T10:42:56.943Z", "content": "Thank you!
", "rawMarkdown": "Thank you!" } ] }, { "id": 993951, "postDate": "2020-09-01T08:59:31.007Z", "content": "Great solution!! Thanks for sharing.
\nHowever, I am just wonder why you used Optimizer SGD, as I know training with Adam is much faster?
", "rawMarkdown": "Great solution!! Thanks for sharing.\n\nHowever, I am just wonder why you used Optimizer SGD, as I know training with Adam is much faster?", "votes": 1 }, { "id": 1287619, "postDate": "2021-04-29T08:22:57.087Z", "content": "For ensemble How did you come up with the weights for the model?
", "rawMarkdown": "For ensemble How did you come up with the weights for the model?" }, { "id": 1243934, "postDate": "2021-03-18T15:38:44.673Z", "content": "Interesting training strategy. How did you select the image sizes? Is there a heuristic behind the numbers?
", "rawMarkdown": "Interesting training strategy. How did you select the image sizes? Is there a heuristic behind the numbers? " }, { "id": 1044005, "postDate": "2020-10-09T12:18:00.713Z", "content": "Hi, how you can use cosine softmax in code of your submission, I want to know how to use?
\nThanks.
Hi @keetar , did you use any trick like focal loss or label smoothing?
", "rawMarkdown": "Hi @keetar , did you use any trick like focal loss or label smoothing?", "replies": [ { "id": 1036517, "postDate": "2020-10-03T21:46:01.480Z", "content": "I tried label smoothing but not worked well.
", "rawMarkdown": "I tried label smoothing but not worked well." } ] }, { "id": 1003621, "postDate": "2020-09-09T06:12:28.513Z", "content": "superb work
", "rawMarkdown": "superb work\n" }, { "id": 1001382, "postDate": "2020-09-07T09:50:56.167Z", "content": "Hello, I want to know how to clean the dataset and how you make validation set? I'm just a newbie with python and this problem.
\nOr do you have example source to do the cleaning data step?
\nThank you.
super
", "rawMarkdown": "super" }, { "id": 999356, "postDate": "2020-09-05T15:28:28.770Z", "content": "Thanks for sharing your solution. As an image retrieval beginner, could you share a link to what GAP and DNN stand for? Thanks in advance and congratulations on your win!
", "rawMarkdown": "Thanks for sharing your solution. As an image retrieval beginner, could you share a link to what GAP and DNN stand for? Thanks in advance and congratulations on your win!", "replies": [ { "id": 1038467, "postDate": "2020-10-05T19:53:44.317Z", "content": "GAP: Global Average Pooling
\nDNN: Deep Nueral Network
I am sure you know this by now though @yassinealouini
", "rawMarkdown": "GAP: Global Average Pooling \nDNN: Deep Nueral Network\n\nI am sure you know this by now though @yassinealouini " } ] }, { "id": 994429, "postDate": "2020-09-01T15:19:29.590Z", "content": "Congratulation.(And busy at denoting down the structure xDDD)
", "rawMarkdown": "Congratulation.(And busy at denoting down the structure xDDD)" }, { "id": 990597, "postDate": "2020-08-29T18:00:05.107Z", "content": "Congratulations!
\nThanks for the well-written solution.
Great work. Congratulations!
", "rawMarkdown": "Great work. Congratulations!\n" }, { "id": 1439655, "postDate": "2021-08-03T17:15:14.197Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1234736, "postDate": "2021-03-11T14:33:43.470Z", "content": "Great ! Thanks for sharing. Congrats :)
", "rawMarkdown": "Great ! Thanks for sharing. Congrats :)" }, { "id": 991833, "postDate": "2020-08-30T17:25:35.893Z", "content": "Thanks for sharing.. great insights!
", "rawMarkdown": "Thanks for sharing.. great insights!" } ], "comments": [ { "id": 979107, "author_name": "Chan Kha Vu", "author_url": "", "post_date": "2020-08-20T16:10:57.210000", "content": "Congratulations! Can't wait for your ArXiv paper 😊
\nDid I understood correctly? You trained only on Colab TPUs? 😲
", "votes": 5, "replies": [ { "id": 979371, "author_name": "keetar", "author_url": "", "post_date": "2020-08-20T19:31:51.997000", "content": "Thank you! It's too hard to write a paper..(my first time) yes, colab tpus are quite fast!
", "votes": 4, "replies": [] }, { "id": 979386, "author_name": "Chan Kha Vu", "author_url": "", "post_date": "2020-08-20T19:42:23.623000", "content": "Wow, that's empowering 😲 You just gave a lot of folks hope that they too can get good results in Computer Vision competitions without having a DL rig or spending to much money on cloud GPUs 😲
", "votes": 3, "replies": [] } ] }, { "id": 989187, "author_name": "Vetrivel-PS", "author_url": "", "post_date": "2020-08-28T15:38:41.300000", "content": "This Solution is Absolutely Amazing @keetar Thanks a lot for sharing 👍✔️💯
", "votes": 3, "replies": [ { "id": 990103, "author_name": "keetar", "author_url": "", "post_date": "2020-08-29T10:42:49.523000", "content": "Thank you! :)
", "votes": 0, "replies": [] } ] }, { "id": 978720, "author_name": "QYQ", "author_url": "", "post_date": "2020-08-20T10:48:01.200000", "content": "Congratulations! Can I ask you a question ,How you concat efn7+efn6+efn5+efn5 ?
\nafter GAP
\nI use : output=tf.concat([efnb6_224x224_model_0.output,efnb6_256x256_model_1.output],axis=-1)
\nOutput dimension：（2304x2,)
\nThe submitted result shows：Notebook Exceeded Allowed Compute
Thanks! It seems output dimension is wrong.
\nMaybe you shoud use output[0] instead of output to correct it.
\nIn my case, I concatenated in this way.
\noutputs = model((image[tf.newaxis], image[tf.newaxis], image[tf.newaxis], image[tf.newaxis]))
\noutput1 = tf.math.l2_normalize(outputs[0][0])
output2 = 0.8*tf.math.l2_normalize(outputs[1][0])
\noutput3 = 0.5*tf.math.l2_normalize(outputs[2][0])
\noutput4 = 0.5*tf.math.l2_normalize(outputs[3][0])
\nfeatures = tf.concat([output1,output2,output3,output4],axis=-1)
", "votes": 6, "replies": [] }, { "id": 978753, "author_name": "QYQ", "author_url": "", "post_date": "2020-08-20T11:09:25.507000", "content": "thank you, very much!
", "votes": 0, "replies": [] } ] }, { "id": 979542, "author_name": "tereka", "author_url": "", "post_date": "2020-08-20T23:50:35.930000", "content": "Congrats and thank you for sharing. it's a very great solution!
\nHow did you store a big dataset for Colab TPU? In this competition dataset is very big(1TB). It's difficult to handle.
Thank you! I used private GCP bucket to store data
", "votes": 3, "replies": [] } ] }, { "id": 978974, "author_name": "qiubit", "author_url": "", "post_date": "2020-08-20T14:47:29.333000", "content": "Huge props for achieving this awesome result with just Colab TPUs, proving that this competition was pretty accessible after all.
\nI was a bit depressed after reading about \"21 P100 GPUs\" in original DELG paper or \"32 GPU cluster\" in 4th place solution but it turns out you could win by a big margin without those crazy resources after all…
", "votes": 2, "replies": [ { "id": 979382, "author_name": "keetar", "author_url": "", "post_date": "2020-08-20T19:39:25.193000", "content": "Thank you! I was suprised during the process, too!
", "votes": 0, "replies": [] }, { "id": 985641, "author_name": "Surui Li", "author_url": "", "post_date": "2020-08-25T22:33:55.890000", "content": "TPU is a 32 GPU cluster tho
", "votes": 2, "replies": [] } ] }, { "id": 995545, "author_name": "", "author_url": "", "post_date": "2020-09-02T14:45:29.890000", "content": "Congratulations and thank you for sharing your solution 🙏
", "votes": 1, "replies": [] }, { "id": 988109, "author_name": "Rauf Yagfarov", "author_url": "", "post_date": "2020-08-27T19:27:31.060000", "content": "Congratulations!
\nI have a question:
\n\ns=determined by adacos
\n
Have you used static or dynamic s parameter from adacos? If static, have you calculated it using sqrt(2)*(log(C)-1) formula?
\nAnd why did you use m = 0, but not some very small value for example 1e-4?
Thank you!
\n1) I used fixed adacos. formula is right.
\n2) Because dataset is noisy, I thought trying to cluster more between same class samples could make training more hard.
Awesome! I really learned a lot.
\nCongratulations!! I
Thank you! :)
", "votes": 0, "replies": [] } ] }, { "id": 984280, "author_name": "Giba", "author_url": "", "post_date": "2020-08-25T02:37:10.993000", "content": "Congratulation @keetar! And thanks for sharing.
\nDid you used Dropout or Batchnormalization? Did you tried smaller image sizes like 256x256 or 384x384 or EfficientNet B0-B6 ?
Thank you! I did not use dropout nor batchnormalization. I started with 256x256 at first, and dropped those models after seeing 512x512 scores much better. Similary, smaller EfficientNets had lower performance so dropped them. As you can see in my final ensemble, I used 2 efn5s, 1 efn6, 1 efn7.
", "votes": 1, "replies": [] } ] }, { "id": 984258, "author_name": "Shinuk Yi", "author_url": "", "post_date": "2020-08-25T02:02:18.967000", "content": "Congratulations!
", "votes": 1, "replies": [ { "id": 984475, "author_name": "keetar", "author_url": "", "post_date": "2020-08-25T05:36:35.127000", "content": "Thank you!
", "votes": 1, "replies": [] } ] }, { "id": 983598, "author_name": "Sanchit Goel", "author_url": "", "post_date": "2020-08-24T12:40:19.033000", "content": "Congratulations!
", "votes": 1, "replies": [ { "id": 984484, "author_name": "keetar", "author_url": "", "post_date": "2020-08-25T05:42:57.543000", "content": "Thank you!
", "votes": 1, "replies": [] } ] }, { "id": 983317, "author_name": "Aakrit Singhal", "author_url": "", "post_date": "2020-08-24T07:11:29.890000", "content": "Great work thanks!
", "votes": 1, "replies": [ { "id": 984483, "author_name": "keetar", "author_url": "", "post_date": "2020-08-25T05:42:50.260000", "content": "Thank you!
", "votes": 1, "replies": [] } ] }, { "id": 983126, "author_name": "Adonis Page", "author_url": "", "post_date": "2020-08-24T04:29:56.773000", "content": "Thank you!
", "votes": 1, "replies": [ { "id": 984486, "author_name": "keetar", "author_url": "", "post_date": "2020-08-25T05:44:07.613000", "content": "Thanks to you!
", "votes": 1, "replies": [] } ] }, { "id": 982700, "author_name": "Vignesh", "author_url": "", "post_date": "2020-08-23T15:38:31.993000", "content": "Congratulations
", "votes": 1, "replies": [ { "id": 984474, "author_name": "keetar", "author_url": "", "post_date": "2020-08-25T05:36:19.693000", "content": "Thank you!
", "votes": 1, "replies": [] } ] }, { "id": 981032, "author_name": "Utcarsh Agrawal", "author_url": "", "post_date": "2020-08-22T05:49:22.733000", "content": "Congratulations and thanks a lot for sharing so much detailed information👍.
", "votes": 1, "replies": [ { "id": 982100, "author_name": "keetar", "author_url": "", "post_date": "2020-08-23T04:13:44.363000", "content": "Thank you!
", "votes": 1, "replies": [] } ] }, { "id": 981014, "author_name": "Hitesh Gorana", "author_url": "", "post_date": "2020-08-22T05:28:30.963000", "content": "@keetar Embedding Dimension : 512 for every model are you talking about Dense(512) or you use Embedding Layer
Hi, Dense(512) is what I used.
", "votes": 1, "replies": [] } ] }, { "id": 980578, "author_name": "Raoof Naushad", "author_url": "", "post_date": "2020-08-21T17:43:01.337000", "content": "Congrats
", "votes": 1, "replies": [ { "id": 982098, "author_name": "keetar", "author_url": "", "post_date": "2020-08-23T04:13:23.917000", "content": "Thank you!
", "votes": 1, "replies": [] } ] }, { "id": 980354, "author_name": "Zacchaeus", "author_url": "", "post_date": "2020-08-21T14:32:41.400000", "content": "Congrats! Could you please share your training pipeline. I'm new to TPU and want to learn how it works :)
", "votes": 1, "replies": [ { "id": 982105, "author_name": "keetar", "author_url": "", "post_date": "2020-08-23T04:21:05.850000", "content": "Thank you! About TPU pipelines, you can refer chris deotte's notebooks in melanoma competition. https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords
", "votes": 1, "replies": [] }, { "id": 982564, "author_name": "Zacchaeus", "author_url": "", "post_date": "2020-08-23T13:28:21.707000", "content": "@keetar Thanks for the reply! One more newbie question, how many images did you put in each .tfrecords file? I generated a 40GB train.tfrecords on my custom dataset, should I split it into smaller ones?
", "votes": 0, "replies": [] }, { "id": 982625, "author_name": "keetar", "author_url": "", "post_date": "2020-08-23T14:30:58.420000", "content": "I used 128fold for GLDv2 total, and 16fold for GLDv2 clean. I don't know what size is optimal, too :)
", "votes": 1, "replies": [] } ] }, { "id": 979749, "author_name": "Jeremy Ma", "author_url": "", "post_date": "2020-08-21T05:19:26.470000", "content": "Congrats! @keetar I have a question - did you only use one type of augmentation (left-right flip) for all the training steps?
", "votes": 1, "replies": [ { "id": 982096, "author_name": "keetar", "author_url": "", "post_date": "2020-08-23T04:12:00.030000", "content": "Thank you, congratulations! Yes, only left-right flip was used. I thought train datasets were too large that It's hard to overfits. And another intention was not to disturb image distribution, since I don't use TTA in submissions.
", "votes": 0, "replies": [] } ] }, { "id": 979279, "author_name": "torch", "author_url": "", "post_date": "2020-08-20T18:25:27.360000", "content": "Hi @keetar can you please direct me to some articles or anything that i can find useful to learn more about image retrieval. Believe me, I tried many different things github repos, articles, previous year solution, videos but all of them was either a failure or not complete enough to give me a proper understanding. I have fair idea of the Image Retrieval problem, but still don't have a clue as to what is the use of index images. I read the discussions and one of them host explained but then I was confused why there are no index images for training then. I maybe totally wrong with my understanding.
\nI would be extremely thankful to you if you could list down some resources for helping me get started. Please if you can then do release your code. Thank and congratulations for your victory. 👍
Thank you! I'm new to this field so I was confused about those concepts as well. You can think test images as query, and index images as database. So, for this competition, if we submit our model(trained using train sets) that extract embeddings from images, scoring system uses it to 1) Extract embeddings for the private test and index sets 2) Create a kNN(k=100) lookup for each test sample, using the Euclidean distance between test and index embeddings 3) Score the quality of the lookups using the competition metric.
\nIn summary, this retrieval competition evaluates our model's performance by measuring 'quality of the lookups(from index sets) for each test image'.
\nYou can refer this paper for more details. https://arxiv.org/pdf/2004.01804.pdf
Thanks a ton @keetar. Just one more thing, are there any simple implementation of this task that you can direct me to.? It will extremely helpful.
", "votes": 1, "replies": [] }, { "id": 983455, "author_name": "Rashmi Margani", "author_url": "", "post_date": "2020-08-24T10:15:10.380000", "content": "@keetar, it seems like paper link is not valid.
", "votes": 0, "replies": [] }, { "id": 983792, "author_name": "torch", "author_url": "", "post_date": "2020-08-24T15:44:31.410000", "content": "Copy and paste that link in another tab. I had the same issue.
", "votes": 1, "replies": [] } ] }, { "id": 979273, "author_name": "Brandon Nova", "author_url": "", "post_date": "2020-08-20T18:22:51.440000", "content": "Congratulations on first!
", "votes": 1, "replies": [ { "id": 979365, "author_name": "keetar", "author_url": "", "post_date": "2020-08-20T19:28:42.330000", "content": "Thank you!
", "votes": 0, "replies": [] } ] }, { "id": 979223, "author_name": "Abishek Sudarshan", "author_url": "", "post_date": "2020-08-20T17:44:09.283000", "content": "Congrats and thanks for sharing!
", "votes": 1, "replies": [ { "id": 979366, "author_name": "keetar", "author_url": "", "post_date": "2020-08-20T19:28:57.527000", "content": "Thank you!
", "votes": 0, "replies": [] } ] }, { "id": 979159, "author_name": "yqz", "author_url": "", "post_date": "2020-08-20T16:59:58.160000", "content": "Thanks for sharing. I am interested in the arXiv upload. May I ask, is this generally all done in PyTorch?
", "votes": 1, "replies": [ { "id": 979370, "author_name": "keetar", "author_url": "", "post_date": "2020-08-20T19:29:46.447000", "content": "Thanks! All works are done in tensorflow 2
", "votes": 1, "replies": [] } ] }, { "id": 979136, "author_name": "Bo Peng", "author_url": "", "post_date": "2020-08-20T16:42:03.347000", "content": "Congrats man, good job
", "votes": 1, "replies": [ { "id": 979383, "author_name": "keetar", "author_url": "", "post_date": "2020-08-20T19:39:43.300000", "content": "Thank you!
", "votes": 0, "replies": [] } ] }, { "id": 979039, "author_name": "Md Fahim", "author_url": "", "post_date": "2020-08-20T15:23:33.837000", "content": "@keetar, congratulations for winning. Just to know , how much time needed to train the model ?
", "votes": 1, "replies": [ { "id": 979373, "author_name": "keetar", "author_url": "", "post_date": "2020-08-20T19:35:13.357000", "content": "Thank you! It's hard to say strictly because of too many expriment failures and so on, but if things were going smoothly It will take about 3 weeks to get my score.
", "votes": 1, "replies": [] } ] }, { "id": 978984, "author_name": "Rashmi Margani", "author_url": "", "post_date": "2020-08-20T14:51:53.630000", "content": "Congrats!!! well deserved, any preprocessing or postprocessing techniques used to extract the local features?
", "votes": 1, "replies": [ { "id": 979376, "author_name": "keetar", "author_url": "", "post_date": "2020-08-20T19:36:46.553000", "content": "Thank you! No, I only relied on 512 global features for each model.
", "votes": 1, "replies": [] } ] }, { "id": 978894, "author_name": "Eduardo Rocha de Andrade", "author_url": "", "post_date": "2020-08-20T13:30:30.580000", "content": "Congratulations! Well deserved win! :)
", "votes": 1, "replies": [ { "id": 978896, "author_name": "keetar", "author_url": "", "post_date": "2020-08-20T13:34:47.170000", "content": "Thank you! Congratulations to you!
", "votes": 2, "replies": [] }, { "id": 980437, "author_name": "Eduardo Rocha de Andrade", "author_url": "", "post_date": "2020-08-21T15:35:52.777000", "content": "\n\nCosine softmax : s=determined by adacos, m=0
\n
@keetar , did you really use margin=0 or is it a typo?
\nThanks
I used margin to be zero, because train datasets are quite noisy so I thought trying to cluster more between same classes could make training more hard. It's not experimentally proved.
", "votes": 1, "replies": [] } ] }, { "id": 978847, "author_name": "Shivam", "author_url": "", "post_date": "2020-08-20T12:55:48.563000", "content": "Thanks for sharing great work man!!!
", "votes": 1, "replies": [ { "id": 978900, "author_name": "keetar", "author_url": "", "post_date": "2020-08-20T13:35:17.580000", "content": "Thank you!!
", "votes": 1, "replies": [] } ] }, { "id": 978729, "author_name": "adimalhotra11", "author_url": "", "post_date": "2020-08-20T10:57:38.430000", "content": "Thanks for sharing your strategy and congratulations on achieving the first place with a great margin. 👍
", "votes": 1, "replies": [ { "id": 978746, "author_name": "keetar", "author_url": "", "post_date": "2020-08-20T11:06:03.083000", "content": "Thank you!
", "votes": 2, "replies": [] } ] }, { "id": 978687, "author_name": "", "author_url": "", "post_date": "2020-08-20T10:27:16.353000", "content": "", "votes": 1, "replies": [ { "id": 978701, "author_name": "", "author_url": "", "post_date": "2020-08-20T10:35:02.390000", "content": "", "votes": 4, "replies": [] }, { "id": 978735, "author_name": "", "author_url": "", "post_date": "2020-08-20T11:02:11.353000", "content": "", "votes": 0, "replies": [] }, { "id": 978768, "author_name": "", "author_url": "", "post_date": "2020-08-20T11:23:52.957000", "content": "", "votes": 6, "replies": [] }, { "id": 983656, "author_name": "", "author_url": "", "post_date": "2020-08-24T13:30:20.313000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 978653, "author_name": "", "author_url": "", "post_date": "2020-08-20T09:57:44.310000", "content": "", "votes": 1, "replies": [ { "id": 978723, "author_name": "", "author_url": "", "post_date": "2020-08-20T10:48:56.450000", "content": "", "votes": 5, "replies": [] } ] }, { "id": 978604, "author_name": "", "author_url": "", "post_date": "2020-08-20T09:27:20.120000", "content": "", "votes": 1, "replies": [ { "id": 978683, "author_name": "", "author_url": "", "post_date": "2020-08-20T10:22:04.803000", "content": "", "votes": 5, "replies": [] } ] }, { "id": 985649, "author_name": "", "author_url": "", "post_date": "2020-08-25T22:49:07.847000", "content": "", "votes": 2, "replies": [ { "id": 985666, "author_name": "", "author_url": "", "post_date": "2020-08-25T23:18:41.957000", "content": "", "votes": 1, "replies": [] }, { "id": 985677, "author_name": "", "author_url": "", "post_date": "2020-08-25T23:40:51.567000", "content": "", "votes": 1, "replies": [] }, { "id": 985685, "author_name": "", "author_url": "", "post_date": "2020-08-25T23:50:12.087000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 984393, "author_name": "", "author_url": "", "post_date": "2020-08-25T04:40:39.743000", "content": "", "votes": 2, "replies": [ { "id": 984488, "author_name": "", "author_url": "", "post_date": "2020-08-25T05:44:31.803000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 980517, "author_name": "", "author_url": "", "post_date": "2020-08-21T16:53:32.960000", "content": "", "votes": 2, "replies": [ { "id": 982097, "author_name": "", "author_url": "", "post_date": "2020-08-23T04:12:55.177000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 979855, "author_name": "", "author_url": "", "post_date": "2020-08-21T06:50:35.557000", "content": "", "votes": 2, "replies": [ { "id": 979890, "author_name": "", "author_url": "", "post_date": "2020-08-21T07:24:18.520000", "content": "", "votes": 1, "replies": [] }, { "id": 979898, "author_name": "", "author_url": "", "post_date": "2020-08-21T07:30:16.193000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 979644, "author_name": "", "author_url": "", "post_date": "2020-08-21T02:59:24.157000", "content": "", "votes": 2, "replies": [ { "id": 979708, "author_name": "", "author_url": "", "post_date": "2020-08-21T04:29:57.103000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 979602, "author_name": "", "author_url": "", "post_date": "2020-08-21T01:40:12.007000", "content": "", "votes": 2, "replies": [ { "id": 979703, "author_name": "", "author_url": "", "post_date": "2020-08-21T04:27:11.777000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 988314, "author_name": "", "author_url": "", "post_date": "2020-08-28T01:10:33.060000", "content": "", "votes": 0, "replies": [ { "id": 990104, "author_name": "", "author_url": "", "post_date": "2020-08-29T10:42:56.943000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 993951, "author_name": "", "author_url": "", "post_date": "2020-09-01T08:59:31.007000", "content": "", "votes": 1, "replies": [] }, { "id": 1287619, "author_name": "", "author_url": "", "post_date": "2021-04-29T08:22:57.087000", "content": "", "votes": 0, "replies": [] }, { "id": 1243934, "author_name": "", "author_url": "", "post_date": "2021-03-18T15:38:44.673000", "content": "", "votes": 0, "replies": [] }, { "id": 1044005, "author_name": "", "author_url": "", "post_date": "2020-10-09T12:18:00.713000", "content": "", "votes": 0, "replies": [] }, { "id": 1004590, "author_name": "", "author_url": "", "post_date": "2020-09-09T20:20:06", "content": "", "votes": 0, "replies": [ { "id": 1036517, "author_name": "", "author_url": "", "post_date": "2020-10-03T21:46:01.480000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1003621, "author_name": "", "author_url": "", "post_date": "2020-09-09T06:12:28.513000", "content": "", "votes": 0, "replies": [] }, { "id": 1001382, "author_name": "", "author_url": "", "post_date": "2020-09-07T09:50:56.167000", "content": "", "votes": 0, "replies": [] }, { "id": 1000263, "author_name": "", "author_url": "", "post_date": "2020-09-06T12:21:41.670000", "content": "", "votes": 0, "replies": [] }, { "id": 999356, "author_name": "", "author_url": "", "post_date": "2020-09-05T15:28:28.770000", "content": "", "votes": 0, "replies": [ { "id": 1038467, "author_name": "", "author_url": "", "post_date": "2020-10-05T19:53:44.317000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 994429, "author_name": "", "author_url": "", "post_date": "2020-09-01T15:19:29.590000", "content": "", "votes": 0, "replies": [] }, { "id": 990597, "author_name": "", "author_url": "", "post_date": "2020-08-29T18:00:05.107000", "content": "", "votes": 0, "replies": [] }, { "id": 990236, "author_name": "", "author_url": "", "post_date": "2020-08-29T13:05:25.857000", "content": "", "votes": 0, "replies": [] }, { "id": 1439655, "author_name": "", "author_url": "", "post_date": "2021-08-03T17:15:14.197000", "content": "", "votes": 0, "replies": [] }, { "id": 1234736, "author_name": "", "author_url": "", "post_date": "2021-03-11T14:33:43.470000", "content": "", "votes": 0, "replies": [] }, { "id": 991833, "author_name": "", "author_url": "", "post_date": "2020-08-30T17:25:35.893000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "978542": "[Update] solution arxiv link : https://arxiv.org/abs/2009.05132\n~~[Update] submission to arxiv is on hold. Please refer to attached pdf paper.(modified version)~~\n# \n Great thanks to google and kaggle team for hosting this competition, and congrats to all participants who finished successfully. I really learned a lot during the competition through reading articles, analysing codes, and doing experiments.\n\nI'd like to share my solution, and detailed solution will be uploaded to arxiv in a few days.\n\nModel structure is as below.\n\n\n# Basic Configuration\nValidation set : 1 sample per class which has >=4 samples in GLD v2 clean dataset(72322/81313 classes)\nCosine softmax : s=determined by adacos, m=0\nWeighted cross entropy : proportional to 1/log(class cnt)\nAugmentation : left-right flip\nOptimizer : SGD(1e-3, momentum=0.9, decay=1e-5)\nEmbedding Dimension : 512 for every model\nHardware : Colab TPUs\n\n# Training Strategy\n1.Use GLD v2 clean dataset to train model to classify 81313 classes\nefn7 512x512 priv.LB:0.30264, pub.LB:0.33907\n2.Take efficientnet backbone from step 1, use GLD v2 total dataset to train model to classify 203094 classes\n efn7 512x512 priv.LB:0.33749, pub.LB:0.36576\n\n3.Take whole model from step 2, give increasingly bigger images to the model\n efn7 640x640 priv.LB:0.35389, pub.LB:0.39121\n efn7 736x736 priv.LB:0.36364, pub.LB:0.40174\n4.Take whole model from step 3, set twice loss weight for GLD v2 clean samples\n efn7 640x640 priv.LB:0.35932, pub.LB:0.39881\n efn7 736x736 priv.LB:0.36569, pub.LB:0.40215\n\n# Ensemble\n1.736x736 efn7+efn6+efn5+efn5 weighted concat\n(all train step3, weight : efn7=1.0, efn6=0.8, efn5=0.5)\npriv.LB:0.38366, pub.LB: 0.41986\n2.Same Config, with train step 4 for efn7\npriv.LB:0.38677, pub.LB: 0.42328\n\n# \nIf you have any questions, feel free to ask.\nThank you. \n", "979107": "Congratulations! Can't wait for your ArXiv paper 😊\n\nDid I understood correctly? You **trained only on Colab TPUs?** 😲", "989187": "This Solution is Absolutely Amazing @keetar Thanks a lot for sharing 👍✔️💯", "978720": "Congratulations! Can I ask you a question ,How you concat efn7+efn6+efn5+efn5 ?\nafter GAP\nI use : output=tf.concat([efnb6_224x224_model_0.output,efnb6_256x256_model_1.output],axis=-1)\nOutput dimension：（2304x2,)\nThe submitted result shows：Notebook Exceeded Allowed Compute", "979542": "Congrats and thank you for sharing. it's a very great solution!\nHow did you store a big dataset for Colab TPU? In this competition dataset is very big(1TB). It's difficult to handle.", "978974": "Huge props for achieving this awesome result with just Colab TPUs, proving that this competition was pretty accessible after all.\n\nI was a bit depressed after reading about \"21 P100 GPUs\" in original DELG paper or \"32 GPU cluster\" in 4th place solution but it turns out you could win by a big margin without those crazy resources after all...", "995545": "Congratulations and thank you for sharing your solution 🙏", "988109": "Congratulations! \nI have a question:\n \n> s=determined by adacos\n\nHave you used static or dynamic **s** parameter from adacos? If static, have you calculated it using sqrt(2)*(log(C)-1) formula?\nAnd why did you use **m** = 0, but not some very small value for example 1e-4? ", "987788": "Awesome! I really learned a lot.\nCongratulations!! I ", "984280": "Congratulation @keetar! And thanks for sharing.\nDid you used Dropout or Batchnormalization? Did you tried smaller image sizes like 256x256 or 384x384 or EfficientNet B0-B6 ?", "984258": "Congratulations!", "983598": "Congratulations!", "983317": "Great work thanks!", "983126": "Thank you!", "982700": "Congratulations", "981032": "Congratulations and thanks a lot for sharing so much detailed information👍.", "981014": "@keetar `Embedding Dimension : 512 for every model` are you talking about `Dense(512)` or you use `Embedding Layer`", "980578": "Congrats", "980354": "Congrats! Could you please share your training pipeline. I'm new to TPU and want to learn how it works :)", "979749": "Congrats! @keetar I have a question - did you only use one type of augmentation (left-right flip) for all the training steps?", "979279": "Hi @keetar can you please direct me to some articles or anything that i can find useful to learn more about image retrieval. Believe me, I tried many different things github repos, articles, previous year solution, videos but all of them was either a failure or not complete enough to give me a proper understanding. I have fair idea of the Image Retrieval problem, but still don't have a clue as to what is the use of index images. I read the discussions and one of them host explained but then I was confused why there are no index images for training then. I maybe totally wrong with my understanding.\nI would be extremely thankful to you if you could list down some resources for helping me get started. Please if you can then do release your code. Thank and congratulations for your victory. 👍", "979273": "Congratulations on first!", "979223": "Congrats and thanks for sharing! ", "979159": "Thanks for sharing. I am interested in the arXiv upload. May I ask, is this generally all done in PyTorch?", "979136": "Congrats man, good job", "979039": "@keetar, congratulations for winning. Just to know , how much time needed to train the model ?", "978984": "Congrats!!! well deserved, any preprocessing or postprocessing techniques used to extract the local features?", "978894": "Congratulations! Well deserved win! :)", "978847": "Thanks for sharing great work man!!! ", "978729": "Thanks for sharing your strategy and congratulations on achieving the first place with a great margin. 👍 \n\n", "978687": "Congratulations! May I know were you using TPU for this? What learning rate / learning rate schedule u used in all the steps?", "978653": "Hi Keetar, Congratulation on achieving 1st place with a huge margin.\n\nI used a similar structure and setup as your first step. But with limited time and TPU quota, I only finished training 4 epochs of EFN-B6 and and 2 epochs of EFN-B7. They both only obtained about .27 in pub.LB.\n\nHow many epochs of training set have you fitted to achieve each milestone?\n\nThanks.", "978604": "Congrats on achieving such an insanely high score!\n\nWhat was the hardware config/training times for each step?", "985649": "Thanks for the extremely detailed write-up 🙏🙏🙏. It feels like a tutorial \"here is how to pwn this competition\", not a formal paper \"hey look how awesome I am but I won't give you details\" that we usually see.\n\nI can see a a bloodbath coming in Recognition challenge, where everyone will copy this strategy for global feature network :)) \n\nBtw turns out that a year ago we were colleagues (but in different countries) xD", "984393": "Congratulations Master :)", "980517": "Good Job on first @keetar ", "979855": "I congratulate @keetar for sharing the details and for his excellent achievement. I do not have any doubt, a lot of hard work and discipline has gone in to achieve that score.\n\nHowever, I have a small point to make to the organizers. Will it not be better to make a 'research' competiton at least 3-4 months long? I say this, so that competitiors can come up with some new techniques, that can make improvements over the benchmark scores, without using traditional methods used for better scores. This is in no way to criticize the traditional methods. But to encourage a practice that can make this community better , in terms of paper reading, contemporary technique implementation and perhaps, to innovate a new technique. \n\nOtherwise, I feel, kaggle is increasingly becoming a battleground for GPU/TPU based execution of pretrained models, using traditional methods. \n\n@andrefaraujo @tobiasweyand @maggie ", "979644": "Congrats on 1st place and thanks for sharing details solution @keetar ", "979602": "Congrats and thanks for sharing!\nHave you try to train with index set GLD-v2? \nDoes \"use GLD v2 total dataset to train model to classify 203094 classes\" mean using train.csv to train?\n", "988314": "Congratulation 👏", "993951": "Great solution!! Thanks for sharing.\n\nHowever, I am just wonder why you used Optimizer SGD, as I know training with Adam is much faster?", "1287619": "For ensemble How did you come up with the weights for the model?", "1243934": "Interesting training strategy. How did you select the image sizes? Is there a heuristic behind the numbers? ", "1044005": "Hi, how you can use cosine softmax in code of your submission, I want to know how to use?\nThanks.", "1004590": "Hi @keetar , did you use any trick like focal loss or label smoothing?", "1003621": "superb work\n", "1001382": "Hello, I want to know how to clean the dataset and how you make validation set? I'm just a newbie with python and this problem.\nOr do you have example source to do the cleaning data step?\nThank you. ", "1000263": "super", "999356": "Thanks for sharing your solution. As an image retrieval beginner, could you share a link to what GAP and DNN stand for? Thanks in advance and congratulations on your win!", "994429": "Congratulation.(And busy at denoting down the structure xDDD)", "990597": "Congratulations!\nThanks for the well-written solution.", "990236": "Great work. Congratulations!\n", "1439655": "", "1234736": "Great ! Thanks for sharing. Congrats :)", "991833": "Thanks for sharing.. great insights!" } }