Thanks to Kaggle and Berkeley SETI Research Center for this competition. Finding the aliens was very exciting. I tried many things to close the gap between cv and lb, but few succeeded, and as a result, my solution is very simple.
\nMy solution is a two-step process. The first stage is training with train data only, and the next stage is training with test data and pseudo labels. In the first stage, I use only vflip, cutout, and mixup to prevent the model from being confused by losing the signal due to unexpected augmentation. In addition, StochasticDepth and Dropout are strongly used to prevent overtraining. The mixup uses logical OR instead of alpha blending when mixing targets. This allows creating a model that responds strongly to weak signals. At this stage, we achieved PublicLB 0.800 by training efficientnetB5.
\nIn the second stage, we finetune the first stage model using pseudo-labels created by using the pre-trained model from the first stage. In this case, we refer to NoisyStudent and use Stochastic Depth and Dropout more strongly to prevent overfitting to the pseudo-label. By repeating this process several times, the PublicLB 0.813 is reached. Finally, I simply averaged the five submissions with the highest LB scores.
\nYou can see a sample of the code here.
\nhttps://www.kaggle.com/hirune924/2ndplace-solution
y = y + y[index] - (y * y[index])\nSometimes, the loss became nan or the validation auc became 0.5 and the training did not progress. In this case, I solved the problem by training only the head and encoder BNs for one epoch first.
\nThe code is almost the same as the first step, adding pseudo-labels and making Stochastic Depth and Dropout even stronger for training.
\nThanks to Kaggle and Berkeley SETI Research Center for this competition. Finding the aliens was very exciting. I tried many things to close the gap between cv and lb, but few succeeded, and as a result, my solution is very simple.
\nMy solution is a two-step process. The first stage is training with train data only, and the next stage is training with test data and pseudo labels. In the first stage, I use only vflip, cutout, and mixup to prevent the model from being confused by losing the signal due to unexpected augmentation. In addition, StochasticDepth and Dropout are strongly used to prevent overtraining. The mixup uses logical OR instead of alpha blending when mixing targets. This allows creating a model that responds strongly to weak signals. At this stage, we achieved PublicLB 0.800 by training efficientnetB5.
\nIn the second stage, we finetune the first stage model using pseudo-labels created by using the pre-trained model from the first stage. In this case, we refer to NoisyStudent and use Stochastic Depth and Dropout more strongly to prevent overfitting to the pseudo-label. By repeating this process several times, the PublicLB 0.813 is reached. Finally, I simply averaged the five submissions with the highest LB scores.
\nYou can see a sample of the code here.
\nhttps://www.kaggle.com/hirune924/2ndplace-solution
y = y + y[index] - (y * y[index])\nSometimes, the loss became nan or the validation auc became 0.5 and the training did not progress. In this case, I solved the problem by training only the head and encoder BNs for one epoch first.
\nThe code is almost the same as the first step, adding pseudo-labels and making Stochastic Depth and Dropout even stronger for training.
\nCongrats, very impressive results. When we saw you climbing one point every day, we were quite convinced that you are pseudo tagging :)
\nThe important part here seems to do heavy regularization when incorporating the pseudo tags, so that the model does not overfit too heavily on the pseudo tags coming from images of a different distribution, we shortly discuss this also in our solution post. I believe that the best way to incorporate pseudo tags, in theory, would be to only add target=1 labels, because, at least from CV, the models make less mistakes when they predict and find the signal, compared to when they don't. However, this does not work here, because then the model would just learn that target=1 comes always from test (due to distribution shift). There might be some clever way to do this, by studying more the research on accounting for distribution shift (e.g., additional losses that try to prevent overfitting on the distribution). Unfortunately, didnt have time to look into it due to our other findings.
\nAgain: very impressive job and congrats on solo gold!
", "rawMarkdown": "Congrats, very impressive results. When we saw you climbing one point every day, we were quite convinced that you are pseudo tagging :)\n\nThe important part here seems to do heavy regularization when incorporating the pseudo tags, so that the model does not overfit too heavily on the pseudo tags coming from images of a different distribution, we shortly discuss this also in our solution post. I believe that the best way to incorporate pseudo tags, in theory, would be to only add target=1 labels, because, at least from CV, the models make less mistakes when they predict and find the signal, compared to when they don't. However, this does not work here, because then the model would just learn that target=1 comes always from test (due to distribution shift). There might be some clever way to do this, by studying more the research on accounting for distribution shift (e.g., additional losses that try to prevent overfitting on the distribution). Unfortunately, didnt have time to look into it due to our other findings.\n\nAgain: very impressive job and congrats on solo gold!", "votes": 11, "replies": [ { "id": 1480741, "postDate": "2021-08-19T06:48:19.727Z", "content": "Your team's solution is also a great job. Congratulations on the 1st place.
\nActually, I thought that using all the pseudo-labels instead of only target=1 is dangerous because it may overfit to the pseudo-labels.
\nTherefore, we tried to use only those pseudo-labels that showed extreme target scores by threshold, but the Noisy Student style method using all pseudo-labels showed better and more stable results.
\nI would have liked to increase the model size according to the paper, but there was not enough time.
Congratulations! your boost on 2nd stage is very inpressive!
\n\n\ny = y + y[index] - (y * y[index])
\n
This is completely the same as our mixup: torch.stack([y, y[index]], 0).max(0).values
But we didn't adopted PL or NS…
", "rawMarkdown": "Congratulations! your boost on 2nd stage is very inpressive!\n\n> y = y + y[index] - (y * y[index])\n\nThis is completely the same as our mixup: `torch.stack([y, y[index]], 0).max(0).values`\n\nBut we didn't adopted PL or NS...", "votes": 5, "replies": [ { "id": 1480737, "postDate": "2021-08-19T06:45:27.797Z", "content": "Hi @haqishen my intuition was that if we use mixup it will only distort the signal and will not be good . I am still very weak in CV can you please tell us the thought process(intuition) behind using mixup and the trick used above
", "rawMarkdown": "Hi @haqishen my intuition was that if we use mixup it will only distort the signal and will not be good . I am still very weak in CV can you please tell us the thought process(intuition) behind using mixup and the trick used above" }, { "id": 1481066, "postDate": "2021-08-19T09:42:49.290Z", "content": "Actually weaken the injected signal is what we want, because we need the model to gain the ability to detect weak signal as well. The modified mixup is base on this intuition.
", "rawMarkdown": "Actually weaken the injected signal is what we want, because we need the model to gain the ability to detect weak signal as well. The modified mixup is base on this intuition.", "votes": 4 }, { "id": 1481145, "postDate": "2021-08-19T10:42:21.307Z", "content": "Thank you
", "rawMarkdown": "Thank you " } ] }, { "id": 1480397, "postDate": "2021-08-19T02:26:27.783Z", "content": "Congrats! Your solution looks quite similar to ours, but you gain more from pseudo labeling test. I wonder why. Interestingly I used a very similar trick for target in mixup:
\n target = alpha * target + (1 - alpha) * target[perm]\n target = torch.clamp(2*target, 0, 1)\nI'm honored to have arrived at the same trick as you !
", "rawMarkdown": "I'm honored to have arrived at the same trick as you !", "votes": 2 }, { "id": 1481175, "postDate": "2021-08-19T11:02:06.487Z", "content": "I am the one who is honored given you did much better than us!
", "rawMarkdown": "I am the one who is honored given you did much better than us!", "votes": 1 }, { "id": 1482292, "postDate": "2021-08-20T02:23:21.267Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1482342, "postDate": "2021-08-20T03:26:46.580Z", "content": "let say alpha is 0.5 i.e. the new image is the mean of the two mixed images. If one of them is of target 1 and the other 0, the mean target is 0.5. By multiplying by 2 it becomes 1=, which is what we want here.
", "rawMarkdown": "let say alpha is 0.5 i.e. the new image is the mean of the two mixed images. If one of them is of target 1 and the other 0, the mean target is 0.5. By multiplying by 2 it becomes 1=, which is what we want here.", "votes": 1 }, { "id": 1482385, "postDate": "2021-08-20T04:00:37.020Z", "content": "Hi @cpmpml , regarding how to mix the target value during Mixup, sometimes people use standard(weighted) way like original paper, sometimes use maximum(logical OR) way like you did, I'm wondering what is the benefit for each way? The latter way is better for datasets which have very few positive samples, or target singal is very weak like this competition?
", "rawMarkdown": "Hi @cpmpml , regarding how to mix the target value during Mixup, sometimes people use standard(weighted) way like original [paper](https://arxiv.org/pdf/1710.09412v2.pdf), sometimes use maximum(logical OR) way like you did, I'm wondering what is the benefit for each way? The latter way is better for datasets which have very few positive samples, or target singal is very weak like this competition?" }, { "id": 1483260, "postDate": "2021-08-20T14:36:55.963Z", "content": "Here we look for messages. A message stays in the mixed image, and the hope is that this way model will learn to recognized weaker signals (weaker because they are blended with another image).
", "rawMarkdown": "Here we look for messages. A message stays in the mixed image, and the hope is that this way model will learn to recognized weaker signals (weaker because they are blended with another image).\n\n", "votes": 1 } ] }, { "id": 1480373, "postDate": "2021-08-19T02:04:17.630Z", "content": "Thanks for sharing!! Congratulations on the solo gold medal!!
\nI also tried arcface(m=0.5) to cope with unseen cls, but it failed. it takes 2x times than normal training and score wasn't good(LB=0.776,B0, size=7687683 )
I tried L2 constrained softmax Loss, but the max I could get was about lb0.73.
", "rawMarkdown": "I tried L2 constrained softmax Loss, but the max I could get was about lb0.73.", "votes": 1 } ] }, { "id": 1495846, "postDate": "2021-08-29T20:40:49.727Z", "content": "Great work! Congrats
", "rawMarkdown": "Great work! Congrats", "votes": 1 }, { "id": 1480504, "postDate": "2021-08-19T03:56:00.087Z", "content": "Congratulations for the solo gold @hirune924 🎉
\n\n\nthe loss became nan or the validation auc became 0.5 and the training did not progress. In this case, I solved the problem by training only the head and encoder BNs for one epoch first.
\n
I also observed sometimes my model does not learn anything as validation AUC is alway ~0.5, especially when I add more augmentations like ShiftScaleRotate or RandomResizedCrop, finally I found switch to OneCycleLR scheduler would help a bit, I suspect it`s due to it will start training with a small LR(1e-6 eg.) to find corrent direction to converge. May I know what is your theory about this \"always 0.5 CV\" problem? And why training with only head and encoder BNs in 1st epoch would help? Thanks.
", "rawMarkdown": "Congratulations for the solo gold @hirune924 🎉\n\n> the loss became nan or the validation auc became 0.5 and the training did not progress. In this case, I solved the problem by training only the head and encoder BNs for one epoch first.\n\nI also observed sometimes my model does not learn anything as validation AUC is alway ~0.5, especially when I add more augmentations like ShiftScaleRotate or RandomResizedCrop, finally I found switch to OneCycleLR scheduler would help a bit, I suspect it`s due to it will start training with a small LR(1e-6 eg.) to find corrent direction to converge. May I know what is your theory about this \"always 0.5 CV\" problem? And why training with only head and encoder BNs in 1st epoch would help? Thanks.", "votes": 1, "replies": [ { "id": 1480528, "postDate": "2021-08-19T04:16:26.527Z", "content": "Typically, we use imagenet pre-trained models for the encoder and untrained Linear for the classifier. In the early stage of training, the encoder parameters are corrupted by the meaningless gradient through the untrained Linear, which makes the training unstable. To prevent this, I trained only the classifier first.
", "rawMarkdown": "Typically, we use imagenet pre-trained models for the encoder and untrained Linear for the classifier. In the early stage of training, the encoder parameters are corrupted by the meaningless gradient through the untrained Linear, which makes the training unstable. To prevent this, I trained only the classifier first.", "votes": 7 } ] }, { "id": 1483145, "postDate": "2021-08-20T13:21:06.960Z", "content": "Congratulations great job! I think your mixup, pseudo, and strong regularization was key!
\nI've never used StochasticDepth before. Is it deployed with timm parameter drop_path_rate? Can it be applied to all timm models?
timm.create_model( drop_path_rate=RATE)\nYes, that's what I'm using. Apparently, it does not work for all models, but it does work for many models that are structured like ResNet.
", "rawMarkdown": "Yes, that's what I'm using. Apparently, it does not work for all models, but it does work for many models that are structured like ResNet.", "votes": 1 } ] }, { "id": 1486638, "postDate": "2021-08-23T05:48:55.310Z", "content": "Congratulations! It seems like a simple probability operation can help improve prediction scores significantly when properly applied.
", "rawMarkdown": "Congratulations! It seems like a simple probability operation can help improve prediction scores significantly when properly applied. " }, { "id": 1481719, "postDate": "2021-08-19T16:36:32.273Z", "content": "Congrats and great summary! Thanks for sharing @hirune924
", "rawMarkdown": "Congrats and great summary! Thanks for sharing @hirune924 " }, { "id": 1480970, "postDate": "2021-08-19T08:51:30.243Z", "content": "Congratulations on coming in second! And I want to a question about pseudo label.
\nIs the data used in your second stage are training data and pseudo label? or only pseudo label?
", "rawMarkdown": "Congratulations on coming in second! And I want to a question about pseudo label.\n\nIs the data used in your second stage are training data and pseudo label? or only pseudo label?", "replies": [ { "id": 1480997, "postDate": "2021-08-19T09:05:48.603Z", "content": "Use both train and test.
", "rawMarkdown": "Use both train and test." }, { "id": 1481090, "postDate": "2021-08-19T09:56:21.820Z", "content": "Thanks for your answer. We do the same thing as your stage2 but we don't get much improve. It seems like regularizations are really important.
", "rawMarkdown": "Thanks for your answer. We do the same thing as your stage2 but we don't get much improve. It seems like regularizations are really important." } ] }, { "id": 1480923, "postDate": "2021-08-19T08:20:11.807Z", "content": "Congrats and thanks for sharing your solution!
", "rawMarkdown": "Congrats and thanks for sharing your solution!" }, { "id": 1480761, "postDate": "2021-08-19T07:00:54.287Z", "content": "Hi @hirune924 can you tell us the model names that you have used in the final score?
", "rawMarkdown": "Hi @hirune924 can you tell us the model names that you have used in the final score?", "replies": [ { "id": 1480785, "postDate": "2021-08-19T07:13:02.237Z", "content": "Finally, the following models were blended. v2 was introduced to increase diversity.
\n| model | \npublic LB | \nprivate LB | \n
|---|---|---|
| tf_efficientnet_b5_ns | \n0.81341 | \n0.81094 | \n
| tf_efficientnet_b5_ns | \n0.81384 | \n0.81196 | \n
| tf_efficientnet_b5_ns | \n0.81203 | \n0.81209 | \n
| tf_efficientnetv2_m_in21ft1k | \n0.81169 | \n0.80978 | \n
| tf_efficientnetv2_m_in21ft1k | \n0.80626 | \n0.80486 | \n
Thanks for sharing 😌
", "rawMarkdown": "Thanks for sharing 😌" } ] }, { "id": 1480462, "postDate": "2021-08-19T03:21:11.483Z", "content": "Congratulations! That is a surprisingly simple solution. Thank you for sharing.
\nI also used \"logical OR\" mixup, which is effective for CV and LB.
\nI didn't try pseudo labeling, I think that is a key.
BTW, how did you use the \"logical OR\" mixup in training by pseudo label? by hard pseudo labeling?
", "rawMarkdown": "Congratulations! That is a surprisingly simple solution. Thank you for sharing.\n\nI also used \"logical OR\" mixup, which is effective for CV and LB.\nI didn't try pseudo labeling, I think that is a key.\n\nBTW, how did you use the \"logical OR\" mixup in training by pseudo label? by hard pseudo labeling?", "replies": [ { "id": 1480500, "postDate": "2021-08-19T03:52:07.677Z", "content": "My method differs from strict logical OR, and works on soft targets because I used the following continuous method that works similarly.
\ny = y1 + y2 - (y1 * y2)
Thanks! I'll try it in late submission.
", "rawMarkdown": "Thanks! I'll try it in late submission." } ] }, { "id": 1480456, "postDate": "2021-08-19T03:18:22.497Z", "content": "Congratulations & thanks for sharing the solution.
\nI also tried logical-OR for the mixup target, but it didn't improve my CV score I wonder why.
\ny = torch.clamp(y_a + y_b, min=0, max=1)\nCan you share us the 1st stage only score?
\nIn my experiments, I can't make LB score higher than 0.765, so I'm curious if the pseudo label is the key in the solution.
I achieve Public LB 0.80 at the first level alone. Also, the larger the model, the higher the score, but I reached the limit with b5!
", "rawMarkdown": "I achieve Public LB 0.80 at the first level alone. Also, the larger the model, the higher the score, but I reached the limit with b5!", "votes": 2 }, { "id": 1480554, "postDate": "2021-08-19T04:42:40.680Z", "content": "So the model size are also important. I was stopping efficientnet-b2 due to computational resource limit. Thank you.
", "rawMarkdown": "So the model size are also important. I was stopping efficientnet-b2 due to computational resource limit. Thank you." } ] }, { "id": 1480411, "postDate": "2021-08-19T02:43:24.133Z", "content": "Thanks for sharing your approach!
\nI am curious - how did you trained the b5 efficientnet on the 512x512 image size without overloading the GPU? Thanks for any help :)
", "rawMarkdown": "Thanks for sharing your approach!\n\nI am curious - how did you trained the b5 efficientnet on the 512x512 image size without overloading the GPU? Thanks for any help :)", "replies": [ { "id": 1480425, "postDate": "2021-08-19T02:54:25.077Z", "content": "V100x1 GPU is used to train b5, and mixed precision training is used to save GPU RAM.
", "rawMarkdown": "V100x1 GPU is used to train b5, and mixed precision training is used to save GPU RAM.", "votes": 2 } ] }, { "id": 1480366, "postDate": "2021-08-19T01:53:15.340Z", "content": "@hirune924 Congrats!🎉
\nThanks for sharing your approach. Interesting!
", "rawMarkdown": "@hirune924 Congrats!🎉\n\nThanks for sharing your approach. Interesting!" } ], "comments": [ { "id": 1480685, "author_name": "Psi", "author_url": "", "post_date": "2021-08-19T06:17:21.457000", "content": "Congrats, very impressive results. When we saw you climbing one point every day, we were quite convinced that you are pseudo tagging :)
\nThe important part here seems to do heavy regularization when incorporating the pseudo tags, so that the model does not overfit too heavily on the pseudo tags coming from images of a different distribution, we shortly discuss this also in our solution post. I believe that the best way to incorporate pseudo tags, in theory, would be to only add target=1 labels, because, at least from CV, the models make less mistakes when they predict and find the signal, compared to when they don't. However, this does not work here, because then the model would just learn that target=1 comes always from test (due to distribution shift). There might be some clever way to do this, by studying more the research on accounting for distribution shift (e.g., additional losses that try to prevent overfitting on the distribution). Unfortunately, didnt have time to look into it due to our other findings.
\nAgain: very impressive job and congrats on solo gold!
", "votes": 11, "replies": [ { "id": 1480741, "author_name": "hirune924", "author_url": "", "post_date": "2021-08-19T06:48:19.727000", "content": "Your team's solution is also a great job. Congratulations on the 1st place.
\nActually, I thought that using all the pseudo-labels instead of only target=1 is dangerous because it may overfit to the pseudo-labels.
\nTherefore, we tried to use only those pseudo-labels that showed extreme target scores by threshold, but the Noisy Student style method using all pseudo-labels showed better and more stable results.
\nI would have liked to increase the model size according to the paper, but there was not enough time.
Congratulations! your boost on 2nd stage is very inpressive!
\n\n\ny = y + y[index] - (y * y[index])
\n
This is completely the same as our mixup: torch.stack([y, y[index]], 0).max(0).values
But we didn't adopted PL or NS…
", "votes": 5, "replies": [ { "id": 1480737, "author_name": "Mr_KnowNothing", "author_url": "", "post_date": "2021-08-19T06:45:27.797000", "content": "Hi @haqishen my intuition was that if we use mixup it will only distort the signal and will not be good . I am still very weak in CV can you please tell us the thought process(intuition) behind using mixup and the trick used above
", "votes": 0, "replies": [] }, { "id": 1481066, "author_name": "Qishen Ha", "author_url": "", "post_date": "2021-08-19T09:42:49.290000", "content": "Actually weaken the injected signal is what we want, because we need the model to gain the ability to detect weak signal as well. The modified mixup is base on this intuition.
", "votes": 4, "replies": [] }, { "id": 1481145, "author_name": "Mr_KnowNothing", "author_url": "", "post_date": "2021-08-19T10:42:21.307000", "content": "Thank you
", "votes": 0, "replies": [] } ] }, { "id": 1480397, "author_name": "CPMP", "author_url": "", "post_date": "2021-08-19T02:26:27.783000", "content": "Congrats! Your solution looks quite similar to ours, but you gain more from pseudo labeling test. I wonder why. Interestingly I used a very similar trick for target in mixup:
\n target = alpha * target + (1 - alpha) * target[perm]\n target = torch.clamp(2*target, 0, 1)\nI'm honored to have arrived at the same trick as you !
", "votes": 2, "replies": [] }, { "id": 1481175, "author_name": "CPMP", "author_url": "", "post_date": "2021-08-19T11:02:06.487000", "content": "I am the one who is honored given you did much better than us!
", "votes": 1, "replies": [] }, { "id": 1482292, "author_name": "", "author_url": "", "post_date": "2021-08-20T02:23:21.267000", "content": "", "votes": 0, "replies": [] }, { "id": 1482342, "author_name": "CPMP", "author_url": "", "post_date": "2021-08-20T03:26:46.580000", "content": "let say alpha is 0.5 i.e. the new image is the mean of the two mixed images. If one of them is of target 1 and the other 0, the mean target is 0.5. By multiplying by 2 it becomes 1=, which is what we want here.
", "votes": 1, "replies": [] }, { "id": 1482385, "author_name": "Hao", "author_url": "", "post_date": "2021-08-20T04:00:37.020000", "content": "Hi @cpmpml , regarding how to mix the target value during Mixup, sometimes people use standard(weighted) way like original paper, sometimes use maximum(logical OR) way like you did, I'm wondering what is the benefit for each way? The latter way is better for datasets which have very few positive samples, or target singal is very weak like this competition?
", "votes": 0, "replies": [] }, { "id": 1483260, "author_name": "CPMP", "author_url": "", "post_date": "2021-08-20T14:36:55.963000", "content": "Here we look for messages. A message stays in the mixed image, and the hope is that this way model will learn to recognized weaker signals (weaker because they are blended with another image).
", "votes": 1, "replies": [] } ] }, { "id": 1480373, "author_name": "patriot", "author_url": "", "post_date": "2021-08-19T02:04:17.630000", "content": "Thanks for sharing!! Congratulations on the solo gold medal!!
\nI also tried arcface(m=0.5) to cope with unseen cls, but it failed. it takes 2x times than normal training and score wasn't good(LB=0.776,B0, size=7687683 )
I tried L2 constrained softmax Loss, but the max I could get was about lb0.73.
", "votes": 1, "replies": [] } ] }, { "id": 1495846, "author_name": "Fuco", "author_url": "", "post_date": "2021-08-29T20:40:49.727000", "content": "Great work! Congrats
", "votes": 1, "replies": [] }, { "id": 1480504, "author_name": "Hao", "author_url": "", "post_date": "2021-08-19T03:56:00.087000", "content": "Congratulations for the solo gold @hirune924 🎉
\n\n\nthe loss became nan or the validation auc became 0.5 and the training did not progress. In this case, I solved the problem by training only the head and encoder BNs for one epoch first.
\n
I also observed sometimes my model does not learn anything as validation AUC is alway ~0.5, especially when I add more augmentations like ShiftScaleRotate or RandomResizedCrop, finally I found switch to OneCycleLR scheduler would help a bit, I suspect it`s due to it will start training with a small LR(1e-6 eg.) to find corrent direction to converge. May I know what is your theory about this \"always 0.5 CV\" problem? And why training with only head and encoder BNs in 1st epoch would help? Thanks.
", "votes": 1, "replies": [ { "id": 1480528, "author_name": "hirune924", "author_url": "", "post_date": "2021-08-19T04:16:26.527000", "content": "Typically, we use imagenet pre-trained models for the encoder and untrained Linear for the classifier. In the early stage of training, the encoder parameters are corrupted by the meaningless gradient through the untrained Linear, which makes the training unstable. To prevent this, I trained only the classifier first.
", "votes": 7, "replies": [] } ] }, { "id": 1483145, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-08-20T13:21:06.960000", "content": "Congratulations great job! I think your mixup, pseudo, and strong regularization was key!
\nI've never used StochasticDepth before. Is it deployed with timm parameter drop_path_rate? Can it be applied to all timm models?
timm.create_model( drop_path_rate=RATE)\nYes, that's what I'm using. Apparently, it does not work for all models, but it does work for many models that are structured like ResNet.
", "votes": 1, "replies": [] } ] }, { "id": 1486638, "author_name": "Dac-Thanh Van", "author_url": "", "post_date": "2021-08-23T05:48:55.310000", "content": "Congratulations! It seems like a simple probability operation can help improve prediction scores significantly when properly applied.
", "votes": 0, "replies": [] }, { "id": 1481719, "author_name": "Kalilur Rahman", "author_url": "", "post_date": "2021-08-19T16:36:32.273000", "content": "Congrats and great summary! Thanks for sharing @hirune924
", "votes": 0, "replies": [] }, { "id": 1480970, "author_name": "Matthew Wu", "author_url": "", "post_date": "2021-08-19T08:51:30.243000", "content": "Congratulations on coming in second! And I want to a question about pseudo label.
\nIs the data used in your second stage are training data and pseudo label? or only pseudo label?
", "votes": 0, "replies": [ { "id": 1480997, "author_name": "hirune924", "author_url": "", "post_date": "2021-08-19T09:05:48.603000", "content": "Use both train and test.
", "votes": 0, "replies": [] }, { "id": 1481090, "author_name": "Matthew Wu", "author_url": "", "post_date": "2021-08-19T09:56:21.820000", "content": "Thanks for your answer. We do the same thing as your stage2 but we don't get much improve. It seems like regularizations are really important.
", "votes": 0, "replies": [] } ] }, { "id": 1480923, "author_name": "Cuong Thanh-Viet Nguyen", "author_url": "", "post_date": "2021-08-19T08:20:11.807000", "content": "Congrats and thanks for sharing your solution!
", "votes": 0, "replies": [] }, { "id": 1480761, "author_name": "Mr_KnowNothing", "author_url": "", "post_date": "2021-08-19T07:00:54.287000", "content": "Hi @hirune924 can you tell us the model names that you have used in the final score?
", "votes": 0, "replies": [ { "id": 1480785, "author_name": "hirune924", "author_url": "", "post_date": "2021-08-19T07:13:02.237000", "content": "Finally, the following models were blended. v2 was introduced to increase diversity.
\n| model | \npublic LB | \nprivate LB | \n
|---|---|---|
| tf_efficientnet_b5_ns | \n0.81341 | \n0.81094 | \n
| tf_efficientnet_b5_ns | \n0.81384 | \n0.81196 | \n
| tf_efficientnet_b5_ns | \n0.81203 | \n0.81209 | \n
| tf_efficientnetv2_m_in21ft1k | \n0.81169 | \n0.80978 | \n
| tf_efficientnetv2_m_in21ft1k | \n0.80626 | \n0.80486 | \n
Thanks for sharing 😌
", "votes": 0, "replies": [] } ] }, { "id": 1480462, "author_name": "Tawara", "author_url": "", "post_date": "2021-08-19T03:21:11.483000", "content": "Congratulations! That is a surprisingly simple solution. Thank you for sharing.
\nI also used \"logical OR\" mixup, which is effective for CV and LB.
\nI didn't try pseudo labeling, I think that is a key.
BTW, how did you use the \"logical OR\" mixup in training by pseudo label? by hard pseudo labeling?
", "votes": 0, "replies": [ { "id": 1480500, "author_name": "hirune924", "author_url": "", "post_date": "2021-08-19T03:52:07.677000", "content": "My method differs from strict logical OR, and works on soft targets because I used the following continuous method that works similarly.
\ny = y1 + y2 - (y1 * y2)
Thanks! I'll try it in late submission.
", "votes": 0, "replies": [] } ] }, { "id": 1480456, "author_name": "Bilzard", "author_url": "", "post_date": "2021-08-19T03:18:22.497000", "content": "Congratulations & thanks for sharing the solution.
\nI also tried logical-OR for the mixup target, but it didn't improve my CV score I wonder why.
\ny = torch.clamp(y_a + y_b, min=0, max=1)\nCan you share us the 1st stage only score?
\nIn my experiments, I can't make LB score higher than 0.765, so I'm curious if the pseudo label is the key in the solution.
I achieve Public LB 0.80 at the first level alone. Also, the larger the model, the higher the score, but I reached the limit with b5!
", "votes": 2, "replies": [] }, { "id": 1480554, "author_name": "Bilzard", "author_url": "", "post_date": "2021-08-19T04:42:40.680000", "content": "So the model size are also important. I was stopping efficientnet-b2 due to computational resource limit. Thank you.
", "votes": 0, "replies": [] } ] }, { "id": 1480411, "author_name": "Daniel Furman", "author_url": "", "post_date": "2021-08-19T02:43:24.133000", "content": "Thanks for sharing your approach!
\nI am curious - how did you trained the b5 efficientnet on the 512x512 image size without overloading the GPU? Thanks for any help :)
", "votes": 0, "replies": [ { "id": 1480425, "author_name": "hirune924", "author_url": "", "post_date": "2021-08-19T02:54:25.077000", "content": "V100x1 GPU is used to train b5, and mixed precision training is used to save GPU RAM.
", "votes": 2, "replies": [] } ] }, { "id": 1480366, "author_name": "BaAis", "author_url": "", "post_date": "2021-08-19T01:53:15.340000", "content": "@hirune924 Congrats!🎉
\nThanks for sharing your approach. Interesting!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1480362": "Thanks to Kaggle and Berkeley SETI Research Center for this competition. Finding the aliens was very exciting. I tried many things to close the gap between cv and lb, but few succeeded, and as a result, my solution is very simple.\n\n## Abstract\nMy solution is a two-step process. The first stage is training with train data only, and the next stage is training with test data and pseudo labels. In the first stage, I use only vflip, cutout, and mixup to prevent the model from being confused by losing the signal due to unexpected augmentation. In addition, StochasticDepth and Dropout are strongly used to prevent overtraining. The mixup uses logical OR instead of alpha blending when mixing targets. This allows creating a model that responds strongly to weak signals. At this stage, we achieved PublicLB 0.800 by training efficientnetB5.\n\nIn the second stage, we finetune the first stage model using pseudo-labels created by using the pre-trained model from the first stage. In this case, we refer to NoisyStudent and use Stochastic Depth and Dropout more strongly to prevent overfitting to the pseudo-label. By repeating this process several times, the PublicLB 0.813 is reached. Finally, I simply averaged the five submissions with the highest LB scores.\n\n## 1st Stage Details\nYou can see a sample of the code here.\nhttps://www.kaggle.com/hirune924/2ndplace-solution\n* The input images are merged in the time direction and resized to a size of 512x512.\n* The mixup target can be mixed by using the following expression to express a logical OR, which also supports soft targets when using pseudo labels.\n```\ny = y + y[index] - (y * y[index])\n```\n* TTA with vflip is used for inference\n\nSometimes, the loss became nan or the validation auc became 0.5 and the training did not progress. In this case, I solved the problem by training only the head and encoder BNs for one epoch first.\n\n## 2nd Stage Details\nThe code is almost the same as the first step, adding pseudo-labels and making Stochastic Depth and Dropout even stronger for training.\n\n## Didn't work\n* Unsupervised Domain Adaptation by finetune only BN\n* Add artificial signal, chi2noise, referring to setigen.\n* input stem stride=1\n* using old data\n* Anomaly detection based metric learning\n* Semi Supervised Learning\n* many other things...\n\n", "1480685": "Congrats, very impressive results. When we saw you climbing one point every day, we were quite convinced that you are pseudo tagging :)\n\nThe important part here seems to do heavy regularization when incorporating the pseudo tags, so that the model does not overfit too heavily on the pseudo tags coming from images of a different distribution, we shortly discuss this also in our solution post. I believe that the best way to incorporate pseudo tags, in theory, would be to only add target=1 labels, because, at least from CV, the models make less mistakes when they predict and find the signal, compared to when they don't. However, this does not work here, because then the model would just learn that target=1 comes always from test (due to distribution shift). There might be some clever way to do this, by studying more the research on accounting for distribution shift (e.g., additional losses that try to prevent overfitting on the distribution). Unfortunately, didnt have time to look into it due to our other findings.\n\nAgain: very impressive job and congrats on solo gold!", "1480428": "Congratulations! your boost on 2nd stage is very inpressive!\n\n> y = y + y[index] - (y * y[index])\n\nThis is completely the same as our mixup: `torch.stack([y, y[index]], 0).max(0).values`\n\nBut we didn't adopted PL or NS...", "1480397": "Congrats! Your solution looks quite similar to ours, but you gain more from pseudo labeling test. I wonder why. Interestingly I used a very similar trick for target in mixup:\n\n target = alpha * target + (1 - alpha) * target[perm]\n target = torch.clamp(2*target, 0, 1)\n", "1480373": "Thanks for sharing!! Congratulations on the solo gold medal!!\nI also tried arcface(m=0.5) to cope with unseen cls, but it failed. it takes 2x times than normal training and score wasn't good(LB=0.776,B0, size=768*768*3 )", "1495846": "Great work! Congrats", "1480504": "Congratulations for the solo gold @hirune924 🎉\n\n> the loss became nan or the validation auc became 0.5 and the training did not progress. In this case, I solved the problem by training only the head and encoder BNs for one epoch first.\n\nI also observed sometimes my model does not learn anything as validation AUC is alway ~0.5, especially when I add more augmentations like ShiftScaleRotate or RandomResizedCrop, finally I found switch to OneCycleLR scheduler would help a bit, I suspect it`s due to it will start training with a small LR(1e-6 eg.) to find corrent direction to converge. May I know what is your theory about this \"always 0.5 CV\" problem? And why training with only head and encoder BNs in 1st epoch would help? Thanks.", "1483145": "Congratulations great job! I think your mixup, pseudo, and strong regularization was key!\n\nI've never used `StochasticDepth` before. Is it deployed with timm parameter `drop_path_rate`? Can it be applied to all timm models?\n\n timm.create_model( drop_path_rate=RATE)", "1486638": "Congratulations! It seems like a simple probability operation can help improve prediction scores significantly when properly applied. ", "1481719": "Congrats and great summary! Thanks for sharing @hirune924 ", "1480970": "Congratulations on coming in second! And I want to a question about pseudo label.\n\nIs the data used in your second stage are training data and pseudo label? or only pseudo label?", "1480923": "Congrats and thanks for sharing your solution!", "1480761": "Hi @hirune924 can you tell us the model names that you have used in the final score?", "1480462": "Congratulations! That is a surprisingly simple solution. Thank you for sharing.\n\nI also used \"logical OR\" mixup, which is effective for CV and LB.\nI didn't try pseudo labeling, I think that is a key.\n\nBTW, how did you use the \"logical OR\" mixup in training by pseudo label? by hard pseudo labeling?", "1480456": "Congratulations & thanks for sharing the solution.\n\nI also tried logical-OR for the mixup target, but it didn't improve my CV score I wonder why.\n```\ny = torch.clamp(y_a + y_b, min=0, max=1)\n```\n\nCan you share us the 1st stage only score?\nIn my experiments, I can't make LB score higher than 0.765, so I'm curious if the pseudo label is the key in the solution.", "1480411": "Thanks for sharing your approach!\n\nI am curious - how did you trained the b5 efficientnet on the 512x512 image size without overloading the GPU? Thanks for any help :)", "1480366": "@hirune924 Congrats!🎉\n\nThanks for sharing your approach. Interesting!" } }