I see many heated comments after some novices climbed the LB.
\nLet's wait for two things before getting real angry:
\nPrivate LB. Final competition rank is not public LB. There will be a shakeup. Current LB ranks don't mean much.
Kaggle staff is pretty good at detecting private sharing. We will see some of these accounts disappear after competition end.
The one thing that is probably most useful is to share evidence as in this post: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175156
\nThis can help Kaggle staff clean the LB.
\nAnyway, it is easier for me to not be bothered as I am not fighting for a top spot here. Chris Deotte baseline was just too strong to beat starting from scratch for a computer vision newbie like me. And there is something with TF Effnet that isn't reproducible with Pytorch. I hope to investigate it further after competition end.
\nBut I can understand how people who were in gold zone or close to it feel when they say people who bought they way in. I sincerely hope they'll be removed.
", "messages": [ { "id": "974254", "postDate": "08/17/2020 20:42:37", "content": "I see many heated comments after some novices climbed the LB.
\nLet's wait for two things before getting real angry:
\nPrivate LB. Final competition rank is not public LB. There will be a shakeup. Current LB ranks don't mean much.
Kaggle staff is pretty good at detecting private sharing. We will see some of these accounts disappear after competition end.
The one thing that is probably most useful is to share evidence as in this post: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175156
\nThis can help Kaggle staff clean the LB.
\nAnyway, it is easier for me to not be bothered as I am not fighting for a top spot here. Chris Deotte baseline was just too strong to beat starting from scratch for a computer vision newbie like me. And there is something with TF Effnet that isn't reproducible with Pytorch. I hope to investigate it further after competition end.
\nBut I can understand how people who were in gold zone or close to it feel when they say people who bought they way in. I sincerely hope they'll be removed.
", "rawMarkdown": "I see many heated comments after some novices climbed the LB.\n\nLet's wait for two things before getting real angry:\n\n1. Private LB. Final competition rank is not public LB. There will be a shakeup. Current LB ranks don't mean much.\n\n2. Kaggle staff is pretty good at detecting private sharing. We will see some of these accounts disappear after competition end.\n\nThe one thing that is probably most useful is to share evidence as in this post: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175156\n\nThis can help Kaggle staff clean the LB.\n\nAnyway, it is easier for me to not be bothered as I am not fighting for a top spot here. Chris Deotte baseline was just too strong to beat starting from scratch for a computer vision newbie like me. And there is something with TF Effnet that isn't reproducible with Pytorch. I hope to investigate it further after competition end.\n\nBut I can understand how people who were in gold zone or close to it feel when they say people who bought they way in. I sincerely hope they'll be removed.", "votes": null }, { "id": "974261", "postDate": "08/17/2020 20:51:02", "content": "Agreed! It's important to stay calm and not witch hunt no matter how sure you are of the \"evidence\". Ever heard of the reddit Boston Bombing witch hunt? Just gather evidence and let Kaggle staff do their job.
", "rawMarkdown": "Agreed! It's important to stay calm and not witch hunt no matter how sure you are of the \"evidence\". Ever heard of the reddit Boston Bombing witch hunt? Just gather evidence and let Kaggle staff do their job.", "votes": null }, { "id": "974282", "postDate": "08/17/2020 21:10:29", "content": "I look forward to your analysis of why Pytorch effnets don't perform as well as TF ones. I suspect initialization or something in that vicinity.
", "rawMarkdown": "I look forward to your analysis of why Pytorch effnets don't perform as well as TF ones. I suspect initialization or something in that vicinity.", "votes": null }, { "id": "974332", "postDate": "08/17/2020 22:31:20", "content": "The thing is, these are not amateurs just trying to enjoy Limelight at the top of the Public LB. These are professionals, who have been doing this (fraud) for a long time. Look at user yikououbao for example. Top 1% in Ion Switching, Top 7% ALASKA2, top 9% in M5, and on and on. Tonight yikououbao will likely take home 3 medals……………her 9 competitions have shown that shakeup's do not affect her, or her teammates, or any of the other angtk.com paying customers. Sure some will not get the medal…….but they will next time, or the time after that. But it will still put plenty of space between others and medals. My only point is, there is a difference between a ton of people overfitting and doing naive stuff, and a shakeup correcting that, vs. professionals farming medals………the information they are using is coming from someone who is a Master/GM……and there is a lot of money riding on their ability to produce medals.
", "rawMarkdown": "The thing is, these are not amateurs just trying to enjoy Limelight at the top of the Public LB. These are professionals, who have been doing this (fraud) for a long time. Look at user yikououbao for example. Top 1% in Ion Switching, Top 7% ALASKA2, top 9% in M5, and on and on. Tonight yikououbao will likely take home 3 medals...............her 9 competitions have shown that shakeup's do not affect her, or her teammates, or any of the other angtk.com paying customers. Sure some will not get the medal.......but they will next time, or the time after that. But it will still put plenty of space between others and medals. My only point is, there is a difference between a ton of people overfitting and doing naive stuff, and a shakeup correcting that, vs. professionals farming medals.........the information they are using is coming from someone who is a Master/GM......and there is a lot of money riding on their ability to produce medals.", "votes": null }, { "id": "974363", "postDate": "08/17/2020 23:44:41", "content": "As you say, this is not new. It is why I prefer research competitions where you cannot just run someone else' code. But I need to learn about computer vision, hence this competition and the previous one I did. I do Kaggle to learn, not to brag about something I did not contribute to.
", "rawMarkdown": "As you say, this is not new. It is why I prefer research competitions where you cannot just run someone else' code. But I need to learn about computer vision, hence this competition and the previous one I did. I do Kaggle to learn, not to brag about something I did not contribute to.", "votes": null }, { "id": "974368", "postDate": "08/17/2020 23:48:33", "content": "Three hypothesis or facts:
\nBut, as you wrote, there may be other reasons, like weight initialization.
", "rawMarkdown": "Three hypothesis or facts:\n1. Data difference\n2. Fact: there are missing kernels in cudnn, which means that effnets don't use tensorcores on Volta GPU. This explains why TPU are much faster for these models, but it doe snot really explain model quality difference.\n3. Popular effnet pytorch implementations do not implement batch norm across multi gpu. I think this is the main reason for poorer performance.\n\nBut, as you wrote, there may be other reasons, like weight initialization.", "votes": null }, { "id": "974386", "postDate": "08/18/2020 00:02:57", "content": "I agree that SyncBatchNorm a la DDP/AMP would likely be a big improvement. In fact, most kernels aren't even DDP, but I wonder for non-AMP situations if there is much difference in Single thread, DataParallel or DDP.
", "rawMarkdown": "I agree that SyncBatchNorm a la DDP/AMP would likely be a big improvement. In fact, most kernels aren't even DDP, but I wonder for non-AMP situations if there is much difference in Single thread, DataParallel or DDP.", "votes": null }, { "id": "974392", "postDate": "08/18/2020 00:05:04", "content": "I am not sure to understand, what fraud are you talking about ?
\nEDIT : I have seen the \"Evidence regarding private sharing\" post, did not know that kind of stuff existed on kaggle
", "rawMarkdown": "I am not sure to understand, what fraud are you talking about ?\n\nEDIT : I have seen the \"Evidence regarding private sharing\" post, did not know that kind of stuff existed on kaggle", "votes": null }, { "id": "974396", "postDate": "08/18/2020 00:07:11", "content": "Only two teams from original top 50 stayed in private top 50. #Brutal
\nI almost wonder if some of those public kernels had private lb data poisoning. 🤔 But that wouldn't / shouldn't have affected to top brass.
", "rawMarkdown": "Only two teams from original top 50 stayed in private top 50. #Brutal\n\nI almost wonder if some of those public kernels had private lb data poisoning. 🤔 But that wouldn't / shouldn't have affected to top brass.", "votes": null }, { "id": "974425", "postDate": "08/18/2020 00:19:48", "content": "See, no need to get angry ;)
", "rawMarkdown": "See, no need to get angry ;)", "votes": null }, { "id": "974428", "postDate": "08/18/2020 00:20:30", "content": "Would weight initialization factor in when using pre-trained weights? I mean, sure for the classifier it would, but I wouldn't think the rest of the model. torchtoolbox has KaimingInitializer and XavierInitializer that were based on well known published papers to try.
", "rawMarkdown": "Would weight initialization factor in when using pre-trained weights? I mean, sure for the classifier it would, but I wouldn't think the rest of the model. [torchtoolbox](https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwjLyP-2vaPrAhWFRDABHenpDI0QFjAAegQIAhAB&url=https%3A%2F%2Fgithub.com%2FPistonY%2Ftorch-toolbox&usg=AOvVaw0WM5IEoEwwJErPEYxtS9YJl) has KaimingInitializer and XavierInitializer that were based on well known published papers to try.", "votes": null }, { "id": "974431", "postDate": "08/18/2020 00:21:33", "content": "Good point.
", "rawMarkdown": "Good point.", "votes": null }, { "id": "974881", "postDate": "08/18/2020 04:40:16", "content": "You are right. Kaggle is best in handling this.
", "rawMarkdown": "You are right. Kaggle is best in handling this.", "votes": null }, { "id": "975112", "postDate": "08/18/2020 06:46:25", "content": "pytorch effnets perform better on private LB ;)
", "rawMarkdown": "pytorch effnets perform better on private LB ;)", "votes": null }, { "id": "975123", "postDate": "08/18/2020 06:53:25", "content": "@christofhenkel what was your team's CV? Did you ensemble with any public notebooks or only use your own models?
", "rawMarkdown": "christofhenkel what was your team's CV? Did you ensemble with any public notebooks or only use your own models?", "votes": null }, { "id": "975136", "postDate": "08/18/2020 07:03:29", "content": "Best for us was an ensemble of effnets (pytorch + GPU )
\nCV 95.3 / Public LB 95.3 / Private LB 0.9441
\nwe also chose one submission where we blended with best public kernel as a gamble, but it performed poorly on private.
", "rawMarkdown": "Best for us was an ensemble of effnets (pytorch + GPU ) \n\nCV 95.3 / Public LB 95.3 / Private LB 0.9441 \n\nwe also chose one submission where we blended with best public kernel as a gamble, but it performed poorly on private.", "votes": null }, { "id": "975159", "postDate": "08/18/2020 07:14:45", "content": "There appears to be a strong correlation between CV and private LB. My CV was 0.9505 and private LB was 0.9420 (then a bit higher using public notebook CSVs) and your CV was 0.953 and private LB 0.9441. Great job Dieter and Psi
", "rawMarkdown": "There appears to be a strong correlation between CV and private LB. My CV was 0.9505 and private LB was 0.9420 (then a bit higher using public notebook CSVs) and your CV was 0.953 and private LB 0.9441. Great job Dieter and Psi", "votes": null }, { "id": "975160", "postDate": "08/18/2020 07:15:08", "content": "Just because they dropped still doesnt make it ok :)
", "rawMarkdown": "Just because they dropped still doesnt make it ok :)", "votes": null }, { "id": "975190", "postDate": "08/18/2020 07:32:23", "content": "Agreed, it needs to be stamped out and made clear that people shouldn't be paying for solutions! The fact that the people using these solutions dropped in some ways makes the scam even worse… Despite this, I think lots of people in the Gold/Silver/Bronze zone are still suspicious (clearly multiple accounts etc) but perhaps this isn't so different to any other Kaggle competition?
", "rawMarkdown": "Agreed, it needs to be stamped out and made clear that people shouldn't be paying for solutions! The fact that the people using these solutions dropped in some ways makes the scam even worse... Despite this, I think lots of people in the Gold/Silver/Bronze zone are still suspicious (clearly multiple accounts etc) but perhaps this isn't so different to any other Kaggle competition?", "votes": null }, { "id": "975333", "postDate": "08/18/2020 08:50:51", "content": "\n\nJust because they dropped still doesnt make it ok :)
\n
Of course. I hope they'll be removed and banned from Kaggle. But would people have gotten angry like that if cheaters public scores were not that high? I don't think so. I was reacting to this precisely: people were acting as if gold medals could be bought. They can't.
", "rawMarkdown": "> Just because they dropped still doesnt make it ok :)\n\nOf course. I hope they'll be removed and banned from Kaggle. But would people have gotten angry like that if cheaters public scores were not that high? I don't think so. I was reacting to this precisely: people were acting as if gold medals could be bought. They can't.", "votes": null }, { "id": "975339", "postDate": "08/18/2020 08:54:54", "content": "I wouldn't be surprised if they could. There looks like there is some evidence that people are paying to join teams of GMs/Masters (and then being a shadow member) in order to obtain gold/silver/bronze medals..
", "rawMarkdown": "I wouldn't be surprised if they could. There looks like there is some evidence that people are paying to join teams of GMs/Masters (and then being a shadow member) in order to obtain gold/silver/bronze medals..", "votes": null }, { "id": "975355", "postDate": "08/18/2020 08:59:39", "content": "That's not the same as what triggered these discussions.
\nI call what you describe the cuckoo strategy. Some Grandmasters are mastering it as well, they join late high ranking teams extremely often.
\nThe best known example a cuckoo has been banned definitely from Kagggle, but not because of it. He was banned because he cheated badly in a comp and the scam was discovered several months later.
\nI have been invited a couple of times to join a high ranked team although I wasn't active in a comp. I feel like an impostor in these cases, and tend to decline now.
", "rawMarkdown": "That's not the same as what triggered these discussions. \n\nI call what you describe the cuckoo strategy. Some Grandmasters are mastering it as well, they join late high ranking teams extremely often.\n\nThe best known example a cuckoo has been banned definitely from Kagggle, but not because of it. He was banned because he cheated badly in a comp and the scam was discovered several months later.\n\nI have been invited a couple of times to join a high ranked team although I wasn't active in a comp. I feel like an impostor in these cases, and tend to decline now.", "votes": null }, { "id": "975477", "postDate": "08/18/2020 10:05:54", "content": "We used TIMM implementation of Efficientnets which can be syncnorm'd in DistributedDataParallel (but not in DataParallel), and this works also in FP16:
\n if (not in_nb()) and (a.local_rank is not None):\n learn.model = nn.SyncBatchNorm.convert_sync_batchnorm(learn.model)\n learn.to_distributed(a.local_rank)\n else:\n learn.to_parallel()\nOne thing I am not sure is how/if you should scale LR based on number of GPUs (assuming BS scales w/ GPUS) and if so, if scaling should be different in DDP vs. DP (due to difference loss computation in both cases).
", "rawMarkdown": "We used TIMM implementation of Efficientnets which can be syncnorm'd in DistributedDataParallel (but not in DataParallel), and this works also in FP16:\n\n```\n if (not in_nb()) and (a.local_rank is not None):\n learn.model = nn.SyncBatchNorm.convert_sync_batchnorm(learn.model)\n learn.to_distributed(a.local_rank)\n else:\n learn.to_parallel()\n```\n\nOne thing I am not sure is how/if you should scale LR based on number of GPUs (assuming BS scales w/ GPUS) and if so, if scaling should be different in DDP vs. DP (due to difference loss computation in both cases).", "votes": null }, { "id": "975481", "postDate": "08/18/2020 10:08:20", "content": "Good description, I always assumed this was happening but thought it was OK for GMs to join teams and not do a lot of work as long as they were providing a good amount of guidance - which I'm sure most do.
", "rawMarkdown": "Good description, I always assumed this was happening but thought it was OK for GMs to join teams and not do a lot of work as long as they were providing a good amount of guidance - which I'm sure most do.", "votes": null } ], "comments": [ { "id": 974261, "author_name": "brachester", "author_url": "", "post_date": "08/17/2020 20:51:02", "content": "Agreed! It's important to stay calm and not witch hunt no matter how sure you are of the \"evidence\". Ever heard of the reddit Boston Bombing witch hunt? Just gather evidence and let Kaggle staff do their job.
", "votes": null, "replies": [] }, { "id": 974282, "author_name": "pheadrus", "author_url": "", "post_date": "08/17/2020 21:10:29", "content": "I look forward to your analysis of why Pytorch effnets don't perform as well as TF ones. I suspect initialization or something in that vicinity.
", "votes": null, "replies": [ { "id": 974368, "author_name": "cpmpml", "author_url": "", "post_date": "08/17/2020 23:48:33", "content": "Three hypothesis or facts:
\nBut, as you wrote, there may be other reasons, like weight initialization.
", "votes": null, "replies": [ { "id": 974386, "author_name": "brianfeeny", "author_url": "", "post_date": "08/18/2020 00:02:57", "content": "I agree that SyncBatchNorm a la DDP/AMP would likely be a big improvement. In fact, most kernels aren't even DDP, but I wonder for non-AMP situations if there is much difference in Single thread, DataParallel or DDP.
", "votes": null, "replies": [] }, { "id": 974428, "author_name": "brianfeeny", "author_url": "", "post_date": "08/18/2020 00:20:30", "content": "Would weight initialization factor in when using pre-trained weights? I mean, sure for the classifier it would, but I wouldn't think the rest of the model. torchtoolbox has KaimingInitializer and XavierInitializer that were based on well known published papers to try.
", "votes": null, "replies": [] }, { "id": 974431, "author_name": "cpmpml", "author_url": "", "post_date": "08/18/2020 00:21:33", "content": "Good point.
", "votes": null, "replies": [] } ] }, { "id": 975112, "author_name": "christofhenkel", "author_url": "", "post_date": "08/18/2020 06:46:25", "content": "pytorch effnets perform better on private LB ;)
", "votes": null, "replies": [] }, { "id": 975123, "author_name": "cdeotte", "author_url": "", "post_date": "08/18/2020 06:53:25", "content": "@christofhenkel what was your team's CV? Did you ensemble with any public notebooks or only use your own models?
", "votes": null, "replies": [] }, { "id": 975136, "author_name": "christofhenkel", "author_url": "", "post_date": "08/18/2020 07:03:29", "content": "Best for us was an ensemble of effnets (pytorch + GPU )
\nCV 95.3 / Public LB 95.3 / Private LB 0.9441
\nwe also chose one submission where we blended with best public kernel as a gamble, but it performed poorly on private.
", "votes": null, "replies": [] }, { "id": 975159, "author_name": "cdeotte", "author_url": "", "post_date": "08/18/2020 07:14:45", "content": "There appears to be a strong correlation between CV and private LB. My CV was 0.9505 and private LB was 0.9420 (then a bit higher using public notebook CSVs) and your CV was 0.953 and private LB 0.9441. Great job Dieter and Psi
", "votes": null, "replies": [] }, { "id": 975477, "author_name": "antorsae", "author_url": "", "post_date": "08/18/2020 10:05:54", "content": "We used TIMM implementation of Efficientnets which can be syncnorm'd in DistributedDataParallel (but not in DataParallel), and this works also in FP16:
\n if (not in_nb()) and (a.local_rank is not None):\n learn.model = nn.SyncBatchNorm.convert_sync_batchnorm(learn.model)\n learn.to_distributed(a.local_rank)\n else:\n learn.to_parallel()\nOne thing I am not sure is how/if you should scale LR based on number of GPUs (assuming BS scales w/ GPUS) and if so, if scaling should be different in DDP vs. DP (due to difference loss computation in both cases).
", "votes": null, "replies": [] } ] }, { "id": 974332, "author_name": "brianfeeny", "author_url": "", "post_date": "08/17/2020 22:31:20", "content": "The thing is, these are not amateurs just trying to enjoy Limelight at the top of the Public LB. These are professionals, who have been doing this (fraud) for a long time. Look at user yikououbao for example. Top 1% in Ion Switching, Top 7% ALASKA2, top 9% in M5, and on and on. Tonight yikououbao will likely take home 3 medals……………her 9 competitions have shown that shakeup's do not affect her, or her teammates, or any of the other angtk.com paying customers. Sure some will not get the medal…….but they will next time, or the time after that. But it will still put plenty of space between others and medals. My only point is, there is a difference between a ton of people overfitting and doing naive stuff, and a shakeup correcting that, vs. professionals farming medals………the information they are using is coming from someone who is a Master/GM……and there is a lot of money riding on their ability to produce medals.
", "votes": null, "replies": [ { "id": 974363, "author_name": "cpmpml", "author_url": "", "post_date": "08/17/2020 23:44:41", "content": "As you say, this is not new. It is why I prefer research competitions where you cannot just run someone else' code. But I need to learn about computer vision, hence this competition and the previous one I did. I do Kaggle to learn, not to brag about something I did not contribute to.
", "votes": null, "replies": [] }, { "id": 974392, "author_name": "ludovick", "author_url": "", "post_date": "08/18/2020 00:05:04", "content": "I am not sure to understand, what fraud are you talking about ?
\nEDIT : I have seen the \"Evidence regarding private sharing\" post, did not know that kind of stuff existed on kaggle
", "votes": null, "replies": [] } ] }, { "id": 974396, "author_name": "authman", "author_url": "", "post_date": "08/18/2020 00:07:11", "content": "Only two teams from original top 50 stayed in private top 50. #Brutal
\nI almost wonder if some of those public kernels had private lb data poisoning. 🤔 But that wouldn't / shouldn't have affected to top brass.
", "votes": null, "replies": [ { "id": 974425, "author_name": "cpmpml", "author_url": "", "post_date": "08/18/2020 00:19:48", "content": "See, no need to get angry ;)
", "votes": null, "replies": [] }, { "id": 975160, "author_name": "philippsinger", "author_url": "", "post_date": "08/18/2020 07:15:08", "content": "Just because they dropped still doesnt make it ok :)
", "votes": null, "replies": [] }, { "id": 975190, "author_name": "fchmiel", "author_url": "", "post_date": "08/18/2020 07:32:23", "content": "Agreed, it needs to be stamped out and made clear that people shouldn't be paying for solutions! The fact that the people using these solutions dropped in some ways makes the scam even worse… Despite this, I think lots of people in the Gold/Silver/Bronze zone are still suspicious (clearly multiple accounts etc) but perhaps this isn't so different to any other Kaggle competition?
", "votes": null, "replies": [] }, { "id": 975333, "author_name": "cpmpml", "author_url": "", "post_date": "08/18/2020 08:50:51", "content": "\n\nJust because they dropped still doesnt make it ok :)
\n
Of course. I hope they'll be removed and banned from Kaggle. But would people have gotten angry like that if cheaters public scores were not that high? I don't think so. I was reacting to this precisely: people were acting as if gold medals could be bought. They can't.
", "votes": null, "replies": [] }, { "id": 975339, "author_name": "fchmiel", "author_url": "", "post_date": "08/18/2020 08:54:54", "content": "I wouldn't be surprised if they could. There looks like there is some evidence that people are paying to join teams of GMs/Masters (and then being a shadow member) in order to obtain gold/silver/bronze medals..
", "votes": null, "replies": [] }, { "id": 975355, "author_name": "cpmpml", "author_url": "", "post_date": "08/18/2020 08:59:39", "content": "That's not the same as what triggered these discussions.
\nI call what you describe the cuckoo strategy. Some Grandmasters are mastering it as well, they join late high ranking teams extremely often.
\nThe best known example a cuckoo has been banned definitely from Kagggle, but not because of it. He was banned because he cheated badly in a comp and the scam was discovered several months later.
\nI have been invited a couple of times to join a high ranked team although I wasn't active in a comp. I feel like an impostor in these cases, and tend to decline now.
", "votes": null, "replies": [] }, { "id": 975481, "author_name": "fchmiel", "author_url": "", "post_date": "08/18/2020 10:08:20", "content": "Good description, I always assumed this was happening but thought it was OK for GMs to join teams and not do a lot of work as long as they were providing a good amount of guidance - which I'm sure most do.
", "votes": null, "replies": [] } ] }, { "id": 974881, "author_name": "raoofnaushad", "author_url": "", "post_date": "08/18/2020 04:40:16", "content": "You are right. Kaggle is best in handling this.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "974254": "I see many heated comments after some novices climbed the LB.\n\nLet's wait for two things before getting real angry:\n\n1. Private LB. Final competition rank is not public LB. There will be a shakeup. Current LB ranks don't mean much.\n\n2. Kaggle staff is pretty good at detecting private sharing. We will see some of these accounts disappear after competition end.\n\nThe one thing that is probably most useful is to share evidence as in this post: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175156\n\nThis can help Kaggle staff clean the LB.\n\nAnyway, it is easier for me to not be bothered as I am not fighting for a top spot here. Chris Deotte baseline was just too strong to beat starting from scratch for a computer vision newbie like me. And there is something with TF Effnet that isn't reproducible with Pytorch. I hope to investigate it further after competition end.\n\nBut I can understand how people who were in gold zone or close to it feel when they say people who bought they way in. I sincerely hope they'll be removed.", "974261": "Agreed! It's important to stay calm and not witch hunt no matter how sure you are of the \"evidence\". Ever heard of the reddit Boston Bombing witch hunt? Just gather evidence and let Kaggle staff do their job.", "974282": "I look forward to your analysis of why Pytorch effnets don't perform as well as TF ones. I suspect initialization or something in that vicinity.", "974332": "The thing is, these are not amateurs just trying to enjoy Limelight at the top of the Public LB. These are professionals, who have been doing this (fraud) for a long time. Look at user yikououbao for example. Top 1% in Ion Switching, Top 7% ALASKA2, top 9% in M5, and on and on. Tonight yikououbao will likely take home 3 medals...............her 9 competitions have shown that shakeup's do not affect her, or her teammates, or any of the other angtk.com paying customers. Sure some will not get the medal.......but they will next time, or the time after that. But it will still put plenty of space between others and medals. My only point is, there is a difference between a ton of people overfitting and doing naive stuff, and a shakeup correcting that, vs. professionals farming medals.........the information they are using is coming from someone who is a Master/GM......and there is a lot of money riding on their ability to produce medals.", "974363": "As you say, this is not new. It is why I prefer research competitions where you cannot just run someone else' code. But I need to learn about computer vision, hence this competition and the previous one I did. I do Kaggle to learn, not to brag about something I did not contribute to.", "974368": "Three hypothesis or facts:\n1. Data difference\n2. Fact: there are missing kernels in cudnn, which means that effnets don't use tensorcores on Volta GPU. This explains why TPU are much faster for these models, but it doe snot really explain model quality difference.\n3. Popular effnet pytorch implementations do not implement batch norm across multi gpu. I think this is the main reason for poorer performance.\n\nBut, as you wrote, there may be other reasons, like weight initialization.", "974386": "I agree that SyncBatchNorm a la DDP/AMP would likely be a big improvement. In fact, most kernels aren't even DDP, but I wonder for non-AMP situations if there is much difference in Single thread, DataParallel or DDP.", "974392": "I am not sure to understand, what fraud are you talking about ?\n\nEDIT : I have seen the \"Evidence regarding private sharing\" post, did not know that kind of stuff existed on kaggle", "974396": "Only two teams from original top 50 stayed in private top 50. #Brutal\n\nI almost wonder if some of those public kernels had private lb data poisoning. 🤔 But that wouldn't / shouldn't have affected to top brass.", "974425": "See, no need to get angry ;)", "974428": "Would weight initialization factor in when using pre-trained weights? I mean, sure for the classifier it would, but I wouldn't think the rest of the model. [torchtoolbox](https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwjLyP-2vaPrAhWFRDABHenpDI0QFjAAegQIAhAB&url=https%3A%2F%2Fgithub.com%2FPistonY%2Ftorch-toolbox&usg=AOvVaw0WM5IEoEwwJErPEYxtS9YJl) has KaimingInitializer and XavierInitializer that were based on well known published papers to try.", "974431": "Good point.", "974881": "You are right. Kaggle is best in handling this.", "975112": "pytorch effnets perform better on private LB ;)", "975123": "christofhenkel what was your team's CV? Did you ensemble with any public notebooks or only use your own models?", "975136": "Best for us was an ensemble of effnets (pytorch + GPU ) \n\nCV 95.3 / Public LB 95.3 / Private LB 0.9441 \n\nwe also chose one submission where we blended with best public kernel as a gamble, but it performed poorly on private.", "975159": "There appears to be a strong correlation between CV and private LB. My CV was 0.9505 and private LB was 0.9420 (then a bit higher using public notebook CSVs) and your CV was 0.953 and private LB 0.9441. Great job Dieter and Psi", "975160": "Just because they dropped still doesnt make it ok :)", "975190": "Agreed, it needs to be stamped out and made clear that people shouldn't be paying for solutions! The fact that the people using these solutions dropped in some ways makes the scam even worse... Despite this, I think lots of people in the Gold/Silver/Bronze zone are still suspicious (clearly multiple accounts etc) but perhaps this isn't so different to any other Kaggle competition?", "975333": "> Just because they dropped still doesnt make it ok :)\n\nOf course. I hope they'll be removed and banned from Kaggle. But would people have gotten angry like that if cheaters public scores were not that high? I don't think so. I was reacting to this precisely: people were acting as if gold medals could be bought. They can't.", "975339": "I wouldn't be surprised if they could. There looks like there is some evidence that people are paying to join teams of GMs/Masters (and then being a shadow member) in order to obtain gold/silver/bronze medals..", "975355": "That's not the same as what triggered these discussions. \n\nI call what you describe the cuckoo strategy. Some Grandmasters are mastering it as well, they join late high ranking teams extremely often.\n\nThe best known example a cuckoo has been banned definitely from Kagggle, but not because of it. He was banned because he cheated badly in a comp and the scam was discovered several months later.\n\nI have been invited a couple of times to join a high ranked team although I wasn't active in a comp. I feel like an impostor in these cases, and tend to decline now.", "975477": "We used TIMM implementation of Efficientnets which can be syncnorm'd in DistributedDataParallel (but not in DataParallel), and this works also in FP16:\n\n```\n if (not in_nb()) and (a.local_rank is not None):\n learn.model = nn.SyncBatchNorm.convert_sync_batchnorm(learn.model)\n learn.to_distributed(a.local_rank)\n else:\n learn.to_parallel()\n```\n\nOne thing I am not sure is how/if you should scale LR based on number of GPUs (assuming BS scales w/ GPUS) and if so, if scaling should be different in DDP vs. DP (due to difference loss computation in both cases).", "975481": "Good description, I always assumed this was happening but thought it was OK for GMs to join teams and not do a lot of work as long as they were providing a good amount of guidance - which I'm sure most do." }, "source": "meta" }