Kaggle's competition is disappointing, I will update this and write a solution for you guys later.\nWhy later? I mean, I love the fact that using tricks is parts of the fun in Kaggle, but I hate to get 1.000 just by doing these unpractical things.
I need to calm down.\nI was right: @sermakarevich understands the data better
\n\n\n\nI published my code here: https://www.kaggle.com/kokecacao/how-to-get-1-000-lb-1-000?scriptVersionId=11987485\nI hope Kaggle can make another test set. But unlike Ship detecting, it is a playground, it is very unlikely that this will happen.\nI spend most of my afterschool time training models for this, hoping that the competition will prove my power in ML. But now?
\n\nOne take away: don't ever participate in Kaggle playground!\nStop training your models and share your thoughts below!
\n\n(The competition rule also says that we are allowed to use external dataset.)\n(I was 41st before the submission)
\n\nI know I kinda ruined the competition. But as @interneuron said if stuff like this is left out, it will be found. To declare that I am not the smartest among all 1,057 participants, I published the solution with code. I did learn a lot from this competition (including looking for leaks) But if you want to learn more, forget about this post and keep going. This post shouldn't be the reason why you stop.
Since the competition leak is out, I will publish my tries on finding the leak (most of them failed).\nSee this kernel: https://www.kaggle.com/kokecacao/fork-of-generating-salt-jigsaw-puzzle-solut-6fa062\nI also updated my explanations for the leak: https://www.kaggle.com/kokecacao/how-to-get-1-000-lb-1-000
\n\nI did not expect more and more people trying to get 1.000 LB even they know they are using this leak and that they did nothing good for the community. Forgive me, please. So I made the kernel private (will open 5 days later), trying to stop this trend...
\n\nThe competition is over, made it public: https://www.kaggle.com/kokecacao/how-to-get-1-000-lb-1-000
", "messages": [ { "id": 498990, "postDate": "2019-03-24T07:02:10.410Z", "content": "First of all, for everyone who worked with inflammation/cancer detection on WSI previously, it was known since the beginning, that correct answers are known. As there are not that many publicly available datasets of this kind. Same thing was with Dogs bread competition (fair score is ~0.13) and with Titanic competition (names of those who survived are known). Thats why it is a playground competition and thats why nobody will change the test set. It's up to everyone here if to play fair or just submit known answers. Unfortunately in every competition I mentioned people tend to show on LB that they know where the correct answer are. However this move might be a little bit disappointing for people who treated this competition seriously and made 200+ submissions. \nSo thanks for sharing and congrats.
", "rawMarkdown": "First of all, for everyone who worked with inflammation/cancer detection on WSI previously, it was known since the beginning, that correct answers are known. As there are not that many publicly available datasets of this kind. Same thing was with [Dogs bread competition](https://www.kaggle.com/c/dog-breed-identification/leaderboard) (fair score is ~0.13) and with Titanic competition (names of those who survived are known). Thats why it is a playground competition and thats why nobody will change the test set. It's up to everyone here if to play fair or just submit known answers. Unfortunately in every competition I mentioned people tend to show on LB that they know where the correct answer are. However this move might be a little bit disappointing for people who treated this competition seriously and made 200+ submissions. \nSo thanks for sharing and congrats. ", "votes": 17 }, { "id": 499045, "postDate": "2019-03-24T08:45:26.637Z", "content": "Im not sure about the right word. Frustrated or disappointed. Invested so much time in this
", "rawMarkdown": "Im not sure about the right word. Frustrated or disappointed. Invested so much time in this", "votes": 7 }, { "id": 499042, "postDate": "2019-03-24T08:35:22.650Z", "content": "Crazy, our first competition, experimented so much with approaches... With our single model without this \"wsi-story\" we get 0.9825. With weighted average of different models (bagging) we get 0,9834.
", "rawMarkdown": "Crazy, our first competition, experimented so much with approaches... With our single model without this \"wsi-story\" we get 0.9825. With weighted average of different models (bagging) we get 0,9834.", "votes": 6, "replies": [ { "id": 499141, "postDate": "2019-03-24T11:35:07.867Z", "content": "Wow! Your single model is amazing. Looking forward to seeing your pipeline after the competition. Did you get 0.9825 after ensembling your cv or just a model with tta?
", "rawMarkdown": "Wow! Your single model is amazing. Looking forward to seeing your pipeline after the competition. Did you get 0.9825 after ensembling your cv or just a model with tta?" }, { "id": 499198, "postDate": "2019-03-24T13:25:46.867Z", "content": "I'm so disappointed that someone submitted result with the true labels of test data even if Chen encourage us not to. Your model is really good, looking forward to learning from you. I'll continue to try to break 0.98 with my single model without wsi too. Wsi maybe good for but must be unfair. Btw, did your single model use TTA?
", "rawMarkdown": "I'm so disappointed that someone submitted result with the true labels of test data even if Chen encourage us not to. Your model is really good, looking forward to learning from you. I'll continue to try to break 0.98 with my single model without wsi too. Wsi maybe good for but must be unfair. Btw, did your single model use TTA? ", "votes": 2 }, { "id": 499258, "postDate": "2019-03-24T14:58:25.717Z", "content": "Thats really great work Dima! I totally agree with you and feel the same pain. At the same time, I hate cancer and will continue to work to improve my models. I am looking forward very much to sharing my solution when the contest is over and learning more from your approach. We may not finish on the top of the leaderboard, but improving AUC by a few percent over the pcam baseline is a real accomplishment that will be of real benefit to people facing this disease. Thanks for your hard work.
", "rawMarkdown": "Thats really great work Dima! I totally agree with you and feel the same pain. At the same time, I hate cancer and will continue to work to improve my models. I am looking forward very much to sharing my solution when the contest is over and learning more from your approach. We may not finish on the top of the leaderboard, but improving AUC by a few percent over the pcam baseline is a real accomplishment that will be of real benefit to people facing this disease. Thanks for your hard work.", "votes": 1 }, { "id": 499264, "postDate": "2019-03-24T15:10:27.283Z", "content": "Thank you for feedback. After the competition I will share our approach.
", "rawMarkdown": "Thank you for feedback. After the competition I will share our approach. ", "votes": 4 }, { "id": 502190, "postDate": "2019-03-28T09:08:16.630Z", "content": "Amazing... can't wait to read your approach
", "rawMarkdown": "Amazing... can't wait to read your approach" } ] }, { "id": 497807, "postDate": "2019-03-24T00:16:45.810Z", "content": "Kaggle's competition is disappointing, I will update this and write a solution for you guys later.\nWhy later? I mean, I love the fact that using tricks is parts of the fun in Kaggle, but I hate to get 1.000 just by doing these unpractical things.
I need to calm down.\nI was right: @sermakarevich understands the data better
\n\n\n\nI published my code here: https://www.kaggle.com/kokecacao/how-to-get-1-000-lb-1-000?scriptVersionId=11987485\nI hope Kaggle can make another test set. But unlike Ship detecting, it is a playground, it is very unlikely that this will happen.\nI spend most of my afterschool time training models for this, hoping that the competition will prove my power in ML. But now?
\n\nOne take away: don't ever participate in Kaggle playground!\nStop training your models and share your thoughts below!
\n\n(The competition rule also says that we are allowed to use external dataset.)\n(I was 41st before the submission)
\n\nI know I kinda ruined the competition. But as @interneuron said if stuff like this is left out, it will be found. To declare that I am not the smartest among all 1,057 participants, I published the solution with code. I did learn a lot from this competition (including looking for leaks) But if you want to learn more, forget about this post and keep going. This post shouldn't be the reason why you stop.
Since the competition leak is out, I will publish my tries on finding the leak (most of them failed).\nSee this kernel: https://www.kaggle.com/kokecacao/fork-of-generating-salt-jigsaw-puzzle-solut-6fa062\nI also updated my explanations for the leak: https://www.kaggle.com/kokecacao/how-to-get-1-000-lb-1-000
\n\nI did not expect more and more people trying to get 1.000 LB even they know they are using this leak and that they did nothing good for the community. Forgive me, please. So I made the kernel private (will open 5 days later), trying to stop this trend...
\n\nThe competition is over, made it public: https://www.kaggle.com/kokecacao/how-to-get-1-000-lb-1-000
", "rawMarkdown": "# The Competition is OVER\n\nKaggle's competition is disappointing, I will update this and write a solution for you guys later.\nWhy later? I mean, I love the fact that using tricks is parts of the fun in Kaggle, but I hate to get 1.000 just by doing these `unpractical` things.\n\nI need to calm down.\nI was right: @sermakarevich understands the data better\n\n\n\n## Update 1\n**I published my code here**: https://www.kaggle.com/kokecacao/how-to-get-1-000-lb-1-000?scriptVersionId=11987485\nI hope Kaggle can make another test set. But unlike Ship detecting, it is a playground, it is very unlikely that this will happen.\nI spend most of my afterschool time training models for this, hoping that the competition will prove my power in ML. But now?\n\nOne take away: don't ever participate in Kaggle playground!\nStop training your models and share your thoughts below!\n\n## Update 2\n(The competition rule also says that we are allowed to use external dataset.)\n(I was 41st before the submission)\n\nI know I kinda ruined the competition. But as @interneuron said `if stuff like this is left out, it will be found`. To declare that I am not the smartest among all 1,057 participants, I published the solution with code. I did learn a lot from this competition (including looking for leaks) But if you want to learn more, forget about this post and keep going. This post shouldn't be the reason why you stop.\n\n## Update 3\n# I encourage you guys not to submit 1.000 as your choices for the private LB. Do not use any labels from the test set. However, if you do, nobody can find out about that, but the shame will remain forever in you.\n\n## Update 4\nSince the competition leak is out, I will publish my tries on finding the leak (most of them failed).\nSee this kernel: https://www.kaggle.com/kokecacao/fork-of-generating-salt-jigsaw-puzzle-solut-6fa062\nI also updated my explanations for the leak: https://www.kaggle.com/kokecacao/how-to-get-1-000-lb-1-000\n\n## Update 5\nI did not expect more and more people trying to get 1.000 LB even they know they are using this leak and that they did nothing good for the community. Forgive me, please. So I made the kernel private (will open 5 days later), trying to stop this trend...\n\n## Update 6\nThe competition is over, made it public: https://www.kaggle.com/kokecacao/how-to-get-1-000-lb-1-000", "votes": 5 }, { "id": 497880, "postDate": "2019-03-24T02:53:30.340Z", "content": "In all candor, it’s better leaks of this magnitude are made public rather than used in secret to make legit looking solutions that send people on snipe hunts. If one person finds it others will. It’s still a disappointment for anyone who spent some real effort to improve automated cancer detection, whatever their score, myself included.
\n\nI’ll post my solution when it’s over, so far I have a single model able to get 0.9808 on public lb and I will keep at improving it until the end.
", "rawMarkdown": "In all candor, it’s better leaks of this magnitude are made public rather than used in secret to make legit looking solutions that send people on snipe hunts. If one person finds it others will. It’s still a disappointment for anyone who spent some real effort to improve automated cancer detection, whatever their score, myself included. \n\nI’ll post my solution when it’s over, so far I have a single model able to get 0.9808 on public lb and I will keep at improving it until the end. ", "votes": 3, "replies": [ { "id": 497882, "postDate": "2019-03-24T03:00:01.427Z", "content": "Really powerful model, looking forward to it!
", "rawMarkdown": "Really powerful model, looking forward to it!", "votes": 1 } ] }, { "id": 497808, "postDate": "2019-03-24T00:18:03.527Z", "content": "Cool, hhhh. Looking forward to it. Maybe you get inspiration from the discussion 100% WSI assignment, and get information of test images in the original github dataset like tumor_patch and center_tumor_patch. lol
", "rawMarkdown": "Cool, hhhh. Looking forward to it. Maybe you get inspiration from the discussion 100% WSI assignment, and get information of test images in the original github dataset like tumor_patch and center_tumor_patch. lol", "votes": 3, "replies": [ { "id": 497823, "postDate": "2019-03-24T01:00:38.370Z", "content": "Not exactly, I was inspired by the name of the user called dirty trick who got to the first only with <20 submissions and the fact that SM (who were for a long time the 1st lb) leaves the competition for a long time and never comes back.
I see. It seems like https://www.kaggle.com/c/histopathologic-cancer-detection/discussion/85283 decode everything. You got 1.0 and can't get lower score, I think you might want to reset your score and display the score of your models not this dirty trick. What a pity to be 1st place........
", "rawMarkdown": "I see. It seems like https://www.kaggle.com/c/histopathologic-cancer-detection/discussion/85283 decode everything. You got 1.0 and can't get lower score, I think you might want to reset your score and display the score of your models not this dirty trick. What a pity to be 1st place........", "votes": 2 }, { "id": 497848, "postDate": "2019-03-24T02:03:48.203Z", "content": "Is there a way to reset your score? But even so, all the rest of the scores will be meaningless unless Kaggle published another test set for us, which is unlikely.
", "rawMarkdown": "Is there a way to reset your score? But even so, all the rest of the scores will be meaningless unless Kaggle published another test set for us, which is unlikely." }, { "id": 497851, "postDate": "2019-03-24T02:09:36.990Z", "content": "It is a shame, the pcam information has been available from the beginning along with the suggestive filenames for the images. Posting it a few days before the competition is over is questionable, but the strength of our models remains.
", "rawMarkdown": "It is a shame, the pcam information has been available from the beginning along with the suggestive filenames for the images. Posting it a few days before the competition is over is questionable, but the strength of our models remains.", "votes": 2 }, { "id": 497879, "postDate": "2019-03-24T02:51:26.720Z", "content": "Understanding data is much more important in ML, so you did a very good job. As a rookie, I have spent all of my time on models and training tricks to build my own pipeline code, I'll learn from you to dig deeply in my next competition.
", "rawMarkdown": "Understanding data is much more important in ML, so you did a very good job. As a rookie, I have spent all of my time on models and training tricks to build my own pipeline code, I'll learn from you to dig deeply in my next competition. ", "votes": 2 }, { "id": 497885, "postDate": "2019-03-24T03:02:22.180Z", "content": "Posting it a few days before the competition is over is questionable @interneuron I posted it within an hour as soon as I found the leak. The next thing we should worry about is how to get 0.9963 (with 16 submissions) and 0.9865 without using the leak.
This falls under forensics really, you can learn some neat tricks from the code but it’s not ml and not understanding the data. Understanding the data means the ability to distinguish cancer from healthy tissue with high confidence and very low false negatives in any similar images, not just those with poorly secured labels.
", "rawMarkdown": "This falls under forensics really, you can learn some neat tricks from the code but it’s not ml and not understanding the data. Understanding the data means the ability to distinguish cancer from healthy tissue with high confidence and very low false negatives in any similar images, not just those with poorly secured labels. ", "votes": 1 }, { "id": 497889, "postDate": "2019-03-24T03:07:04.173Z", "content": "I’m not ragging on you, I appreciate that you disclosed it quickly. I am curious about the top solutions as well, we will see I suppose.
", "rawMarkdown": "I’m not ragging on you, I appreciate that you disclosed it quickly. I am curious about the top solutions as well, we will see I suppose. ", "votes": 1 }, { "id": 497895, "postDate": "2019-03-24T03:15:31.453Z", "content": "I think you may misunderstand me, lol. I'm not saying about this trick, his another discussion shows he was trying to find something from the detail of images not the labels or filenames.
", "rawMarkdown": "I think you may misunderstand me, lol. I'm not saying about this trick, his another discussion shows he was trying to find something from the detail of images not the labels or filenames. ", "votes": 1 }, { "id": 497899, "postDate": "2019-03-24T03:20:00.690Z", "content": "Well, in any case, learn away!
", "rawMarkdown": "Well, in any case, learn away!" } ] }, { "id": 499240, "postDate": "2019-03-24T14:27:45.280Z", "content": "Now I feel bad for sharing my 100% WSI assignment code and process. I should probably have just posted the new WSI without disclosing how I created it. I'm new to kaggle competitions and am here to learn from others sharing so thought it would be helpful to share the slide assignment to allow focus to move from WSI to better models. I had not thought about anyone using the approach to cheat and ruin the competition.
", "rawMarkdown": "Now I feel bad for sharing my 100% WSI assignment code and process. I should probably have just posted the new WSI without disclosing how I created it. I'm new to kaggle competitions and am here to learn from others sharing so thought it would be helpful to share the slide assignment to allow focus to move from WSI to better models. I had not thought about anyone using the approach to cheat and ruin the competition. \n", "votes": 4, "replies": [ { "id": 499283, "postDate": "2019-03-24T15:44:11.560Z", "content": "Not your fault Mark, the link to the information needed is in the description page of this competition. There is of course no benefit to just submitting the labels, even if there was a prize here such subs would be disqualified. It does hurt, but the cheating cannot take away from the strength of the models we have built for this quite difficult task.
", "rawMarkdown": "Not your fault Mark, the link to the information needed is in the description page of this competition. There is of course no benefit to just submitting the labels, even if there was a prize here such subs would be disqualified. It does hurt, but the cheating cannot take away from the strength of the models we have built for this quite difficult task. ", "votes": 8 } ] }, { "id": 499187, "postDate": "2019-03-24T13:02:33.043Z", "content": "really disappointed ... nothing to add. It is also strange that there is not a single organizer for this competition to give us some feedback.
", "rawMarkdown": "really disappointed ... nothing to add. It is also strange that there is not a single organizer for this competition to give us some feedback. ", "votes": 4 }, { "id": 499054, "postDate": "2019-03-24T09:07:48.773Z", "content": "Most people know that data sets are public and don't modify test data. No one is looking for the right test tags, but your results are unfair to other hard-working submit authors. I do not know what to say,
", "rawMarkdown": "Most people know that data sets are public and don't modify test data. No one is looking for the right test tags, but your results are unfair to other hard-working submit authors. I do not know what to say,", "votes": 4 }, { "id": 499004, "postDate": "2019-03-24T07:17:44.410Z", "content": "I remember that someone get LB 1.000 at the beginning of this competition, but was removed after a few days
", "rawMarkdown": "I remember that someone get LB 1.000 at the beginning of this competition, but was removed after a few days", "votes": 4, "replies": [ { "id": 499595, "postDate": "2019-03-25T01:46:35.723Z", "content": "yeah,but soon he launched this game,instead of hurting this game like this
", "rawMarkdown": "yeah,but soon he launched this game,instead of hurting this game like this", "votes": 2 } ] }, { "id": 500644, "postDate": "2019-03-26T11:08:43.980Z", "content": "I have absolutely no clue about what happened, but making a kernel private after it has been read and probably cloned by some is totally unfair, and can be assimilated to private sharing of information outside teams, which is not allowed by the competition rules.
", "rawMarkdown": "I have absolutely no clue about what happened, but making a kernel private after it has been read and probably cloned by some is totally unfair, and can be assimilated to private sharing of information outside teams, which is not allowed by the competition rules.", "votes": 1, "replies": [ { "id": 500742, "postDate": "2019-03-26T13:41:14.583Z", "content": "The kernel is about how to predict test labels with known test labels. You don't need to train anything - a 1,0 score is guaranteed for everyone. This is the best Data Science I've ever seen ))))))))
", "rawMarkdown": "The kernel is about how to predict test labels with known test labels. You don't need to train anything - a 1,0 score is guaranteed for everyone. This is the best Data Science I've ever seen ))))))))", "votes": 1 }, { "id": 500839, "postDate": "2019-03-26T15:58:52.857Z", "content": "I guessed that, but that's not the point I am discussing. I am discussing a private communication that leads to winning the competition.
", "rawMarkdown": "I guessed that, but that's not the point I am discussing. I am discussing a private communication that leads to winning the competition." }, { "id": 500971, "postDate": "2019-03-26T18:42:01.867Z", "content": "Making it a relatively fair game and discourage cheatings are both my purposes. In terms of legitimacy: If making a competition kernel private is illegal, Kaggle would not have this feature in the first place. I did everything on Kaggle's open platform, thus everybody should have an equal chance to see my code (plus the code was badly written and would not run properly on Kaggle.)
", "rawMarkdown": "Making it a relatively fair game and discourage cheatings are both my purposes. In terms of legitimacy: If making a competition kernel private is illegal, Kaggle would not have this feature in the first place. I did everything on Kaggle's open platform, thus everybody should have an equal chance to see my code (plus the code was badly written and would not run properly on Kaggle.)" }, { "id": 501114, "postDate": "2019-03-26T23:41:03.663Z", "content": "Kaggle cannot prevent someone from erasing his/her content, that's why you can make your kernel private after having shared it. It does not mean it is fair.
", "rawMarkdown": "Kaggle cannot prevent someone from erasing his/her content, that's why you can make your kernel private after having shared it. It does not mean it is fair.", "votes": 1 } ] }, { "id": 500519, "postDate": "2019-03-26T06:51:08.113Z", "content": "you should make the kernel private immediately.
", "rawMarkdown": "you should make the kernel private immediately.", "votes": 1 }, { "id": 499038, "postDate": "2019-03-24T08:33:26.150Z", "content": "Can't they solve this problem by augmenting the test data in a way that makes matching it to the known labels much harder? E.g. adding random rotations, very slight noise, etc. Basically by employing techniques used for adversarial attacks? And, of course, change the IDs as well...
\n\nThis should be relatively little effort but solve the problem, no?
", "rawMarkdown": "Can't they solve this problem by augmenting the test data in a way that makes matching it to the known labels much harder? E.g. adding random rotations, very slight noise, etc. Basically by employing techniques used for adversarial attacks? And, of course, change the IDs as well...\n\nThis should be relatively little effort but solve the problem, no?", "votes": 1, "replies": [ { "id": 499118, "postDate": "2019-03-24T11:01:58.193Z", "content": "Sure, but since they can create the data this way, it will become a race of augmentation.
", "rawMarkdown": "Sure, but since they can create the data this way, it will become a race of augmentation.", "votes": -3 }, { "id": 500151, "postDate": "2019-03-25T16:51:48.733Z", "content": "Well, you would have to get a relatively high accuracy in extracting the true labels to compete with the top models, hence it would be much more reasonable to just focus on training a good model.
\n\nTBH I find it a bit sloppy that they did not do this in the first place to avoid this entire situation.
", "rawMarkdown": "Well, you would have to get a relatively high accuracy in extracting the true labels to compete with the top models, hence it would be much more reasonable to just focus on training a good model. \n\nTBH I find it a bit sloppy that they did not do this in the first place to avoid this entire situation." } ] }, { "id": 497835, "postDate": "2019-03-24T01:28:21.483Z", "content": "nice job
", "rawMarkdown": "nice job", "votes": 1 }, { "id": 497828, "postDate": "2019-03-24T01:15:08.083Z", "content": "Nice work, if stuff like this is left out, it will be found. Aside from the trick, it was fun smashing the pcam published benchmark.
", "rawMarkdown": "Nice work, if stuff like this is left out, it will be found. Aside from the trick, it was fun smashing the pcam published benchmark. ", "votes": 1 }, { "id": 500477, "postDate": "2019-03-26T04:45:10.273Z", "content": "You can use another submissions for private LB
", "rawMarkdown": "You can use another submissions for private LB", "votes": 2 }, { "id": 500471, "postDate": "2019-03-26T04:32:48.457Z", "content": "Firstly thanks for making it private. You get many dislikes beacause those want use tricks to get high score without experiments. But it's unfair for people who try experiments several days even months. The aim of compepitions is developing algorithms and people can learn something from the compepitions. Secondly , in my opinion, high score kernels should't been published before the end of the compepitions, it broke the balance. Good idea can be shared in Dicussion and without codes. Although I'm poor in English, sentences may have problems, I still want to speak my voice. Finally I will continue try to get a higher score by myself. even if I can't get top 10 or top20, the rank is useless in my heart, and thanks again for what you did and found, I hope you can get a higher score too.
", "rawMarkdown": "Firstly thanks for making it private. You get many dislikes beacause those want use tricks to get high score without experiments. But it's unfair for people who try experiments several days even months. The aim of compepitions is developing algorithms and people can learn something from the compepitions. Secondly , in my opinion, high score kernels should't been published before the end of the compepitions, it broke the balance. Good idea can be shared in Dicussion and without codes. Although I'm poor in English, sentences may have problems, I still want to speak my voice. Finally I will continue try to get a higher score by myself. even if I can't get top 10 or top20, the rank is useless in my heart, and thanks again for what you did and found, I hope you can get a higher score too.", "votes": 2, "replies": [ { "id": 500487, "postDate": "2019-03-26T05:19:24.287Z", "content": "👍
", "rawMarkdown": "👍", "votes": 2 }, { "id": 500528, "postDate": "2019-03-26T07:12:20.773Z", "content": "Well said in any language.
", "rawMarkdown": "Well said in any language. ", "votes": 1 } ] }, { "id": 499725, "postDate": "2019-03-25T05:53:23.030Z", "content": "You should delete the kernel
", "rawMarkdown": "You should delete the kernel", "votes": 1, "replies": [ { "id": 499937, "postDate": "2019-03-25T12:20:05.817Z", "content": "Good suggestion. I made it private.
", "rawMarkdown": "Good suggestion. I made it private." }, { "id": 500024, "postDate": "2019-03-25T14:35:27.073Z", "content": "for what it's worth I have deleted my code and explanation too.
", "rawMarkdown": "for what it's worth I have deleted my code and explanation too.", "votes": 1 }, { "id": 500135, "postDate": "2019-03-25T16:34:40.607Z", "content": "Don't know what happened. But it seems like I got many dislikes after I made my kernel private. Funny.
Out of curiosity, is this trick matching the test images with patches in the PCam dataset?
", "rawMarkdown": "Out of curiosity, is this trick matching the test images with patches in the PCam dataset?" }, { "id": 501155, "postDate": "2019-03-27T01:36:32.970Z", "content": "This cheating behavior has no meaning
", "rawMarkdown": "This cheating behavior has no meaning" }, { "id": 499741, "postDate": "2019-03-25T06:14:23.343Z", "rawMarkdown": "", "isDeleted": true }, { "id": 499278, "postDate": "2019-03-24T15:36:17.970Z", "rawMarkdown": "", "isDeleted": true }, { "id": 499158, "postDate": "2019-03-24T12:13:53.023Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 498990, "author_name": "SM", "author_url": "", "post_date": "2019-03-24T07:02:10.410000", "content": "First of all, for everyone who worked with inflammation/cancer detection on WSI previously, it was known since the beginning, that correct answers are known. As there are not that many publicly available datasets of this kind. Same thing was with Dogs bread competition (fair score is ~0.13) and with Titanic competition (names of those who survived are known). Thats why it is a playground competition and thats why nobody will change the test set. It's up to everyone here if to play fair or just submit known answers. Unfortunately in every competition I mentioned people tend to show on LB that they know where the correct answer are. However this move might be a little bit disappointing for people who treated this competition seriously and made 200+ submissions. \nSo thanks for sharing and congrats.
", "votes": 17, "replies": [] }, { "id": 499045, "author_name": "Samuel Abramov", "author_url": "", "post_date": "2019-03-24T08:45:26.637000", "content": "Im not sure about the right word. Frustrated or disappointed. Invested so much time in this
", "votes": 7, "replies": [] }, { "id": 499042, "author_name": "Dimitrij Shulkin", "author_url": "", "post_date": "2019-03-24T08:35:22.650000", "content": "Crazy, our first competition, experimented so much with approaches... With our single model without this \"wsi-story\" we get 0.9825. With weighted average of different models (bagging) we get 0,9834.
", "votes": 6, "replies": [ { "id": 499141, "author_name": "Hanke Chen", "author_url": "", "post_date": "2019-03-24T11:35:07.867000", "content": "Wow! Your single model is amazing. Looking forward to seeing your pipeline after the competition. Did you get 0.9825 after ensembling your cv or just a model with tta?
", "votes": 0, "replies": [] }, { "id": 499198, "author_name": "jionie", "author_url": "", "post_date": "2019-03-24T13:25:46.867000", "content": "I'm so disappointed that someone submitted result with the true labels of test data even if Chen encourage us not to. Your model is really good, looking forward to learning from you. I'll continue to try to break 0.98 with my single model without wsi too. Wsi maybe good for but must be unfair. Btw, did your single model use TTA?
", "votes": 2, "replies": [] }, { "id": 499258, "author_name": "interneuron", "author_url": "", "post_date": "2019-03-24T14:58:25.717000", "content": "Thats really great work Dima! I totally agree with you and feel the same pain. At the same time, I hate cancer and will continue to work to improve my models. I am looking forward very much to sharing my solution when the contest is over and learning more from your approach. We may not finish on the top of the leaderboard, but improving AUC by a few percent over the pcam baseline is a real accomplishment that will be of real benefit to people facing this disease. Thanks for your hard work.
", "votes": 1, "replies": [] }, { "id": 499264, "author_name": "Dimitrij Shulkin", "author_url": "", "post_date": "2019-03-24T15:10:27.283000", "content": "Thank you for feedback. After the competition I will share our approach.
", "votes": 4, "replies": [] }, { "id": 502190, "author_name": "Shaohua Li", "author_url": "", "post_date": "2019-03-28T09:08:16.630000", "content": "Amazing... can't wait to read your approach
", "votes": 0, "replies": [] } ] }, { "id": 497880, "author_name": "interneuron", "author_url": "", "post_date": "2019-03-24T02:53:30.340000", "content": "In all candor, it’s better leaks of this magnitude are made public rather than used in secret to make legit looking solutions that send people on snipe hunts. If one person finds it others will. It’s still a disappointment for anyone who spent some real effort to improve automated cancer detection, whatever their score, myself included.
\n\nI’ll post my solution when it’s over, so far I have a single model able to get 0.9808 on public lb and I will keep at improving it until the end.
", "votes": 3, "replies": [ { "id": 497882, "author_name": "jionie", "author_url": "", "post_date": "2019-03-24T03:00:01.427000", "content": "Really powerful model, looking forward to it!
", "votes": 1, "replies": [] } ] }, { "id": 497808, "author_name": "jionie", "author_url": "", "post_date": "2019-03-24T00:18:03.527000", "content": "Cool, hhhh. Looking forward to it. Maybe you get inspiration from the discussion 100% WSI assignment, and get information of test images in the original github dataset like tumor_patch and center_tumor_patch. lol
", "votes": 3, "replies": [ { "id": 497823, "author_name": "Hanke Chen", "author_url": "", "post_date": "2019-03-24T01:00:38.370000", "content": "Not exactly, I was inspired by the name of the user called dirty trick who got to the first only with <20 submissions and the fact that SM (who were for a long time the 1st lb) leaves the competition for a long time and never comes back.
I see. It seems like https://www.kaggle.com/c/histopathologic-cancer-detection/discussion/85283 decode everything. You got 1.0 and can't get lower score, I think you might want to reset your score and display the score of your models not this dirty trick. What a pity to be 1st place........
", "votes": 2, "replies": [] }, { "id": 497848, "author_name": "Hanke Chen", "author_url": "", "post_date": "2019-03-24T02:03:48.203000", "content": "Is there a way to reset your score? But even so, all the rest of the scores will be meaningless unless Kaggle published another test set for us, which is unlikely.
", "votes": 0, "replies": [] }, { "id": 497851, "author_name": "interneuron", "author_url": "", "post_date": "2019-03-24T02:09:36.990000", "content": "It is a shame, the pcam information has been available from the beginning along with the suggestive filenames for the images. Posting it a few days before the competition is over is questionable, but the strength of our models remains.
", "votes": 2, "replies": [] }, { "id": 497879, "author_name": "jionie", "author_url": "", "post_date": "2019-03-24T02:51:26.720000", "content": "Understanding data is much more important in ML, so you did a very good job. As a rookie, I have spent all of my time on models and training tricks to build my own pipeline code, I'll learn from you to dig deeply in my next competition.
", "votes": 2, "replies": [] }, { "id": 497885, "author_name": "Hanke Chen", "author_url": "", "post_date": "2019-03-24T03:02:22.180000", "content": "Posting it a few days before the competition is over is questionable @interneuron I posted it within an hour as soon as I found the leak. The next thing we should worry about is how to get 0.9963 (with 16 submissions) and 0.9865 without using the leak.
This falls under forensics really, you can learn some neat tricks from the code but it’s not ml and not understanding the data. Understanding the data means the ability to distinguish cancer from healthy tissue with high confidence and very low false negatives in any similar images, not just those with poorly secured labels.
", "votes": 1, "replies": [] }, { "id": 497889, "author_name": "interneuron", "author_url": "", "post_date": "2019-03-24T03:07:04.173000", "content": "I’m not ragging on you, I appreciate that you disclosed it quickly. I am curious about the top solutions as well, we will see I suppose.
", "votes": 1, "replies": [] }, { "id": 497895, "author_name": "jionie", "author_url": "", "post_date": "2019-03-24T03:15:31.453000", "content": "I think you may misunderstand me, lol. I'm not saying about this trick, his another discussion shows he was trying to find something from the detail of images not the labels or filenames.
", "votes": 1, "replies": [] }, { "id": 497899, "author_name": "interneuron", "author_url": "", "post_date": "2019-03-24T03:20:00.690000", "content": "Well, in any case, learn away!
", "votes": 0, "replies": [] } ] }, { "id": 499240, "author_name": "Mark De Simone", "author_url": "", "post_date": "2019-03-24T14:27:45.280000", "content": "Now I feel bad for sharing my 100% WSI assignment code and process. I should probably have just posted the new WSI without disclosing how I created it. I'm new to kaggle competitions and am here to learn from others sharing so thought it would be helpful to share the slide assignment to allow focus to move from WSI to better models. I had not thought about anyone using the approach to cheat and ruin the competition.
", "votes": 4, "replies": [ { "id": 499283, "author_name": "interneuron", "author_url": "", "post_date": "2019-03-24T15:44:11.560000", "content": "Not your fault Mark, the link to the information needed is in the description page of this competition. There is of course no benefit to just submitting the labels, even if there was a prize here such subs would be disqualified. It does hurt, but the cheating cannot take away from the strength of the models we have built for this quite difficult task.
", "votes": 8, "replies": [] } ] }, { "id": 499187, "author_name": "Amirreza Mahbod", "author_url": "", "post_date": "2019-03-24T13:02:33.043000", "content": "really disappointed ... nothing to add. It is also strange that there is not a single organizer for this competition to give us some feedback.
", "votes": 4, "replies": [] }, { "id": 499054, "author_name": "quinwu", "author_url": "", "post_date": "2019-03-24T09:07:48.773000", "content": "Most people know that data sets are public and don't modify test data. No one is looking for the right test tags, but your results are unfair to other hard-working submit authors. I do not know what to say,
", "votes": 4, "replies": [] }, { "id": 499004, "author_name": "StepD", "author_url": "", "post_date": "2019-03-24T07:17:44.410000", "content": "I remember that someone get LB 1.000 at the beginning of this competition, but was removed after a few days
", "votes": 4, "replies": [ { "id": 499595, "author_name": "quinwu", "author_url": "", "post_date": "2019-03-25T01:46:35.723000", "content": "yeah,but soon he launched this game,instead of hurting this game like this
", "votes": 2, "replies": [] } ] }, { "id": 500644, "author_name": "CPMP", "author_url": "", "post_date": "2019-03-26T11:08:43.980000", "content": "I have absolutely no clue about what happened, but making a kernel private after it has been read and probably cloned by some is totally unfair, and can be assimilated to private sharing of information outside teams, which is not allowed by the competition rules.
", "votes": 1, "replies": [ { "id": 500742, "author_name": "Dimitrij Shulkin", "author_url": "", "post_date": "2019-03-26T13:41:14.583000", "content": "The kernel is about how to predict test labels with known test labels. You don't need to train anything - a 1,0 score is guaranteed for everyone. This is the best Data Science I've ever seen ))))))))
", "votes": 1, "replies": [] }, { "id": 500839, "author_name": "CPMP", "author_url": "", "post_date": "2019-03-26T15:58:52.857000", "content": "I guessed that, but that's not the point I am discussing. I am discussing a private communication that leads to winning the competition.
", "votes": 0, "replies": [] }, { "id": 500971, "author_name": "Hanke Chen", "author_url": "", "post_date": "2019-03-26T18:42:01.867000", "content": "Making it a relatively fair game and discourage cheatings are both my purposes. In terms of legitimacy: If making a competition kernel private is illegal, Kaggle would not have this feature in the first place. I did everything on Kaggle's open platform, thus everybody should have an equal chance to see my code (plus the code was badly written and would not run properly on Kaggle.)
", "votes": 0, "replies": [] }, { "id": 501114, "author_name": "CPMP", "author_url": "", "post_date": "2019-03-26T23:41:03.663000", "content": "Kaggle cannot prevent someone from erasing his/her content, that's why you can make your kernel private after having shared it. It does not mean it is fair.
", "votes": 1, "replies": [] } ] }, { "id": 500519, "author_name": "quinwu", "author_url": "", "post_date": "2019-03-26T06:51:08.113000", "content": "you should make the kernel private immediately.
", "votes": 1, "replies": [] }, { "id": 499038, "author_name": "Pablo Gómez", "author_url": "", "post_date": "2019-03-24T08:33:26.150000", "content": "Can't they solve this problem by augmenting the test data in a way that makes matching it to the known labels much harder? E.g. adding random rotations, very slight noise, etc. Basically by employing techniques used for adversarial attacks? And, of course, change the IDs as well...
\n\nThis should be relatively little effort but solve the problem, no?
", "votes": 1, "replies": [ { "id": 499118, "author_name": "Hanke Chen", "author_url": "", "post_date": "2019-03-24T11:01:58.193000", "content": "Sure, but since they can create the data this way, it will become a race of augmentation.
", "votes": -3, "replies": [] }, { "id": 500151, "author_name": "Pablo Gómez", "author_url": "", "post_date": "2019-03-25T16:51:48.733000", "content": "Well, you would have to get a relatively high accuracy in extracting the true labels to compete with the top models, hence it would be much more reasonable to just focus on training a good model.
\n\nTBH I find it a bit sloppy that they did not do this in the first place to avoid this entire situation.
", "votes": 0, "replies": [] } ] }, { "id": 497835, "author_name": "", "author_url": "", "post_date": "2019-03-24T01:28:21.483000", "content": "nice job
", "votes": 1, "replies": [] }, { "id": 497828, "author_name": "interneuron", "author_url": "", "post_date": "2019-03-24T01:15:08.083000", "content": "Nice work, if stuff like this is left out, it will be found. Aside from the trick, it was fun smashing the pcam published benchmark.
", "votes": 1, "replies": [] }, { "id": 500477, "author_name": "zxyu", "author_url": "", "post_date": "2019-03-26T04:45:10.273000", "content": "You can use another submissions for private LB
", "votes": 2, "replies": [] }, { "id": 500471, "author_name": "zxyu", "author_url": "", "post_date": "2019-03-26T04:32:48.457000", "content": "Firstly thanks for making it private. You get many dislikes beacause those want use tricks to get high score without experiments. But it's unfair for people who try experiments several days even months. The aim of compepitions is developing algorithms and people can learn something from the compepitions. Secondly , in my opinion, high score kernels should't been published before the end of the compepitions, it broke the balance. Good idea can be shared in Dicussion and without codes. Although I'm poor in English, sentences may have problems, I still want to speak my voice. Finally I will continue try to get a higher score by myself. even if I can't get top 10 or top20, the rank is useless in my heart, and thanks again for what you did and found, I hope you can get a higher score too.
", "votes": 2, "replies": [ { "id": 500487, "author_name": "Dimitrij Shulkin", "author_url": "", "post_date": "2019-03-26T05:19:24.287000", "content": "👍
", "votes": 2, "replies": [] }, { "id": 500528, "author_name": "interneuron", "author_url": "", "post_date": "2019-03-26T07:12:20.773000", "content": "Well said in any language.
", "votes": 1, "replies": [] } ] }, { "id": 499725, "author_name": "zxyu", "author_url": "", "post_date": "2019-03-25T05:53:23.030000", "content": "You should delete the kernel
", "votes": 1, "replies": [ { "id": 499937, "author_name": "Hanke Chen", "author_url": "", "post_date": "2019-03-25T12:20:05.817000", "content": "Good suggestion. I made it private.
", "votes": 0, "replies": [] }, { "id": 500024, "author_name": "Mark De Simone", "author_url": "", "post_date": "2019-03-25T14:35:27.073000", "content": "for what it's worth I have deleted my code and explanation too.
", "votes": 1, "replies": [] }, { "id": 500135, "author_name": "Hanke Chen", "author_url": "", "post_date": "2019-03-25T16:34:40.607000", "content": "Don't know what happened. But it seems like I got many dislikes after I made my kernel private. Funny.
Out of curiosity, is this trick matching the test images with patches in the PCam dataset?
", "votes": 0, "replies": [] }, { "id": 501155, "author_name": "Levi", "author_url": "", "post_date": "2019-03-27T01:36:32.970000", "content": "This cheating behavior has no meaning
", "votes": 0, "replies": [] }, { "id": 499741, "author_name": "", "author_url": "", "post_date": "2019-03-25T06:14:23.343000", "content": "", "votes": 0, "replies": [] }, { "id": 499278, "author_name": "", "author_url": "", "post_date": "2019-03-24T15:36:17.970000", "content": "", "votes": 0, "replies": [] }, { "id": 499158, "author_name": "", "author_url": "", "post_date": "2019-03-24T12:13:53.023000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "498990": "First of all, for everyone who worked with inflammation/cancer detection on WSI previously, it was known since the beginning, that correct answers are known. As there are not that many publicly available datasets of this kind. Same thing was with [Dogs bread competition](https://www.kaggle.com/c/dog-breed-identification/leaderboard) (fair score is ~0.13) and with Titanic competition (names of those who survived are known). Thats why it is a playground competition and thats why nobody will change the test set. It's up to everyone here if to play fair or just submit known answers. Unfortunately in every competition I mentioned people tend to show on LB that they know where the correct answer are. However this move might be a little bit disappointing for people who treated this competition seriously and made 200+ submissions. \nSo thanks for sharing and congrats. ", "499045": "Im not sure about the right word. Frustrated or disappointed. Invested so much time in this", "499042": "Crazy, our first competition, experimented so much with approaches... With our single model without this \"wsi-story\" we get 0.9825. With weighted average of different models (bagging) we get 0,9834.", "497807": "# The Competition is OVER\n\nKaggle's competition is disappointing, I will update this and write a solution for you guys later.\nWhy later? I mean, I love the fact that using tricks is parts of the fun in Kaggle, but I hate to get 1.000 just by doing these `unpractical` things.\n\nI need to calm down.\nI was right: @sermakarevich understands the data better\n\n\n\n## Update 1\n**I published my code here**: https://www.kaggle.com/kokecacao/how-to-get-1-000-lb-1-000?scriptVersionId=11987485\nI hope Kaggle can make another test set. But unlike Ship detecting, it is a playground, it is very unlikely that this will happen.\nI spend most of my afterschool time training models for this, hoping that the competition will prove my power in ML. But now?\n\nOne take away: don't ever participate in Kaggle playground!\nStop training your models and share your thoughts below!\n\n## Update 2\n(The competition rule also says that we are allowed to use external dataset.)\n(I was 41st before the submission)\n\nI know I kinda ruined the competition. But as @interneuron said `if stuff like this is left out, it will be found`. To declare that I am not the smartest among all 1,057 participants, I published the solution with code. I did learn a lot from this competition (including looking for leaks) But if you want to learn more, forget about this post and keep going. This post shouldn't be the reason why you stop.\n\n## Update 3\n# I encourage you guys not to submit 1.000 as your choices for the private LB. Do not use any labels from the test set. However, if you do, nobody can find out about that, but the shame will remain forever in you.\n\n## Update 4\nSince the competition leak is out, I will publish my tries on finding the leak (most of them failed).\nSee this kernel: https://www.kaggle.com/kokecacao/fork-of-generating-salt-jigsaw-puzzle-solut-6fa062\nI also updated my explanations for the leak: https://www.kaggle.com/kokecacao/how-to-get-1-000-lb-1-000\n\n## Update 5\nI did not expect more and more people trying to get 1.000 LB even they know they are using this leak and that they did nothing good for the community. Forgive me, please. So I made the kernel private (will open 5 days later), trying to stop this trend...\n\n## Update 6\nThe competition is over, made it public: https://www.kaggle.com/kokecacao/how-to-get-1-000-lb-1-000", "497880": "In all candor, it’s better leaks of this magnitude are made public rather than used in secret to make legit looking solutions that send people on snipe hunts. If one person finds it others will. It’s still a disappointment for anyone who spent some real effort to improve automated cancer detection, whatever their score, myself included. \n\nI’ll post my solution when it’s over, so far I have a single model able to get 0.9808 on public lb and I will keep at improving it until the end. ", "497808": "Cool, hhhh. Looking forward to it. Maybe you get inspiration from the discussion 100% WSI assignment, and get information of test images in the original github dataset like tumor_patch and center_tumor_patch. lol", "499240": "Now I feel bad for sharing my 100% WSI assignment code and process. I should probably have just posted the new WSI without disclosing how I created it. I'm new to kaggle competitions and am here to learn from others sharing so thought it would be helpful to share the slide assignment to allow focus to move from WSI to better models. I had not thought about anyone using the approach to cheat and ruin the competition. \n", "499187": "really disappointed ... nothing to add. It is also strange that there is not a single organizer for this competition to give us some feedback. ", "499054": "Most people know that data sets are public and don't modify test data. No one is looking for the right test tags, but your results are unfair to other hard-working submit authors. I do not know what to say,", "499004": "I remember that someone get LB 1.000 at the beginning of this competition, but was removed after a few days", "500644": "I have absolutely no clue about what happened, but making a kernel private after it has been read and probably cloned by some is totally unfair, and can be assimilated to private sharing of information outside teams, which is not allowed by the competition rules.", "500519": "you should make the kernel private immediately.", "499038": "Can't they solve this problem by augmenting the test data in a way that makes matching it to the known labels much harder? E.g. adding random rotations, very slight noise, etc. Basically by employing techniques used for adversarial attacks? And, of course, change the IDs as well...\n\nThis should be relatively little effort but solve the problem, no?", "497835": "nice job", "497828": "Nice work, if stuff like this is left out, it will be found. Aside from the trick, it was fun smashing the pcam published benchmark. ", "500477": "You can use another submissions for private LB", "500471": "Firstly thanks for making it private. You get many dislikes beacause those want use tricks to get high score without experiments. But it's unfair for people who try experiments several days even months. The aim of compepitions is developing algorithms and people can learn something from the compepitions. Secondly , in my opinion, high score kernels should't been published before the end of the compepitions, it broke the balance. Good idea can be shared in Dicussion and without codes. Although I'm poor in English, sentences may have problems, I still want to speak my voice. Finally I will continue try to get a higher score by myself. even if I can't get top 10 or top20, the rank is useless in my heart, and thanks again for what you did and found, I hope you can get a higher score too.", "499725": "You should delete the kernel", "502195": "Out of curiosity, is this trick matching the test images with patches in the PCam dataset?", "501155": "This cheating behavior has no meaning", "499741": "", "499278": "", "499158": "" } }