Hi all,
\n\nI realise this might not make me the most popular person on this forum but it's something I think needs to be addressed and kaggle (e.g @philculliton) should provide an answer.
\n\nTucked away in a discussion here is the revelation that the (difficult to access) HPA data has been used to boost @tomomimoriyama (plus I imagine others) to the top of the LB. In all fairness to the few people discussing this topic they have acknowledged it's uncertain as to whether it's allowed or not but I for one would like this explicit and clarified.
\n\nUse of the HPA data: I'm only going on what little is mentioned but it appears the HPA data is not being used in the sense I understand external data should to be used (e.g. word embeddings) in order to supplement a problem but instead people have basically found the same images in the HPA data as appear in the train and test sets. i.e. they've found some of the answers. I cannot imagine this is what the competition organisers intend when they challenge us to automate 'biomedical image analysis to accelerate the understanding of human cells and disease'.
\n\nFurther, from what I can gather, the HPA data is a pain to access and as pointed out by @stecasasso (Chase the Trane) this sort of external data has been banned in previous competitions. I can't force the kaggle or the organisers to ban it in this case but I can safely say that if a prerequisite for tackling image competitions with huge data is to have to parse even more hard to reach huge data to simply uncover test images it's definitely not what I had in mind when I decided to start this competition and use machine learning for good.
\n\nA final point: if I've gotten any of the above mixed up I am happy to amend my post for inaccuracies or clarify misunderstanding. I'd also like to thank those using the HPA data for being honest about finding similar images in it, particularly @ldm314 (Brian) and @tomomimoriyama.
\n\nCheers,
\n\nMark
\n\nP. S. also mentioned here
", "messages": [ { "id": 431652, "postDate": "2018-12-02T17:28:20.183Z", "content": "Hi all,
\n\nI realise this might not make me the most popular person on this forum but it's something I think needs to be addressed and kaggle (e.g @philculliton) should provide an answer.
\n\nTucked away in a discussion here is the revelation that the (difficult to access) HPA data has been used to boost @tomomimoriyama (plus I imagine others) to the top of the LB. In all fairness to the few people discussing this topic they have acknowledged it's uncertain as to whether it's allowed or not but I for one would like this explicit and clarified.
\n\nUse of the HPA data: I'm only going on what little is mentioned but it appears the HPA data is not being used in the sense I understand external data should to be used (e.g. word embeddings) in order to supplement a problem but instead people have basically found the same images in the HPA data as appear in the train and test sets. i.e. they've found some of the answers. I cannot imagine this is what the competition organisers intend when they challenge us to automate 'biomedical image analysis to accelerate the understanding of human cells and disease'.
\n\nFurther, from what I can gather, the HPA data is a pain to access and as pointed out by @stecasasso (Chase the Trane) this sort of external data has been banned in previous competitions. I can't force the kaggle or the organisers to ban it in this case but I can safely say that if a prerequisite for tackling image competitions with huge data is to have to parse even more hard to reach huge data to simply uncover test images it's definitely not what I had in mind when I decided to start this competition and use machine learning for good.
\n\nA final point: if I've gotten any of the above mixed up I am happy to amend my post for inaccuracies or clarify misunderstanding. I'd also like to thank those using the HPA data for being honest about finding similar images in it, particularly @ldm314 (Brian) and @tomomimoriyama.
\n\nCheers,
\n\nMark
\n\nP. S. also mentioned here
", "rawMarkdown": "Hi all,\n\nI realise this might not make me the most popular person on this forum but it's something I think needs to be addressed and kaggle (e.g @philculliton) should provide an answer.\n\nTucked away in a discussion [here][1] is the revelation that the (difficult to access) HPA data has been used to boost @tomomimoriyama (plus I imagine others) to the top of the LB. In all fairness to the few people discussing this topic they have acknowledged it's uncertain as to whether it's allowed or not but I for one would like this explicit and clarified. \n\n**Use of the HPA data**: I'm only going on what little is mentioned but it appears the HPA data is not being used in the sense I understand external data should to be used (e.g. word embeddings) in order to supplement a problem but instead people have basically found the same images in the HPA data as appear in the train and test sets. i.e. they've found some of the answers. I cannot imagine this is what the competition organisers intend when they challenge us to automate *'biomedical image analysis to accelerate the understanding of human cells and disease'.*\n\nFurther, from what I can gather, the HPA data is a pain to access and as pointed out by @stecasasso (Chase the Trane) this sort of external data has been banned in previous competitions. I can't force the kaggle or the organisers to ban it in this case but I can safely say that if a prerequisite for tackling image competitions with huge data is to have to parse even more hard to reach huge data to simply uncover test images it's definitely not what I had in mind when I decided to start this competition and use machine learning for good. \n\nA final point: if I've gotten any of the above mixed up I am happy to amend my post for inaccuracies or clarify misunderstanding. I'd also like to thank those using the HPA data for being honest about finding similar images in it, particularly @ldm314 (Brian) and @tomomimoriyama.\n\nCheers,\n\nMark\n\nP. S. also mentioned [here][2]\n\n\n [1]: https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/70206\n [2]: https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/72534", "votes": 52 }, { "id": 433361, "postDate": "2018-12-05T02:09:09.363Z", "content": "It's a mistake to use competition participants' findings (the leaks) for test set update. Just ban the external data.
", "rawMarkdown": "It's a mistake to use competition participants' findings (the leaks) for test set update. Just ban the external data.", "votes": 21, "replies": [ { "id": 433971, "postDate": "2018-12-05T17:54:39.610Z", "content": "Lots of people have already spent lots of time on the external data, so it would be unfair to ban it now.
", "rawMarkdown": "Lots of people have already spent lots of time on the external data, so it would be unfair to ban it now.", "votes": -2 } ] }, { "id": 433019, "postDate": "2018-12-04T15:12:24.667Z", "content": "Hi all - thanks for bringing this to my attention. The samples in question will no longer be included in scoring. I'll be making that change (and rescoring / resetting the leaderboard) today.
\n\nMark, @ldm314 and @tomomimoriyama, I'll be in touch with you via DM. I appreciate your efforts on this.
", "rawMarkdown": "Hi all - thanks for bringing this to my attention. The samples in question will no longer be included in scoring. I'll be making that change (and rescoring / resetting the leaderboard) today.\n\nMark, @ldm314 and @tomomimoriyama, I'll be in touch with you via DM. I appreciate your efforts on this.", "votes": 12, "replies": [ { "id": 433021, "postDate": "2018-12-04T15:16:38.683Z", "content": "Thank you!
\n\nCould you please provide a full list of leaked test IDs as well as their classes after rescoring?
", "rawMarkdown": "Thank you!\n\nCould you please provide a full list of leaked test IDs as well as their classes after rescoring?", "votes": 2 }, { "id": 433022, "postDate": "2018-12-04T15:17:48.313Z", "content": "Thank you Phil!
", "rawMarkdown": "Thank you Phil!" }, { "id": 433031, "postDate": "2018-12-04T15:32:10.070Z", "content": "Thank you!
", "rawMarkdown": "Thank you!" }, { "id": 433062, "postDate": "2018-12-04T16:08:07.513Z", "content": "Hi @philculliton,
\n\nFirst thanks for your quick reaction on this.
\n\nIs the aim to simply remove those samples ? Or replace them with images that are not part of the public data ?
\n\nI'm afraid that removing them will make the distribution even more imbalanced and increase the instability of the f1 score.
\n\nFor instance for class 27, we have only 11 samples in the training set for 31072 images. By removing those 259 samples you'd remove 5 samples with class 27 from 11702 samples. So it's probably a very important % of this class, if not the whole class.
", "rawMarkdown": "Hi @philculliton,\n\nFirst thanks for your quick reaction on this.\n\nIs the aim to simply remove those samples ? Or replace them with images that are not part of the public data ? \n\nI'm afraid that removing them will make the distribution even more imbalanced and increase the instability of the f1 score.\n\nFor instance for class 27, we have only 11 samples in the training set for 31072 images. By removing those 259 samples you'd remove 5 samples with class 27 from 11702 samples. So it's probably a very important % of this class, if not the whole class.", "votes": 2 }, { "id": 433091, "postDate": "2018-12-04T17:22:17.183Z", "content": "Thank you Phil.
", "rawMarkdown": "Thank you Phil." }, { "id": 433186, "postDate": "2018-12-04T20:13:34.483Z", "content": "Thanks!
", "rawMarkdown": "Thanks!" }, { "id": 433348, "postDate": "2018-12-05T01:48:00.893Z", "content": "Looks like the leaderboard has not been reset yet - is that true? I guess the comments here mean that new submissions ARE being reset?
", "rawMarkdown": "Looks like the leaderboard has not been reset yet - is that true? I guess the comments here mean that new submissions ARE being reset?" }, { "id": 433372, "postDate": "2018-12-05T02:20:06.457Z", "content": "I just submitted one of my earlier submissions as a test and got the same score as before, so new submissions have not been reset as yet.
\n\nIncidentally my latest model which got 0.595 on this un-rescored LB uses the external HPA data but does not do anything special with the offending images. It will be interesting to see the difference once the leaderboard has been re-scored. I would have assumed that re-scoring will cause people that made special use of those images would see their scores drop, but based on Brian's comment below, maybe that will not always be the case!
", "rawMarkdown": "I just submitted one of my earlier submissions as a test and got the same score as before, so new submissions have not been reset as yet.\n\nIncidentally my latest model which got 0.595 on this un-rescored LB uses the external HPA data but does not do anything special with the offending images. It will be interesting to see the difference once the leaderboard has been re-scored. I would have assumed that re-scoring will cause people that made special use of those images would see their scores drop, but based on Brian's comment below, maybe that will not always be the case!" }, { "id": 433394, "postDate": "2018-12-05T02:58:16.230Z", "content": "looks ike something's happening to the LB now. In progress.
", "rawMarkdown": "looks ike something's happening to the LB now. In progress." }, { "id": 433481, "postDate": "2018-12-05T05:53:13.037Z", "content": "Thanks Phil for your prompt attention to this problem.\nAre you also going to delete the leaked test images from test.zip and test_full_size.7z?
", "rawMarkdown": "Thanks Phil for your prompt attention to this problem.\nAre you also going to delete the leaked test images from test.zip and test_full_size.7z?" }, { "id": 433981, "postDate": "2018-12-05T18:10:51.790Z", "content": "Hi @dslate -
\n\nGood question! I won't be removing the leaked test images. All current competitors have already downloaded them, so this would primarily negatively impact later competitors. Removing them from scoring mitigates the effects of the leak just as readily, without asking later competitors to go without information / images that everyone else had.
", "rawMarkdown": "Hi @dslate - \n\nGood question! I won't be removing the leaked test images. All current competitors have already downloaded them, so this would primarily negatively impact later competitors. Removing them from scoring mitigates the effects of the leak just as readily, without asking later competitors to go without information / images that everyone else had." } ] }, { "id": 433234, "postDate": "2018-12-04T21:48:41.850Z", "content": "Here is the 126 I've been using, this time with the image IDS.
", "rawMarkdown": "Here is the 126 I've been using, this time with the image IDS. ", "votes": 9 }, { "id": 431818, "postDate": "2018-12-03T00:14:07.397Z", "content": "My proposal to even out the playing field is to take all of the v18 HPA and include it here with the training data. Remove the leaked images from the test set. I found and use ~125 matches and @tomomimoriyama found over twice that. With the test set being 11k images, removing the overlap would leave plenty to test with.
", "rawMarkdown": "My proposal to even out the playing field is to take all of the v18 HPA and include it here with the training data. Remove the leaked images from the test set. I found and use ~125 matches and @tomomimoriyama found over twice that. With the test set being 11k images, removing the overlap would leave plenty to test with.", "votes": 7, "replies": [ { "id": 431976, "postDate": "2018-12-03T08:03:11.823Z", "content": "Yeah, or just announce they aren't part of the private LB in which case using for public LB will bias your results. This of course relies on them being able to reliably identify the true extent of the overlap/leak.
", "rawMarkdown": "Yeah, or just announce they aren't part of the private LB in which case using for public LB will bias your results. This of course relies on them being able to reliably identify the true extent of the overlap/leak.", "votes": 1 }, { "id": 432107, "postDate": "2018-12-03T12:24:11.930Z", "content": "I agree with Brian proposal concerning making HPA data available.
\n\nFor the leakage, I don't think the organizers will take any action on that, because the leakage is not severe and the vast majority of the test set is not affected. This expectation is based on the few competitions \"with leak\" that I took part in. For example, in Santander the leakage was much more evident and the organizers did not do anything about it. However, someone shared part of the leakage in kernels, but the full set of leaky rows of the test set was never released and only the top of the LB managed to exploit it 100%.
", "rawMarkdown": "I agree with Brian proposal concerning making HPA data available. \n\nFor the leakage, I don't think the organizers will take any action on that, because the leakage is not severe and the vast majority of the test set is not affected. This expectation is based on the few competitions \"with leak\" that I took part in. For example, in Santander the leakage was much more evident and the organizers did not do anything about it. However, someone shared part of the leakage in kernels, but the full set of leaky rows of the test set was never released and only the top of the LB managed to exploit it 100%. " }, { "id": 432140, "postDate": "2018-12-03T13:04:45.983Z", "content": "Jumping from 0.528 to 0.588 because of the leak is severe in my view.
\n\nI also don't think adding the HPA data to the training data really levels the playing field - it favours even more those with massive compute power.
", "rawMarkdown": "Jumping from 0.528 to 0.588 because of the leak is severe in my view. \n\nI also don't think adding the HPA data to the training data really levels the playing field - it favours even more those with massive compute power." }, { "id": 432251, "postDate": "2018-12-03T16:01:10.893Z", "content": "Regarding the jump in the LB: I may agree with you (what I wrote before is what I think it's the reasoning of the organizers, not necessarily mine...) .
\n\nFor the second point, some personal considerations:
\n - Kaggle is biased towards competitors with large computing power and there's little or nothing we can do about it .
\n - Making the additional data available to everyone is the only thing that can be done to level the field again. External data has been announced (by Brian) on the public thread a month ago and no one from the Kaggle team has replied that it was prohibited. If they banned it now, it would be very unfair towards people who have built their models and spent their submissions using that data (N.B.: it's not me, I am not using HPA data so far).
\n - More data instances is, technically speaking, more \"democratic\" than higher resolution data. You could train a model in several steps on different Kaggle kernels, by re-loading the weights from the previous step. On the other hand, higher resolution images require gpu with larger memory, which are very expensive and clearly not available to everyone.
Now, my opinion on what happened in this competition:
\n- it was a mistake of the organizers to make higher resolution images available, as it increases the \"technology gap\" a lot;
\n- the HPA dataset should be made available by someone who owns it - better if in agreement with the organizers, so we are sure that really all the available external data is included;
\n- we should not blame on people who discovered and used the leak: this is just fine according to the rules, as they announced publicly the dataset they were using
Hi Chase the Trane (@stecasasso),
\n\nI agree with nearly all of what you said but think you are slightly conflating two points.
\n\nI don't think it's an issue for people to use the HPA data (and don't want it banned) - we just need to make sure that the HPA data doesn't contain any of the answers to the live competition.
\n\nMark
", "rawMarkdown": "Hi Chase the Trane (@stecasasso),\n\nI agree with nearly all of what you said but think you are slightly conflating two points.\n\nI don't think it's an issue for people to use the HPA data (and don't want it banned) - we just need to make sure that the HPA data doesn't contain any of the answers to the live competition. \n\nMark", "votes": 2 }, { "id": 432517, "postDate": "2018-12-04T01:39:40.273Z", "content": "Hi Mark Worrall,\nI agree with you that people should be able to use the HPA data but that data shouldn't contain any leaked labels for the competition test set. As far as I can see, the only way to accomplish this would be to find all the leaks and remove their images from the test set, both public and private. This has the downside that every submission will have to be rescored and the LB recomputed, but with over a month to go until the contest ends this may be the best solution. The organizers would have to decide to do this pretty soon, however.
", "rawMarkdown": "Hi Mark Worrall,\nI agree with you that people should be able to use the HPA data but that data shouldn't contain any leaked labels for the competition test set. As far as I can see, the only way to accomplish this would be to find all the leaks and remove their images from the test set, both public and private. This has the downside that every submission will have to be rescored and the LB recomputed, but with over a month to go until the contest ends this may be the best solution. The organizers would have to decide to do this pretty soon, however.", "votes": 5 }, { "id": 432810, "postDate": "2018-12-04T10:40:37.930Z", "content": "Even if no other action is taken, we need an urgent examination of the private test set to find out just how contaminated it is. 126 images is only >1% of the public test. For all we know the percentage is much higher in the unreleased data. I would really like some reassurance that contamination is really only the minor issue that it appears to be at present time.
", "rawMarkdown": "Even if no other action is taken, we need an **urgent examination of the private test set** to find out just how contaminated it is. 126 images is only >1% of the public test. For all we know the percentage is much higher in the unreleased data. I would really like some reassurance that contamination is really only the minor issue that it appears to be at present time." } ] }, { "id": 433830, "postDate": "2018-12-05T14:35:58.963Z", "content": "Hey all -
\n\nI updated and reset the LB last night. Here's what changed:
\n\n1) The majority of the impacted samples are no longer included in scoring.
\n\n2) A small number of samples with very rare labels were not removed from scoring, but were shifted to the public leaderboard.
\n\nBoth of these moves will affect public LB scores, especially for less common labels. If you're concerned about your score changing after this fix was implemented, the fix did impact everyone's scores - whether they were using the leak or not. Any shift on the leaderboard, whether ignoring samples or moving them between public and private LBs, will affect scores. I do understand the confusion expressed about scores changing: I sincerely apologize and hope this clears things up!
\n\nThanks again to everyone for your feedback and help! It's much appreciated.
", "rawMarkdown": "Hey all -\n\nI updated and reset the LB last night. Here's what changed:\n\n1) The majority of the impacted samples are no longer included in scoring.\n\n2) A small number of samples with very rare labels were not removed from scoring, but were shifted to the public leaderboard.\n\nBoth of these moves will affect public LB scores, especially for less common labels. If you're concerned about your score changing after this fix was implemented, the fix did impact everyone's scores - whether they were using the leak or not. Any shift on the leaderboard, whether ignoring samples or moving them between public and private LBs, will affect scores. I do understand the confusion expressed about scores changing: I sincerely apologize and hope this clears things up!\n\nThanks again to everyone for your feedback and help! It's much appreciated.", "votes": 7, "replies": [ { "id": 433899, "postDate": "2018-12-05T16:21:31.900Z", "content": "@philculliton Thanks for the info. I think this should be a separate topic and pinned to the top of discussion board, as not everyone is subscribed to the whole forum.
", "rawMarkdown": "@philculliton Thanks for the info. I think this should be a separate topic and pinned to the top of discussion board, as not everyone is subscribed to the whole forum.", "votes": 5 }, { "id": 433917, "postDate": "2018-12-05T16:45:29.167Z", "content": "I guess that rare classes that were already on the public LB were left there? And more shifted from the private? It’s hard to see how that would lower public scores of those using the leak. It might generate a massive shakeup at the end, though. Wouldnt it have been better to just remove leaks entirely, even from rare classes? And if some classes were rendered 1/10000 or something, remove the classes.
", "rawMarkdown": "I guess that rare classes that were already on the public LB were left there? And more shifted from the private? It’s hard to see how that would lower public scores of those using the leak. It might generate a massive shakeup at the end, though. Wouldnt it have been better to just remove leaks entirely, even from rare classes? And if some classes were rendered 1/10000 or something, remove the classes." }, { "id": 433920, "postDate": "2018-12-05T16:49:30.530Z", "content": "Yeah, so using the file can cover for a model missing some of the rare classes as these are kept in public LB...but this means private LB will likely be very different.
", "rawMarkdown": "Yeah, so using the file can cover for a model missing some of the rare classes as these are kept in public LB...but this means private LB will likely be very different." }, { "id": 433946, "postDate": "2018-12-05T17:20:24.307Z", "content": "There going to be a nice shakeup in this competition :)
", "rawMarkdown": "There going to be a nice shakeup in this competition :)", "votes": 1 }, { "id": 433972, "postDate": "2018-12-05T17:58:50.057Z", "content": "Thank you for the update.\nCould you please confirm that there is no leakage in the private set?\nCould you also publish a list of leaked images with their classes to even the odds?
", "rawMarkdown": "Thank you for the update.\nCould you please confirm that there is no leakage in the private set?\nCould you also publish a list of leaked images with their classes to even the odds?" }, { "id": 433975, "postDate": "2018-12-05T18:03:34.657Z", "content": "Hi Dmytro -
\n\nI can confirm that there is no leakage in the private set. Re: posting a list of the leaked images... I'm considering it. Because some of the images aren't simply being ignored, that makes the impact of releasing the list more complex. I may release the list of the ignored images.
", "rawMarkdown": "Hi Dmytro -\n\nI can confirm that there is no leakage in the private set. Re: posting a list of the leaked images... I'm considering it. Because some of the images aren't simply being ignored, that makes the impact of releasing the list more complex. I may release the list of the ignored images.", "votes": 2 }, { "id": 433986, "postDate": "2018-12-05T18:21:10.297Z", "content": "Here's some random chatter from someone without much experience in DL or Kaggle:
\n\nI'll incorporate the leak to get the 0.05 public LB asap, so can brag to my wife. But the question is now: how do I avoid the shakeup?
\n\nPoint 2 might not be a huge killer because we will be in the prediction stage so the model will have been trained with non-decimated rare classes. However, the leak may reduce the impetus to deal with rare classes, thus deteriorating the effectiveness of the winning models.
\n\nSo the strategy I guess will be to incorporate the leak (altering submission file only) for bragging rights in the next month (and to see exactly where you are in the competition) and develop your model just like nothing happened.
\n\nOr you could copy all leak data (rare classes) into your personal training set to help you better train those classes. It won't hurt the public LB as you are replacing the predictions of those samples anyway.
\n\nRegarding point 1, for completeness, I wonder how many of the leaks have been found. It appears that the organizers are just using the results contributed on these threads - not doing any checks. Perhaps there are many more leaks with very small differences (say by different resampling/antiallias, different versions of photo(?)). And it's not easy to compare a hundred thousand huge images - hashing has been used - how accurate is that in this context? Is there a better way that runs in finite time?
\n\nDefinitely a footnote in the Nature paper.
", "rawMarkdown": "Here's some random chatter from someone without much experience in DL or Kaggle:\n\nI'll incorporate the leak to get the 0.05 public LB asap, so can brag to my wife. But the question is now: how do I avoid the shakeup?\n\n 1. the leak advantage will go away (democratically, I guess)\n 2. the private LB will have a relative dearth of rare classes.\n\nPoint 2 might not be a huge killer because we will be in the prediction stage so the model will have been trained with non-decimated rare classes. However, the leak may reduce the impetus to deal with rare classes, thus deteriorating the effectiveness of the winning models.\n\nSo the strategy I guess will be to incorporate the leak (altering submission file only) for bragging rights in the next month (and to see exactly where you are in the competition) and develop your model just like nothing happened.\n\nOr you could copy all leak data (rare classes) into your personal training set to help you better train those classes. It won't hurt the public LB as you are replacing the predictions of those samples anyway.\n\nRegarding point 1, for completeness, I wonder how many of the leaks have been found. It appears that the organizers are just using the results contributed on these threads - not doing any checks. Perhaps there are many more leaks with very small differences (say by different resampling/antiallias, different versions of photo(?)). And it's not easy to compare a hundred thousand huge images - hashing has been used - how accurate is that in this context? Is there a better way that runs in finite time?\n\nDefinitely a footnote in the Nature paper.", "votes": 2 }, { "id": 434012, "postDate": "2018-12-05T19:08:29.897Z", "content": "Hi pete -
\n\nThanks for the feedback - I appreciate the thought everyone is putting into this!
\n\nRe: your point 2. The rare classes in question still exist in the private LB. Samples were shifted between the private and public LB to ensure this. Anyone who feels a reduced impetus to deal with rare classes will definitely take a hit on the private LB.
\n\nRe: the leaks... we absolutely did our own checks, and did not use the results contributed on the forums. The forum results were useful because they indicated that people had identified a problem. We know the full extent of the potential leakage based on our own work.
\n\nOn a related note, I should mention that hashing is not 100% accurate in this context, and that some of the \"leaked\" images are not, in fact, leaks (and further, some of the labels assigned them are incorrect). The leak appears to be so effective because some of the real, actual leaks include rare classes that have a high impact on LB score.
\n\nThe people who found the leak did excellent work, but it's very difficult to get completely right. I'd recommend not using the leak in any way.
", "rawMarkdown": "Hi pete -\n\nThanks for the feedback - I appreciate the thought everyone is putting into this!\n\nRe: your point 2. The rare classes in question still exist in the private LB. Samples were shifted between the private and public LB to ensure this. Anyone who feels a reduced impetus to deal with rare classes will definitely take a hit on the private LB.\n\nRe: the leaks... we absolutely did our own checks, and did *not* use the results contributed on the forums. The forum results were useful because they indicated that people had identified a problem. We know the full extent of the potential leakage based on our own work.\n\nOn a related note, I should mention that hashing is *not* 100% accurate in this context, and that some of the \"leaked\" images are not, in fact, leaks (and further, some of the labels assigned them are incorrect). The leak appears to be so effective because some of the real, actual leaks include rare classes that have a high impact on LB score.\n\nThe people who found the leak did excellent work, but it's very difficult to get completely right. I'd recommend not using the leak in any way.", "votes": 5 }, { "id": 434100, "postDate": "2018-12-05T22:10:15.150Z", "content": "Phil,
\n\nThanks very much for your comments. My apologies for assuming that you did not look carefully into this - looks like you definitely did. While I'm not fully on board with all of your suggestions, they are very helpful and I appreciate what you have done with this, and your policy of keeping us up to date.
\n\nAlso to those who found the leak and did great work calling attention to it and helped others understand, thank you also. I am super impressed with how fast you understood that complicated website (I still don't really understand it). You improved the competition immeasurably.
", "rawMarkdown": "Phil,\n\nThanks very much for your comments. My apologies for assuming that you did not look carefully into this - looks like you definitely did. While I'm not fully on board with all of your suggestions, they are very helpful and I appreciate what you have done with this, and your policy of keeping us up to date.\n\nAlso to those who found the leak and did great work calling attention to it and helped others understand, thank you also. I am super impressed with how fast you understood that complicated website (I still don't really understand it). You improved the competition immeasurably.", "votes": 2 }, { "id": 434463, "postDate": "2018-12-06T12:43:31.860Z", "content": "Hi Phil - thanks for the info. Do we need to download the test set again if samples were shifted between private and public LB?
", "rawMarkdown": "Hi Phil - thanks for the info. Do we need to download the test set again if samples were shifted between private and public LB?" }, { "id": 434472, "postDate": "2018-12-06T13:08:17.340Z", "content": "@Fionnán Alt: I guess no... \ntest set = public LB + private LB
", "rawMarkdown": "@Fionnán Alt: I guess no... \ntest set = public LB + private LB" }, { "id": 434473, "postDate": "2018-12-06T13:08:18.310Z", "content": "Hi Fionnán,\nGood question! You don't need to download anything new - the shift happened entirely on our end, so you shouldn't have to make any changes.
", "rawMarkdown": "Hi Fionnán,\nGood question! You don't need to download anything new - the shift happened entirely on our end, so you shouldn't have to make any changes." }, { "id": 434501, "postDate": "2018-12-06T14:08:53.157Z", "content": "Thanks Phil.
", "rawMarkdown": "Thanks Phil." } ] }, { "id": 432969, "postDate": "2018-12-04T14:26:38.920Z", "content": "Test matches found by Tomomi Moriyama with labels from HPA.
", "rawMarkdown": "Test matches found by Tomomi Moriyama with labels from HPA.", "votes": 5, "replies": [ { "id": 432980, "postDate": "2018-12-04T14:41:25.410Z", "content": "Thank you Alexander - I hadn't been through all those files.
\n\nP. S. that just took me from 0.482 -> 0.527 ... wonder how many others in the top 100 are using it already. You might also wish to make that csv more public (i.e. kernel).
", "rawMarkdown": "Thank you Alexander - I hadn't been through all those files.\n\nP. S. that just took me from 0.482 -> 0.527 ... wonder how many others in the top 100 are using it already. You might also wish to make that csv more public (i.e. kernel)." }, { "id": 433045, "postDate": "2018-12-04T15:49:52.583Z", "content": "It's funny, using the leak our 0.537 submission using oversampling only got to 0.544, while the 0.507 submission which clearly overfitted to the major classes got to 0.556. I'm glad they decided to take action.
", "rawMarkdown": "It's funny, using the leak our 0.537 submission using oversampling only got to 0.544, while the 0.507 submission which clearly overfitted to the major classes got to 0.556. I'm glad they decided to take action." }, { "id": 433190, "postDate": "2018-12-04T20:20:06.293Z", "content": "Amusingly enough if I apply this file, my score goes down from .531 to .528...
", "rawMarkdown": "Amusingly enough if I apply this file, my score goes down from .531 to .528...", "votes": 1 }, { "id": 433202, "postDate": "2018-12-04T20:39:31.683Z", "content": "@ldm314 do you think @tomomimoriyama is trying to miss lead us? :D
", "rawMarkdown": " @ldm314 do you think @tomomimoriyama is trying to miss lead us? :D", "votes": -1 }, { "id": 433231, "postDate": "2018-12-04T21:47:13.377Z", "content": "I don't think so. I came up with a similar list independently after @tomomimoriyama demonstrated that the yellow channels were available. I ended up with the same score drop vs the 126 list I came up with from only RGB.
", "rawMarkdown": "I don't think so. I came up with a similar list independently after @tomomimoriyama demonstrated that the yellow channels were available. I ended up with the same score drop vs the 126 list I came up with from only RGB." }, { "id": 433461, "postDate": "2018-12-05T05:18:30.957Z", "content": "It is very upset that I don't use leak but all my scores were dropped :(
", "rawMarkdown": "It is very upset that I don't use leak but all my scores were dropped :(" } ] }, { "id": 433563, "postDate": "2018-12-05T07:24:09.127Z", "content": "I don't think the LB has been updated properly (or something weird is going on) as my score has gone from:
\n\n0.482 no leaked data\n0.527 using leaked data\n0.528 post LB update?
", "rawMarkdown": "I don't think the LB has been updated properly (or something weird is going on) as my score has gone from:\n\n0.482 no leaked data\n0.527 using leaked data\n0.528 post LB update?", "votes": 6, "replies": [ { "id": 433575, "postDate": "2018-12-05T07:39:32.640Z", "content": "Yes! I suppose the same! My score without leak has dropped!
", "rawMarkdown": "Yes! I suppose the same! My score without leak has dropped!" }, { "id": 433576, "postDate": "2018-12-05T07:40:53.367Z", "content": "Definitely something weird going on
\n\n0.525 no leak\n0.561 leak \n0.561 after update
", "rawMarkdown": "Definitely something weird going on\n\n0.525 no leak\n0.561 leak \n0.561 after update", "votes": 3 }, { "id": 433600, "postDate": "2018-12-05T08:05:46.760Z", "content": "Drop the samples, they said\nIt would be fair, they said
\n\nP.S. no offense :)
", "rawMarkdown": "Drop the samples, they said\nIt would be fair, they said\n\nP.S. no offense :)", "votes": 5 }, { "id": 433603, "postDate": "2018-12-05T08:08:26.280Z", "content": "Haha - no offence taken. I don't think it's clear what's gone on at the moment but the ids from the csv you provided are still being scored in the public LB somehow.
", "rawMarkdown": "Haha - no offence taken. I don't think it's clear what's gone on at the moment but the ids from the csv you provided are still being scored in the public LB somehow." }, { "id": 433610, "postDate": "2018-12-05T08:13:51.300Z", "content": "Same! Our score without leak has decreased from 0.506 to 0.491 .
", "rawMarkdown": "Same! Our score without leak has decreased from 0.506 to 0.491 ." }, { "id": 433648, "postDate": "2018-12-05T09:43:38.643Z", "rawMarkdown": "", "isDeleted": true }, { "id": 433653, "postDate": "2018-12-05T09:49:56.027Z", "content": "I did not use the leaked labels, and my highest LB score has dropped from 0.504 to 0.499. But that doesn't necessarily indicate any scoring error by the organizers. It could be that even without the leaks, my model did a good job of predicting the affected images, so without them my score dropped a bit.\nThe LB still looks dubious at the high end, however, since those scores seem to have been hardly affected by the \"reset\" at all.
", "rawMarkdown": "I did not use the leaked labels, and my highest LB score has dropped from 0.504 to 0.499. But that doesn't necessarily indicate any scoring error by the organizers. It could be that even without the leaks, my model did a good job of predicting the affected images, so without them my score dropped a bit.\nThe LB still looks dubious at the high end, however, since those scores seem to have been hardly affected by the \"reset\" at all." }, { "id": 433697, "postDate": "2018-12-05T11:14:43.027Z", "content": "my score also decreased from 0.501 to 0.495.
", "rawMarkdown": "my score also decreased from 0.501 to 0.495." }, { "id": 433756, "postDate": "2018-12-05T12:50:26.927Z", "content": "0.570 -> 0.553\nWas I using too many leaks? :P
", "rawMarkdown": "0.570 -> 0.553\nWas I using too many leaks? :P", "votes": 1 }, { "id": 433963, "postDate": "2018-12-05T17:43:55.623Z", "content": "haha, I think so.\nmy score dropped from 0.498 to 0.494,\nusing the leaks, it goes to 0.549
", "rawMarkdown": "haha, I think so.\nmy score dropped from 0.498 to 0.494,\nusing the leaks, it goes to 0.549", "votes": 1 } ] }, { "id": 434133, "postDate": "2018-12-06T00:06:11.560Z", "content": "After the leaderboard changes, in general, models submitted with the 126 samples I have posted went down in score. Models using the list from @tomomimoriyama went up in score. What meaning this has, if any, I won't speculate on as some of these posted answers are confirmed to be inaccurate.
\n\nThanks to @philculliton for cleaning thing up and evening out the field.
", "rawMarkdown": "After the leaderboard changes, in general, models submitted with the 126 samples I have posted went down in score. Models using the list from @tomomimoriyama went up in score. What meaning this has, if any, I won't speculate on as some of these posted answers are confirmed to be inaccurate.\n\nThanks to @philculliton for cleaning thing up and evening out the field.\n\n", "votes": 3 }, { "id": 436658, "postDate": "2018-12-10T17:56:22.613Z", "content": "since this challenge is permited to use the external datasets, can anyone provide a complete HPA dataset to download including full HPA image and train.csv? don't think we should cost much time exploring external datasets.
", "rawMarkdown": "since this challenge is permited to use the external datasets, can anyone provide a complete HPA dataset to download including full HPA image and train.csv? don't think we should cost much time exploring external datasets. ", "votes": 4, "replies": [ { "id": 442604, "postDate": "2018-12-20T07:29:07.140Z", "rawMarkdown": "", "votes": 3, "isDeleted": true }, { "id": 442840, "postDate": "2018-12-20T15:39:13.173Z", "content": "Hi Jarvis, Thank you for these useful information! Are we allowed to use HPA datasets in training a model? Does it not have any implication such as data leakage? Sorry, I didn't check all HPA datasets. Thanks
", "rawMarkdown": "Hi Jarvis, Thank you for these useful information! Are we allowed to use HPA datasets in training a model? Does it not have any implication such as data leakage? Sorry, I didn't check all HPA datasets. Thanks ", "votes": 9 } ] }, { "id": 443202, "postDate": "2018-12-21T07:55:34.103Z", "content": "Hello guys,\nCould anybody clarify, does it make sense to include the leak in the submission now or not?\nThank you!
", "rawMarkdown": "Hello guys,\nCould anybody clarify, does it make sense to include the leak in the submission now or not?\nThank you!", "votes": 1, "replies": [ { "id": 443225, "postDate": "2018-12-21T08:41:46.163Z", "content": "The advantage provided by the leak will probably vanish in the private leaderboard but it doesnt hurt to include it if only to see where you stand. The hpa data themselves give an extra edge in addition to any leak advantage. I dont see any reason why this extra edge wont remain in the private leaderboard.
", "rawMarkdown": "The advantage provided by the leak will probably vanish in the private leaderboard but it doesnt hurt to include it if only to see where you stand. The hpa data themselves give an extra edge in addition to any leak advantage. I dont see any reason why this extra edge wont remain in the private leaderboard." } ] }, { "id": 433691, "postDate": "2018-12-05T10:58:30.890Z", "content": "@philculliton After LB rescoring, my leak only submission (no training) scored 0.085, now it is scoring 0.062 (about two rare classes with F1 close to 1.00), so definitely there are still leakage in public LB. Don't know about private LB though.
", "rawMarkdown": "@philculliton After LB rescoring, my leak only submission (no training) scored 0.085, now it is scoring 0.062 (about two rare classes with F1 close to 1.00), so definitely there are still leakage in public LB. Don't know about private LB though.", "votes": 2 }, { "id": 432185, "postDate": "2018-12-03T14:25:42.633Z", "content": "I concur.
\n\nIf they make this data official and give us an easy zipped download of the HPA in a smaller image size (512p), that would be the best approach. If HPA can afford to have people trawling their servers for tens of thousands of 2048p images from individual urls they can afford to give a direct link to a compressed archive. They must have known that the HPA would be the obvious first point of call for anybody looking for external data.
\n\n@tomomimoriyama and @ldm314 (Brian) have kindly provided access to parsing scripts that allow download of the larger images and label them accordingly. Having to download at mostly 2048px and resize is still a pain. @artemtprv provided a web-scraping kernel to access 'preview' images from the HPA website that are 800x800, but these appear to be either cropped (or at high magnification), and there are far fewer than in the XML urls.
\n\nI am guessing the organisers pre-reduced the training data for the competition to 512px to allow easy download and training for amateur enthusiasts and typical users. Some people don't even have the connection or hdd space to acquire the larger original image set, let alone access to memory and GPUs capable of training models with them properly. Having to download a full size external dataset (even if resized later) to make it onto the top half of the leaderboard is a little extreme.
\n\nThere's still a good amount of time left to fix this so please address it. Even if you just exclude the leaked images, that leaves the option for kind community members to upload the data somewhere independent at a reduced image size with accompanied csv files.
", "rawMarkdown": "I concur.\n\nIf they make this data official and give us an easy zipped download of the HPA in a smaller image size (512p), that would be the best approach. If HPA can afford to have people trawling their servers for tens of thousands of 2048p images from individual urls they can afford to give a direct link to a compressed archive. They must have known that the HPA would be the obvious first point of call for anybody looking for external data.\n\n@tomomimoriyama and @ldm314 (Brian) have kindly provided access to parsing scripts that allow download of the larger images and label them accordingly. Having to download at mostly 2048px and resize is still a pain. @artemtprv provided a web-scraping kernel to access 'preview' images from the HPA website that are 800x800, but these appear to be either cropped (or at high magnification), and there are far fewer than in the XML urls.\n\nI am guessing the organisers pre-reduced the training data for the competition to 512px to allow easy download and training for amateur enthusiasts and typical users. Some people don't even have the connection or hdd space to acquire the larger original image set, let alone access to memory and GPUs capable of training models with them properly. Having to download a full size external dataset (even if resized later) to make it onto the top half of the leaderboard is a little extreme.\n\nThere's still a good amount of time left to fix this so please address it. Even if you just exclude the leaked images, that leaves the option for kind community members to upload the data somewhere independent at a reduced image size with accompanied csv files.", "votes": 2 }, { "id": 432072, "postDate": "2018-12-03T11:14:54.117Z", "content": "Cannot agree more with you. The competition needs to focus on methods and skills instead of \"cheating\" to get a high score.
", "rawMarkdown": "Cannot agree more with you. The competition needs to focus on methods and skills instead of \"cheating\" to get a high score.", "votes": 2 }, { "id": 432160, "postDate": "2018-12-03T13:46:03.287Z", "content": "Wait, what? There is a leak?
", "rawMarkdown": "Wait, what? There is a leak?" }, { "id": 433965, "postDate": "2018-12-05T17:48:11.510Z", "content": "Hmm... based on the scores that people are posting it looks like most of the leaks are still being scored.
", "rawMarkdown": "Hmm... based on the scores that people are posting it looks like most of the leaks are still being scored.", "replies": [ { "id": 433974, "postDate": "2018-12-05T18:00:19.807Z", "content": "Hi Robert -
\n\nWe've changed the public LB exposure for two rare classes, which is having a noticeable effect on public LB scores. An overwhelming majority of the leaked samples are no longer being scored at all.
", "rawMarkdown": "Hi Robert -\n\nWe've changed the public LB exposure for two rare classes, which is having a noticeable effect on public LB scores. An overwhelming majority of the leaked samples are no longer being scored at all." }, { "id": 435873, "postDate": "2018-12-09T00:00:58.267Z", "content": "Is there any mistake here? Before you adjusted, my score was 0.529. After the adjustment, it was 0.513. I didn't use data leak before, but when I use data leak,now my score is 0.555.
", "rawMarkdown": "Is there any mistake here? Before you adjusted, my score was 0.529. After the adjustment, it was 0.513. I didn't use data leak before, but when I use data leak,now my score is 0.555. " }, { "id": 435874, "postDate": "2018-12-09T00:05:24.707Z", "content": "It sounds like reasonable behaviour. I would say it's OK.
", "rawMarkdown": "It sounds like reasonable behaviour. I would say it's OK." }, { "id": 435877, "postDate": "2018-12-09T00:12:00.187Z", "content": "Hi,pete.\nI don't think it's a problem either. I'm mainly concerned about whether there will be a second adjustment, because the adjustment still has a great impact on the score. This may cause some trouble for me to adjust the model.
", "rawMarkdown": "Hi,pete.\nI don't think it's a problem either. I'm mainly concerned about whether there will be a second adjustment, because the adjustment still has a great impact on the score. This may cause some trouble for me to adjust the model. " }, { "id": 439847, "postDate": "2018-12-16T14:09:50.683Z", "content": "Yes, I'm agree with you. This will cause trouble to adjust the model.
", "rawMarkdown": "Yes, I'm agree with you. This will cause trouble to adjust the model." } ] }, { "id": 433469, "postDate": "2018-12-05T05:30:00.543Z", "content": "Leaderboard have been reseted, but leak is applied to public LB.
\n* no leak submission: public LB. 0.496
\n* no leak submission + leak: public LB. 0.502
\nThe effective of leak is small, but public LB is improved.
From my perspective they didn't fix anything much.
\n\nBefore reset + leak: 0.556
After reset + leak: 0.555
Thanks Khoi. I think so too.
\nIt is strange that the scores of many participants have not changed.
I want to point out that some classes are very rare in train and test set, so removing them from test set will reduce them even more and score will be affected a lot by correct/incorrect predictions for that classes. IMHO it will cause a big shakeup later.
", "rawMarkdown": "I want to point out that some classes are very rare in train and test set, so removing them from test set will reduce them even more and score will be affected a lot by correct/incorrect predictions for that classes. IMHO it will cause a big shakeup later.", "replies": [ { "id": 432892, "postDate": "2018-12-04T12:43:51.427Z", "content": "Well this is only the case if the HPA data contains the answers to some of the rare classes - that's the only case where we need to remove from test. The flip side of this is that the leak is even more serious as the impact is large.
\n\nTo be honest, I'm not sure why we are worrying about LB stability when it's just been revealed part of the answers are externally available - this compromises the integrity of the competition which has a metric defined precisely to try to encourage rare class predictions.
", "rawMarkdown": "Well this is only the case if the HPA data contains the answers to some of the rare classes - that's the only case where we need to remove from test. The flip side of this is that the leak is even more serious as the impact is large.\n\nTo be honest, I'm not sure why we are worrying about LB stability when it's just been revealed part of the answers are externally available - this compromises the integrity of the competition which has a metric defined precisely to try to encourage rare class predictions." }, { "id": 432908, "postDate": "2018-12-04T13:07:35.870Z", "content": "\"Leaked\" data have been published already here https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/72534 (you can download images like described here https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/69984 ) and available to everyone. So it should be +- ok if everyone use same labels for that samples, as it doesn't change overall rating. So yes, I'm worrying about stability more.
", "rawMarkdown": "\"Leaked\" data have been published already here https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/72534 (you can download images like described here https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/69984 ) and available to everyone. So it should be +- ok if everyone use same labels for that samples, as it doesn't change overall rating. So yes, I'm worrying about stability more.", "votes": 1 }, { "id": 432915, "postDate": "2018-12-04T13:12:59.543Z", "content": "The leaked test examples have a relative majority of class 16 (Cytokinetic bridge), which is a pretty rare class overall, so it is a bit worrying. https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/69984#423736
", "rawMarkdown": "The leaked test examples have a relative majority of class 16 (Cytokinetic bridge), which is a pretty rare class overall, so it is a bit worrying. https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/69984#423736" }, { "id": 432919, "postDate": "2018-12-04T13:20:49.757Z", "content": "Part of the outcome of this competition is meant to be a Nature Methods paper - hardly worth much if the metric is biased as some people found the answers online.
\nThis is exacerbated as the extent isn't confirmed officially (it's being left to drag out) though I agree if the organisers can confirm the extent and either give a csv with those answers in or exclude them I think that is fine.
\nJust to be clear: in my view asking people to parse and download massive amounts of extra data just to plug a leak is a weak fix. You can worry all you like about stability but if the integrity is compromised it undermines the whole competition.
\nMark
\nP. S. Alexander Kiselev: if you truly want to help out why not provide a csv with all the images from the test set and their labels from HPA if you have them? We can all get back on with the problem at hand then.
", "rawMarkdown": "Part of the outcome of this competition is meant to be a Nature Methods paper - hardly worth much if the metric is biased as some people found the answers online.\n\nThis is exacerbated as the extent isn't confirmed officially (it's being left to drag out) though I agree if the organisers can confirm the extent and either give a csv with those answers in or exclude them I think that is fine.\n\n**Just to be clear:** in my view asking people to parse and download massive amounts of extra data just to plug a leak is a weak fix. You can worry all you like about stability but if the integrity is compromised it undermines the whole competition.\n\nMark\n\nP. S. Alexander Kiselev: if you truly want to help out why not provide a csv with all the images from the test set and their labels from HPA if you have them? We can all get back on with the problem at hand then." }, { "id": 432968, "postDate": "2018-12-04T14:24:38.563Z", "content": "That nice csv file have been already sincerely provided by Tomomi Moriyama (TestEtraMatchingUnder_259_R14_G12_B10.csv from https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/72534 ) all you need is just join it with another csv... Ok let me do it.
", "rawMarkdown": "That nice csv file have been already sincerely provided by Tomomi Moriyama (TestEtraMatchingUnder_259_R14_G12_B10.csv from https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/72534 ) all you need is just join it with another csv... Ok let me do it." }, { "id": 433039, "postDate": "2018-12-04T15:42:35.720Z", "content": "Hi Alexander - thanks for bringing this up! The host and I are aware of the presence / impact of rare classes here. I'll be doing what I can to mitigate the impact - there will be some shifts made outside of the affected samples.
", "rawMarkdown": "Hi Alexander - thanks for bringing this up! The host and I are aware of the presence / impact of rare classes here. I'll be doing what I can to mitigate the impact - there will be some shifts made outside of the affected samples." }, { "id": 433601, "postDate": "2018-12-05T08:06:14.010Z", "content": "I guess you have done these shifts because I have not trained yet with the new data leak and there's been a huge jump with my LB.
", "rawMarkdown": "I guess you have done these shifts because I have not trained yet with the new data leak and there's been a huge jump with my LB.", "votes": 1 } ] }, { "id": 432301, "postDate": "2018-12-03T17:44:01.497Z", "content": "This is so annoying, another leak of the solutions to the test set. So it looks like getting a high score is about finding the leaked test set data rather than building a good model.\nI agree with some of the suggestions for making this extra data public from someone who already has it. The organizers could then limit external data to this extra data. It would also need to be done soon so that we have time to train on the new data.
", "rawMarkdown": "This is so annoying, another leak of the solutions to the test set. So it looks like getting a high score is about finding the leaked test set data rather than building a good model.\nI agree with some of the suggestions for making this extra data public from someone who already has it. The organizers could then limit external data to this extra data. It would also need to be done soon so that we have time to train on the new data." } ], "comments": [ { "id": 433361, "author_name": "Guanshuo Xu", "author_url": "", "post_date": "2018-12-05T02:09:09.363000", "content": "It's a mistake to use competition participants' findings (the leaks) for test set update. Just ban the external data.
", "votes": 21, "replies": [ { "id": 433971, "author_name": "Dmytro Panchenko", "author_url": "", "post_date": "2018-12-05T17:54:39.610000", "content": "Lots of people have already spent lots of time on the external data, so it would be unfair to ban it now.
", "votes": -2, "replies": [] } ] }, { "id": 433019, "author_name": "Phil Culliton", "author_url": "", "post_date": "2018-12-04T15:12:24.667000", "content": "Hi all - thanks for bringing this to my attention. The samples in question will no longer be included in scoring. I'll be making that change (and rescoring / resetting the leaderboard) today.
\n\nMark, @ldm314 and @tomomimoriyama, I'll be in touch with you via DM. I appreciate your efforts on this.
", "votes": 12, "replies": [ { "id": 433021, "author_name": "Dmytro Panchenko", "author_url": "", "post_date": "2018-12-04T15:16:38.683000", "content": "Thank you!
\n\nCould you please provide a full list of leaked test IDs as well as their classes after rescoring?
", "votes": 2, "replies": [] }, { "id": 433022, "author_name": "DStjhb", "author_url": "", "post_date": "2018-12-04T15:17:48.313000", "content": "Thank you Phil!
", "votes": 0, "replies": [] }, { "id": 433031, "author_name": "TomomiMoriyama", "author_url": "", "post_date": "2018-12-04T15:32:10.070000", "content": "Thank you!
", "votes": 0, "replies": [] }, { "id": 433062, "author_name": "Antoine", "author_url": "", "post_date": "2018-12-04T16:08:07.513000", "content": "Hi @philculliton,
\n\nFirst thanks for your quick reaction on this.
\n\nIs the aim to simply remove those samples ? Or replace them with images that are not part of the public data ?
\n\nI'm afraid that removing them will make the distribution even more imbalanced and increase the instability of the f1 score.
\n\nFor instance for class 27, we have only 11 samples in the training set for 31072 images. By removing those 259 samples you'd remove 5 samples with class 27 from 11702 samples. So it's probably a very important % of this class, if not the whole class.
", "votes": 2, "replies": [] }, { "id": 433091, "author_name": "Robert", "author_url": "", "post_date": "2018-12-04T17:22:17.183000", "content": "Thank you Phil.
", "votes": 0, "replies": [] }, { "id": 433186, "author_name": "Brian", "author_url": "", "post_date": "2018-12-04T20:13:34.483000", "content": "Thanks!
", "votes": 0, "replies": [] }, { "id": 433348, "author_name": "pete", "author_url": "", "post_date": "2018-12-05T01:48:00.893000", "content": "Looks like the leaderboard has not been reset yet - is that true? I guess the comments here mean that new submissions ARE being reset?
", "votes": 0, "replies": [] }, { "id": 433372, "author_name": "Tim H", "author_url": "", "post_date": "2018-12-05T02:20:06.457000", "content": "I just submitted one of my earlier submissions as a test and got the same score as before, so new submissions have not been reset as yet.
\n\nIncidentally my latest model which got 0.595 on this un-rescored LB uses the external HPA data but does not do anything special with the offending images. It will be interesting to see the difference once the leaderboard has been re-scored. I would have assumed that re-scoring will cause people that made special use of those images would see their scores drop, but based on Brian's comment below, maybe that will not always be the case!
", "votes": 0, "replies": [] }, { "id": 433394, "author_name": "pete", "author_url": "", "post_date": "2018-12-05T02:58:16.230000", "content": "looks ike something's happening to the LB now. In progress.
", "votes": 0, "replies": [] }, { "id": 433481, "author_name": "David J. Slate", "author_url": "", "post_date": "2018-12-05T05:53:13.037000", "content": "Thanks Phil for your prompt attention to this problem.\nAre you also going to delete the leaked test images from test.zip and test_full_size.7z?
", "votes": 0, "replies": [] }, { "id": 433981, "author_name": "Phil Culliton", "author_url": "", "post_date": "2018-12-05T18:10:51.790000", "content": "Hi @dslate -
\n\nGood question! I won't be removing the leaked test images. All current competitors have already downloaded them, so this would primarily negatively impact later competitors. Removing them from scoring mitigates the effects of the leak just as readily, without asking later competitors to go without information / images that everyone else had.
", "votes": 0, "replies": [] } ] }, { "id": 433234, "author_name": "Brian", "author_url": "", "post_date": "2018-12-04T21:48:41.850000", "content": "Here is the 126 I've been using, this time with the image IDS.
", "votes": 9, "replies": [] }, { "id": 431818, "author_name": "Brian", "author_url": "", "post_date": "2018-12-03T00:14:07.397000", "content": "My proposal to even out the playing field is to take all of the v18 HPA and include it here with the training data. Remove the leaked images from the test set. I found and use ~125 matches and @tomomimoriyama found over twice that. With the test set being 11k images, removing the overlap would leave plenty to test with.
", "votes": 7, "replies": [ { "id": 431976, "author_name": "Mark Worrall", "author_url": "", "post_date": "2018-12-03T08:03:11.823000", "content": "Yeah, or just announce they aren't part of the private LB in which case using for public LB will bias your results. This of course relies on them being able to reliably identify the true extent of the overlap/leak.
", "votes": 1, "replies": [] }, { "id": 432107, "author_name": "bluetrain", "author_url": "", "post_date": "2018-12-03T12:24:11.930000", "content": "I agree with Brian proposal concerning making HPA data available.
\n\nFor the leakage, I don't think the organizers will take any action on that, because the leakage is not severe and the vast majority of the test set is not affected. This expectation is based on the few competitions \"with leak\" that I took part in. For example, in Santander the leakage was much more evident and the organizers did not do anything about it. However, someone shared part of the leakage in kernels, but the full set of leaky rows of the test set was never released and only the top of the LB managed to exploit it 100%.
", "votes": 0, "replies": [] }, { "id": 432140, "author_name": "Mark Worrall", "author_url": "", "post_date": "2018-12-03T13:04:45.983000", "content": "Jumping from 0.528 to 0.588 because of the leak is severe in my view.
\n\nI also don't think adding the HPA data to the training data really levels the playing field - it favours even more those with massive compute power.
", "votes": 0, "replies": [] }, { "id": 432251, "author_name": "bluetrain", "author_url": "", "post_date": "2018-12-03T16:01:10.893000", "content": "Regarding the jump in the LB: I may agree with you (what I wrote before is what I think it's the reasoning of the organizers, not necessarily mine...) .
\n\nFor the second point, some personal considerations:
\n - Kaggle is biased towards competitors with large computing power and there's little or nothing we can do about it .
\n - Making the additional data available to everyone is the only thing that can be done to level the field again. External data has been announced (by Brian) on the public thread a month ago and no one from the Kaggle team has replied that it was prohibited. If they banned it now, it would be very unfair towards people who have built their models and spent their submissions using that data (N.B.: it's not me, I am not using HPA data so far).
\n - More data instances is, technically speaking, more \"democratic\" than higher resolution data. You could train a model in several steps on different Kaggle kernels, by re-loading the weights from the previous step. On the other hand, higher resolution images require gpu with larger memory, which are very expensive and clearly not available to everyone.
Now, my opinion on what happened in this competition:
\n- it was a mistake of the organizers to make higher resolution images available, as it increases the \"technology gap\" a lot;
\n- the HPA dataset should be made available by someone who owns it - better if in agreement with the organizers, so we are sure that really all the available external data is included;
\n- we should not blame on people who discovered and used the leak: this is just fine according to the rules, as they announced publicly the dataset they were using
Hi Chase the Trane (@stecasasso),
\n\nI agree with nearly all of what you said but think you are slightly conflating two points.
\n\nI don't think it's an issue for people to use the HPA data (and don't want it banned) - we just need to make sure that the HPA data doesn't contain any of the answers to the live competition.
\n\nMark
", "votes": 2, "replies": [] }, { "id": 432517, "author_name": "David J. Slate", "author_url": "", "post_date": "2018-12-04T01:39:40.273000", "content": "Hi Mark Worrall,\nI agree with you that people should be able to use the HPA data but that data shouldn't contain any leaked labels for the competition test set. As far as I can see, the only way to accomplish this would be to find all the leaks and remove their images from the test set, both public and private. This has the downside that every submission will have to be rescored and the LB recomputed, but with over a month to go until the contest ends this may be the best solution. The organizers would have to decide to do this pretty soon, however.
", "votes": 5, "replies": [] }, { "id": 432810, "author_name": "DStjhb", "author_url": "", "post_date": "2018-12-04T10:40:37.930000", "content": "Even if no other action is taken, we need an urgent examination of the private test set to find out just how contaminated it is. 126 images is only >1% of the public test. For all we know the percentage is much higher in the unreleased data. I would really like some reassurance that contamination is really only the minor issue that it appears to be at present time.
", "votes": 0, "replies": [] } ] }, { "id": 433830, "author_name": "Phil Culliton", "author_url": "", "post_date": "2018-12-05T14:35:58.963000", "content": "Hey all -
\n\nI updated and reset the LB last night. Here's what changed:
\n\n1) The majority of the impacted samples are no longer included in scoring.
\n\n2) A small number of samples with very rare labels were not removed from scoring, but were shifted to the public leaderboard.
\n\nBoth of these moves will affect public LB scores, especially for less common labels. If you're concerned about your score changing after this fix was implemented, the fix did impact everyone's scores - whether they were using the leak or not. Any shift on the leaderboard, whether ignoring samples or moving them between public and private LBs, will affect scores. I do understand the confusion expressed about scores changing: I sincerely apologize and hope this clears things up!
\n\nThanks again to everyone for your feedback and help! It's much appreciated.
", "votes": 7, "replies": [ { "id": 433899, "author_name": "Tilii", "author_url": "", "post_date": "2018-12-05T16:21:31.900000", "content": "@philculliton Thanks for the info. I think this should be a separate topic and pinned to the top of discussion board, as not everyone is subscribed to the whole forum.
", "votes": 5, "replies": [] }, { "id": 433917, "author_name": "pete", "author_url": "", "post_date": "2018-12-05T16:45:29.167000", "content": "I guess that rare classes that were already on the public LB were left there? And more shifted from the private? It’s hard to see how that would lower public scores of those using the leak. It might generate a massive shakeup at the end, though. Wouldnt it have been better to just remove leaks entirely, even from rare classes? And if some classes were rendered 1/10000 or something, remove the classes.
", "votes": 0, "replies": [] }, { "id": 433920, "author_name": "Mark Worrall", "author_url": "", "post_date": "2018-12-05T16:49:30.530000", "content": "Yeah, so using the file can cover for a model missing some of the rare classes as these are kept in public LB...but this means private LB will likely be very different.
", "votes": 0, "replies": [] }, { "id": 433946, "author_name": "bluetrain", "author_url": "", "post_date": "2018-12-05T17:20:24.307000", "content": "There going to be a nice shakeup in this competition :)
", "votes": 1, "replies": [] }, { "id": 433972, "author_name": "Dmytro Panchenko", "author_url": "", "post_date": "2018-12-05T17:58:50.057000", "content": "Thank you for the update.\nCould you please confirm that there is no leakage in the private set?\nCould you also publish a list of leaked images with their classes to even the odds?
", "votes": 0, "replies": [] }, { "id": 433975, "author_name": "Phil Culliton", "author_url": "", "post_date": "2018-12-05T18:03:34.657000", "content": "Hi Dmytro -
\n\nI can confirm that there is no leakage in the private set. Re: posting a list of the leaked images... I'm considering it. Because some of the images aren't simply being ignored, that makes the impact of releasing the list more complex. I may release the list of the ignored images.
", "votes": 2, "replies": [] }, { "id": 433986, "author_name": "pete", "author_url": "", "post_date": "2018-12-05T18:21:10.297000", "content": "Here's some random chatter from someone without much experience in DL or Kaggle:
\n\nI'll incorporate the leak to get the 0.05 public LB asap, so can brag to my wife. But the question is now: how do I avoid the shakeup?
\n\nPoint 2 might not be a huge killer because we will be in the prediction stage so the model will have been trained with non-decimated rare classes. However, the leak may reduce the impetus to deal with rare classes, thus deteriorating the effectiveness of the winning models.
\n\nSo the strategy I guess will be to incorporate the leak (altering submission file only) for bragging rights in the next month (and to see exactly where you are in the competition) and develop your model just like nothing happened.
\n\nOr you could copy all leak data (rare classes) into your personal training set to help you better train those classes. It won't hurt the public LB as you are replacing the predictions of those samples anyway.
\n\nRegarding point 1, for completeness, I wonder how many of the leaks have been found. It appears that the organizers are just using the results contributed on these threads - not doing any checks. Perhaps there are many more leaks with very small differences (say by different resampling/antiallias, different versions of photo(?)). And it's not easy to compare a hundred thousand huge images - hashing has been used - how accurate is that in this context? Is there a better way that runs in finite time?
\n\nDefinitely a footnote in the Nature paper.
", "votes": 2, "replies": [] }, { "id": 434012, "author_name": "Phil Culliton", "author_url": "", "post_date": "2018-12-05T19:08:29.897000", "content": "Hi pete -
\n\nThanks for the feedback - I appreciate the thought everyone is putting into this!
\n\nRe: your point 2. The rare classes in question still exist in the private LB. Samples were shifted between the private and public LB to ensure this. Anyone who feels a reduced impetus to deal with rare classes will definitely take a hit on the private LB.
\n\nRe: the leaks... we absolutely did our own checks, and did not use the results contributed on the forums. The forum results were useful because they indicated that people had identified a problem. We know the full extent of the potential leakage based on our own work.
\n\nOn a related note, I should mention that hashing is not 100% accurate in this context, and that some of the \"leaked\" images are not, in fact, leaks (and further, some of the labels assigned them are incorrect). The leak appears to be so effective because some of the real, actual leaks include rare classes that have a high impact on LB score.
\n\nThe people who found the leak did excellent work, but it's very difficult to get completely right. I'd recommend not using the leak in any way.
", "votes": 5, "replies": [] }, { "id": 434100, "author_name": "pete", "author_url": "", "post_date": "2018-12-05T22:10:15.150000", "content": "Phil,
\n\nThanks very much for your comments. My apologies for assuming that you did not look carefully into this - looks like you definitely did. While I'm not fully on board with all of your suggestions, they are very helpful and I appreciate what you have done with this, and your policy of keeping us up to date.
\n\nAlso to those who found the leak and did great work calling attention to it and helped others understand, thank you also. I am super impressed with how fast you understood that complicated website (I still don't really understand it). You improved the competition immeasurably.
", "votes": 2, "replies": [] }, { "id": 434463, "author_name": "Fionnán Alt", "author_url": "", "post_date": "2018-12-06T12:43:31.860000", "content": "Hi Phil - thanks for the info. Do we need to download the test set again if samples were shifted between private and public LB?
", "votes": 0, "replies": [] }, { "id": 434472, "author_name": "bluetrain", "author_url": "", "post_date": "2018-12-06T13:08:17.340000", "content": "@Fionnán Alt: I guess no... \ntest set = public LB + private LB
", "votes": 0, "replies": [] }, { "id": 434473, "author_name": "Phil Culliton", "author_url": "", "post_date": "2018-12-06T13:08:18.310000", "content": "Hi Fionnán,\nGood question! You don't need to download anything new - the shift happened entirely on our end, so you shouldn't have to make any changes.
", "votes": 0, "replies": [] }, { "id": 434501, "author_name": "Fionnán Alt", "author_url": "", "post_date": "2018-12-06T14:08:53.157000", "content": "Thanks Phil.
", "votes": 0, "replies": [] } ] }, { "id": 432969, "author_name": "Aleksandr Kiselev", "author_url": "", "post_date": "2018-12-04T14:26:38.920000", "content": "Test matches found by Tomomi Moriyama with labels from HPA.
", "votes": 5, "replies": [ { "id": 432980, "author_name": "Mark Worrall", "author_url": "", "post_date": "2018-12-04T14:41:25.410000", "content": "Thank you Alexander - I hadn't been through all those files.
\n\nP. S. that just took me from 0.482 -> 0.527 ... wonder how many others in the top 100 are using it already. You might also wish to make that csv more public (i.e. kernel).
", "votes": 0, "replies": [] }, { "id": 433045, "author_name": "Khoi Nguyen", "author_url": "", "post_date": "2018-12-04T15:49:52.583000", "content": "It's funny, using the leak our 0.537 submission using oversampling only got to 0.544, while the 0.507 submission which clearly overfitted to the major classes got to 0.556. I'm glad they decided to take action.
", "votes": 0, "replies": [] }, { "id": 433190, "author_name": "Brian", "author_url": "", "post_date": "2018-12-04T20:20:06.293000", "content": "Amusingly enough if I apply this file, my score goes down from .531 to .528...
", "votes": 1, "replies": [] }, { "id": 433202, "author_name": "Aleksandr Kiselev", "author_url": "", "post_date": "2018-12-04T20:39:31.683000", "content": "@ldm314 do you think @tomomimoriyama is trying to miss lead us? :D
", "votes": -1, "replies": [] }, { "id": 433231, "author_name": "Brian", "author_url": "", "post_date": "2018-12-04T21:47:13.377000", "content": "I don't think so. I came up with a similar list independently after @tomomimoriyama demonstrated that the yellow channels were available. I ended up with the same score drop vs the 126 list I came up with from only RGB.
", "votes": 0, "replies": [] }, { "id": 433461, "author_name": "Valentina Biryukova", "author_url": "", "post_date": "2018-12-05T05:18:30.957000", "content": "It is very upset that I don't use leak but all my scores were dropped :(
", "votes": 0, "replies": [] } ] }, { "id": 433563, "author_name": "Mark Worrall", "author_url": "", "post_date": "2018-12-05T07:24:09.127000", "content": "I don't think the LB has been updated properly (or something weird is going on) as my score has gone from:
\n\n0.482 no leaked data\n0.527 using leaked data\n0.528 post LB update?
", "votes": 6, "replies": [ { "id": 433575, "author_name": "Valentina Biryukova", "author_url": "", "post_date": "2018-12-05T07:39:32.640000", "content": "Yes! I suppose the same! My score without leak has dropped!
", "votes": 0, "replies": [] }, { "id": 433576, "author_name": "SpruceMoose", "author_url": "", "post_date": "2018-12-05T07:40:53.367000", "content": "Definitely something weird going on
\n\n0.525 no leak\n0.561 leak \n0.561 after update
", "votes": 3, "replies": [] }, { "id": 433600, "author_name": "Aleksandr Kiselev", "author_url": "", "post_date": "2018-12-05T08:05:46.760000", "content": "Drop the samples, they said\nIt would be fair, they said
\n\nP.S. no offense :)
", "votes": 5, "replies": [] }, { "id": 433603, "author_name": "Mark Worrall", "author_url": "", "post_date": "2018-12-05T08:08:26.280000", "content": "Haha - no offence taken. I don't think it's clear what's gone on at the moment but the ids from the csv you provided are still being scored in the public LB somehow.
", "votes": 0, "replies": [] }, { "id": 433610, "author_name": "Vladimir Sorokin", "author_url": "", "post_date": "2018-12-05T08:13:51.300000", "content": "Same! Our score without leak has decreased from 0.506 to 0.491 .
", "votes": 0, "replies": [] }, { "id": 433648, "author_name": "", "author_url": "", "post_date": "2018-12-05T09:43:38.643000", "content": "", "votes": 0, "replies": [] }, { "id": 433653, "author_name": "David J. Slate", "author_url": "", "post_date": "2018-12-05T09:49:56.027000", "content": "I did not use the leaked labels, and my highest LB score has dropped from 0.504 to 0.499. But that doesn't necessarily indicate any scoring error by the organizers. It could be that even without the leaks, my model did a good job of predicting the affected images, so without them my score dropped a bit.\nThe LB still looks dubious at the high end, however, since those scores seem to have been hardly affected by the \"reset\" at all.
", "votes": 0, "replies": [] }, { "id": 433697, "author_name": "wchzh", "author_url": "", "post_date": "2018-12-05T11:14:43.027000", "content": "my score also decreased from 0.501 to 0.495.
", "votes": 0, "replies": [] }, { "id": 433756, "author_name": "Guanshuo Xu", "author_url": "", "post_date": "2018-12-05T12:50:26.927000", "content": "0.570 -> 0.553\nWas I using too many leaks? :P
", "votes": 1, "replies": [] }, { "id": 433963, "author_name": "Zhijian Li", "author_url": "", "post_date": "2018-12-05T17:43:55.623000", "content": "haha, I think so.\nmy score dropped from 0.498 to 0.494,\nusing the leaks, it goes to 0.549
", "votes": 1, "replies": [] } ] }, { "id": 434133, "author_name": "Brian", "author_url": "", "post_date": "2018-12-06T00:06:11.560000", "content": "After the leaderboard changes, in general, models submitted with the 126 samples I have posted went down in score. Models using the list from @tomomimoriyama went up in score. What meaning this has, if any, I won't speculate on as some of these posted answers are confirmed to be inaccurate.
\n\nThanks to @philculliton for cleaning thing up and evening out the field.
", "votes": 3, "replies": [] }, { "id": 436658, "author_name": "yunhai", "author_url": "", "post_date": "2018-12-10T17:56:22.613000", "content": "since this challenge is permited to use the external datasets, can anyone provide a complete HPA dataset to download including full HPA image and train.csv? don't think we should cost much time exploring external datasets.
", "votes": 4, "replies": [ { "id": 442604, "author_name": "", "author_url": "", "post_date": "2018-12-20T07:29:07.140000", "content": "", "votes": 3, "replies": [] }, { "id": 442840, "author_name": "", "author_url": "", "post_date": "2018-12-20T15:39:13.173000", "content": "Hi Jarvis, Thank you for these useful information! Are we allowed to use HPA datasets in training a model? Does it not have any implication such as data leakage? Sorry, I didn't check all HPA datasets. Thanks
", "votes": 9, "replies": [] } ] }, { "id": 443202, "author_name": "Maksim Rodin", "author_url": "", "post_date": "2018-12-21T07:55:34.103000", "content": "Hello guys,\nCould anybody clarify, does it make sense to include the leak in the submission now or not?\nThank you!
", "votes": 1, "replies": [ { "id": 443225, "author_name": "pete", "author_url": "", "post_date": "2018-12-21T08:41:46.163000", "content": "The advantage provided by the leak will probably vanish in the private leaderboard but it doesnt hurt to include it if only to see where you stand. The hpa data themselves give an extra edge in addition to any leak advantage. I dont see any reason why this extra edge wont remain in the private leaderboard.
", "votes": 0, "replies": [] } ] }, { "id": 433691, "author_name": "Renan Rodrigues dos Santos", "author_url": "", "post_date": "2018-12-05T10:58:30.890000", "content": "@philculliton After LB rescoring, my leak only submission (no training) scored 0.085, now it is scoring 0.062 (about two rare classes with F1 close to 1.00), so definitely there are still leakage in public LB. Don't know about private LB though.
", "votes": 2, "replies": [] }, { "id": 432185, "author_name": "DStjhb", "author_url": "", "post_date": "2018-12-03T14:25:42.633000", "content": "I concur.
\n\nIf they make this data official and give us an easy zipped download of the HPA in a smaller image size (512p), that would be the best approach. If HPA can afford to have people trawling their servers for tens of thousands of 2048p images from individual urls they can afford to give a direct link to a compressed archive. They must have known that the HPA would be the obvious first point of call for anybody looking for external data.
\n\n@tomomimoriyama and @ldm314 (Brian) have kindly provided access to parsing scripts that allow download of the larger images and label them accordingly. Having to download at mostly 2048px and resize is still a pain. @artemtprv provided a web-scraping kernel to access 'preview' images from the HPA website that are 800x800, but these appear to be either cropped (or at high magnification), and there are far fewer than in the XML urls.
\n\nI am guessing the organisers pre-reduced the training data for the competition to 512px to allow easy download and training for amateur enthusiasts and typical users. Some people don't even have the connection or hdd space to acquire the larger original image set, let alone access to memory and GPUs capable of training models with them properly. Having to download a full size external dataset (even if resized later) to make it onto the top half of the leaderboard is a little extreme.
\n\nThere's still a good amount of time left to fix this so please address it. Even if you just exclude the leaked images, that leaves the option for kind community members to upload the data somewhere independent at a reduced image size with accompanied csv files.
", "votes": 2, "replies": [] }, { "id": 432072, "author_name": "Allan Wang", "author_url": "", "post_date": "2018-12-03T11:14:54.117000", "content": "Cannot agree more with you. The competition needs to focus on methods and skills instead of \"cheating\" to get a high score.
", "votes": 2, "replies": [] }, { "id": 432160, "author_name": "Khoi Nguyen", "author_url": "", "post_date": "2018-12-03T13:46:03.287000", "content": "Wait, what? There is a leak?
", "votes": 0, "replies": [] }, { "id": 433965, "author_name": "Robert", "author_url": "", "post_date": "2018-12-05T17:48:11.510000", "content": "Hmm... based on the scores that people are posting it looks like most of the leaks are still being scored.
", "votes": 0, "replies": [ { "id": 433974, "author_name": "Phil Culliton", "author_url": "", "post_date": "2018-12-05T18:00:19.807000", "content": "Hi Robert -
\n\nWe've changed the public LB exposure for two rare classes, which is having a noticeable effect on public LB scores. An overwhelming majority of the leaked samples are no longer being scored at all.
", "votes": 0, "replies": [] }, { "id": 435873, "author_name": "zymale", "author_url": "", "post_date": "2018-12-09T00:00:58.267000", "content": "Is there any mistake here? Before you adjusted, my score was 0.529. After the adjustment, it was 0.513. I didn't use data leak before, but when I use data leak,now my score is 0.555.
", "votes": 0, "replies": [] }, { "id": 435874, "author_name": "pete", "author_url": "", "post_date": "2018-12-09T00:05:24.707000", "content": "It sounds like reasonable behaviour. I would say it's OK.
", "votes": 0, "replies": [] }, { "id": 435877, "author_name": "zymale", "author_url": "", "post_date": "2018-12-09T00:12:00.187000", "content": "Hi,pete.\nI don't think it's a problem either. I'm mainly concerned about whether there will be a second adjustment, because the adjustment still has a great impact on the score. This may cause some trouble for me to adjust the model.
", "votes": 0, "replies": [] }, { "id": 439847, "author_name": "Wang Xinliang", "author_url": "", "post_date": "2018-12-16T14:09:50.683000", "content": "Yes, I'm agree with you. This will cause trouble to adjust the model.
", "votes": 0, "replies": [] } ] }, { "id": 433469, "author_name": "phalanx", "author_url": "", "post_date": "2018-12-05T05:30:00.543000", "content": "Leaderboard have been reseted, but leak is applied to public LB.
\n* no leak submission: public LB. 0.496
\n* no leak submission + leak: public LB. 0.502
\nThe effective of leak is small, but public LB is improved.
From my perspective they didn't fix anything much.
\n\nBefore reset + leak: 0.556
After reset + leak: 0.555
Thanks Khoi. I think so too.
\nIt is strange that the scores of many participants have not changed.
I want to point out that some classes are very rare in train and test set, so removing them from test set will reduce them even more and score will be affected a lot by correct/incorrect predictions for that classes. IMHO it will cause a big shakeup later.
", "votes": 0, "replies": [ { "id": 432892, "author_name": "Mark Worrall", "author_url": "", "post_date": "2018-12-04T12:43:51.427000", "content": "Well this is only the case if the HPA data contains the answers to some of the rare classes - that's the only case where we need to remove from test. The flip side of this is that the leak is even more serious as the impact is large.
\n\nTo be honest, I'm not sure why we are worrying about LB stability when it's just been revealed part of the answers are externally available - this compromises the integrity of the competition which has a metric defined precisely to try to encourage rare class predictions.
", "votes": 0, "replies": [] }, { "id": 432908, "author_name": "Aleksandr Kiselev", "author_url": "", "post_date": "2018-12-04T13:07:35.870000", "content": "\"Leaked\" data have been published already here https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/72534 (you can download images like described here https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/69984 ) and available to everyone. So it should be +- ok if everyone use same labels for that samples, as it doesn't change overall rating. So yes, I'm worrying about stability more.
", "votes": 1, "replies": [] }, { "id": 432915, "author_name": "DStjhb", "author_url": "", "post_date": "2018-12-04T13:12:59.543000", "content": "The leaked test examples have a relative majority of class 16 (Cytokinetic bridge), which is a pretty rare class overall, so it is a bit worrying. https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/69984#423736
", "votes": 0, "replies": [] }, { "id": 432919, "author_name": "Mark Worrall", "author_url": "", "post_date": "2018-12-04T13:20:49.757000", "content": "Part of the outcome of this competition is meant to be a Nature Methods paper - hardly worth much if the metric is biased as some people found the answers online.
\nThis is exacerbated as the extent isn't confirmed officially (it's being left to drag out) though I agree if the organisers can confirm the extent and either give a csv with those answers in or exclude them I think that is fine.
\nJust to be clear: in my view asking people to parse and download massive amounts of extra data just to plug a leak is a weak fix. You can worry all you like about stability but if the integrity is compromised it undermines the whole competition.
\nMark
\nP. S. Alexander Kiselev: if you truly want to help out why not provide a csv with all the images from the test set and their labels from HPA if you have them? We can all get back on with the problem at hand then.
", "votes": 0, "replies": [] }, { "id": 432968, "author_name": "Aleksandr Kiselev", "author_url": "", "post_date": "2018-12-04T14:24:38.563000", "content": "That nice csv file have been already sincerely provided by Tomomi Moriyama (TestEtraMatchingUnder_259_R14_G12_B10.csv from https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/72534 ) all you need is just join it with another csv... Ok let me do it.
", "votes": 0, "replies": [] }, { "id": 433039, "author_name": "Phil Culliton", "author_url": "", "post_date": "2018-12-04T15:42:35.720000", "content": "Hi Alexander - thanks for bringing this up! The host and I are aware of the presence / impact of rare classes here. I'll be doing what I can to mitigate the impact - there will be some shifts made outside of the affected samples.
", "votes": 0, "replies": [] }, { "id": 433601, "author_name": "FlYM", "author_url": "", "post_date": "2018-12-05T08:06:14.010000", "content": "I guess you have done these shifts because I have not trained yet with the new data leak and there's been a huge jump with my LB.
", "votes": 1, "replies": [] } ] }, { "id": 432301, "author_name": "Robert", "author_url": "", "post_date": "2018-12-03T17:44:01.497000", "content": "This is so annoying, another leak of the solutions to the test set. So it looks like getting a high score is about finding the leaked test set data rather than building a good model.\nI agree with some of the suggestions for making this extra data public from someone who already has it. The organizers could then limit external data to this extra data. It would also need to be done soon so that we have time to train on the new data.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "431652": "Hi all,\n\nI realise this might not make me the most popular person on this forum but it's something I think needs to be addressed and kaggle (e.g @philculliton) should provide an answer.\n\nTucked away in a discussion [here][1] is the revelation that the (difficult to access) HPA data has been used to boost @tomomimoriyama (plus I imagine others) to the top of the LB. In all fairness to the few people discussing this topic they have acknowledged it's uncertain as to whether it's allowed or not but I for one would like this explicit and clarified. \n\n**Use of the HPA data**: I'm only going on what little is mentioned but it appears the HPA data is not being used in the sense I understand external data should to be used (e.g. word embeddings) in order to supplement a problem but instead people have basically found the same images in the HPA data as appear in the train and test sets. i.e. they've found some of the answers. I cannot imagine this is what the competition organisers intend when they challenge us to automate *'biomedical image analysis to accelerate the understanding of human cells and disease'.*\n\nFurther, from what I can gather, the HPA data is a pain to access and as pointed out by @stecasasso (Chase the Trane) this sort of external data has been banned in previous competitions. I can't force the kaggle or the organisers to ban it in this case but I can safely say that if a prerequisite for tackling image competitions with huge data is to have to parse even more hard to reach huge data to simply uncover test images it's definitely not what I had in mind when I decided to start this competition and use machine learning for good. \n\nA final point: if I've gotten any of the above mixed up I am happy to amend my post for inaccuracies or clarify misunderstanding. I'd also like to thank those using the HPA data for being honest about finding similar images in it, particularly @ldm314 (Brian) and @tomomimoriyama.\n\nCheers,\n\nMark\n\nP. S. also mentioned [here][2]\n\n\n [1]: https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/70206\n [2]: https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/72534", "433361": "It's a mistake to use competition participants' findings (the leaks) for test set update. Just ban the external data.", "433019": "Hi all - thanks for bringing this to my attention. The samples in question will no longer be included in scoring. I'll be making that change (and rescoring / resetting the leaderboard) today.\n\nMark, @ldm314 and @tomomimoriyama, I'll be in touch with you via DM. I appreciate your efforts on this.", "433234": "Here is the 126 I've been using, this time with the image IDS. ", "431818": "My proposal to even out the playing field is to take all of the v18 HPA and include it here with the training data. Remove the leaked images from the test set. I found and use ~125 matches and @tomomimoriyama found over twice that. With the test set being 11k images, removing the overlap would leave plenty to test with.", "433830": "Hey all -\n\nI updated and reset the LB last night. Here's what changed:\n\n1) The majority of the impacted samples are no longer included in scoring.\n\n2) A small number of samples with very rare labels were not removed from scoring, but were shifted to the public leaderboard.\n\nBoth of these moves will affect public LB scores, especially for less common labels. If you're concerned about your score changing after this fix was implemented, the fix did impact everyone's scores - whether they were using the leak or not. Any shift on the leaderboard, whether ignoring samples or moving them between public and private LBs, will affect scores. I do understand the confusion expressed about scores changing: I sincerely apologize and hope this clears things up!\n\nThanks again to everyone for your feedback and help! It's much appreciated.", "432969": "Test matches found by Tomomi Moriyama with labels from HPA.", "433563": "I don't think the LB has been updated properly (or something weird is going on) as my score has gone from:\n\n0.482 no leaked data\n0.527 using leaked data\n0.528 post LB update?", "434133": "After the leaderboard changes, in general, models submitted with the 126 samples I have posted went down in score. Models using the list from @tomomimoriyama went up in score. What meaning this has, if any, I won't speculate on as some of these posted answers are confirmed to be inaccurate.\n\nThanks to @philculliton for cleaning thing up and evening out the field.\n\n", "436658": "since this challenge is permited to use the external datasets, can anyone provide a complete HPA dataset to download including full HPA image and train.csv? don't think we should cost much time exploring external datasets. ", "443202": "Hello guys,\nCould anybody clarify, does it make sense to include the leak in the submission now or not?\nThank you!", "433691": "@philculliton After LB rescoring, my leak only submission (no training) scored 0.085, now it is scoring 0.062 (about two rare classes with F1 close to 1.00), so definitely there are still leakage in public LB. Don't know about private LB though.", "432185": "I concur.\n\nIf they make this data official and give us an easy zipped download of the HPA in a smaller image size (512p), that would be the best approach. If HPA can afford to have people trawling their servers for tens of thousands of 2048p images from individual urls they can afford to give a direct link to a compressed archive. They must have known that the HPA would be the obvious first point of call for anybody looking for external data.\n\n@tomomimoriyama and @ldm314 (Brian) have kindly provided access to parsing scripts that allow download of the larger images and label them accordingly. Having to download at mostly 2048px and resize is still a pain. @artemtprv provided a web-scraping kernel to access 'preview' images from the HPA website that are 800x800, but these appear to be either cropped (or at high magnification), and there are far fewer than in the XML urls.\n\nI am guessing the organisers pre-reduced the training data for the competition to 512px to allow easy download and training for amateur enthusiasts and typical users. Some people don't even have the connection or hdd space to acquire the larger original image set, let alone access to memory and GPUs capable of training models with them properly. Having to download a full size external dataset (even if resized later) to make it onto the top half of the leaderboard is a little extreme.\n\nThere's still a good amount of time left to fix this so please address it. Even if you just exclude the leaked images, that leaves the option for kind community members to upload the data somewhere independent at a reduced image size with accompanied csv files.", "432072": "Cannot agree more with you. The competition needs to focus on methods and skills instead of \"cheating\" to get a high score.", "432160": "Wait, what? There is a leak?", "433965": "Hmm... based on the scores that people are posting it looks like most of the leaks are still being scored.", "433469": "Leaderboard have been reseted, but leak is applied to public LB.