What are people's thoughts on where the leaderboard might end up in Stage 2 when we haven't got a dataset with a huge amount of information leakage?
\n\nIs there much information that we can take from the current leaderboard? i.e. what proportion of the top 100 teams are explicitly using the information leakage in their top submissions?
\n\nI'm wondering how to gauge what a 'good' model is, relatively speaking, at this point. I'm guessing that a model yielding a score between 0.7 - 0.8 without using any info leaks could put you in the top 50 when a cleaner dataset comes through... Although who knows!
\n\nInterested to hear what other people are making of this. Are any of the top performing teams willing to divulge whether they are using the leakage in their scores or not? FWIW my ~0.7 score does have leaked info in it.
", "messages": [ { "id": "183587", "postDate": "05/18/2017 17:07:15", "content": "What are people's thoughts on where the leaderboard might end up in Stage 2 when we haven't got a dataset with a huge amount of information leakage?
\n\nIs there much information that we can take from the current leaderboard? i.e. what proportion of the top 100 teams are explicitly using the information leakage in their top submissions?
\n\nI'm wondering how to gauge what a 'good' model is, relatively speaking, at this point. I'm guessing that a model yielding a score between 0.7 - 0.8 without using any info leaks could put you in the top 50 when a cleaner dataset comes through... Although who knows!
\n\nInterested to hear what other people are making of this. Are any of the top performing teams willing to divulge whether they are using the leakage in their scores or not? FWIW my ~0.7 score does have leaked info in it.
", "rawMarkdown": "What are people's thoughts on where the leaderboard might end up in Stage 2 when we haven't got a dataset with a huge amount of information leakage? \n\nIs there much information that we can take from the current leaderboard? i.e. what proportion of the top 100 teams are explicitly using the information leakage in their top submissions? \n\nI'm wondering how to gauge what a 'good' model is, relatively speaking, at this point. I'm guessing that a model yielding a score between 0.7 - 0.8 without using any info leaks could put you in the top 50 when a cleaner dataset comes through... Although who knows!\n\nInterested to hear what other people are making of this. Are any of the top performing teams willing to divulge whether they are using the leakage in their scores or not? FWIW my ~0.7 score does have leaked info in it.", "votes": null }, { "id": "183596", "postDate": "05/18/2017 17:30:29", "content": "The same women occur in the train and stage 1 test data, also, numbers of images in particular classes are similarly distributed in both datasets. What do you do to not use this leak?
", "rawMarkdown": "The same women occur in the train and stage 1 test data, also, numbers of images in particular classes are similarly distributed in both datasets. What do you do to not use this leak?", "votes": null }, { "id": "183605", "postDate": "05/18/2017 18:10:04", "content": "I was just asked what I meant explicitly by information leakage - I meant that you can improve your score by taking your submission and changing the classification for the images that are matched to the additional or test sets to 1-0-0 for a Type 1 match, etc. Obviously this isn't of any use when it comes to Stage 2, it just makes it more difficult to gauge what a good score is at the moment.
", "rawMarkdown": "I was just asked what I meant explicitly by information leakage - I meant that you can improve your score by taking your submission and changing the classification for the images that are matched to the additional or test sets to 1-0-0 for a Type 1 match, etc. Obviously this isn't of any use when it comes to Stage 2, it just makes it more difficult to gauge what a good score is at the moment.", "votes": null }, { "id": "183607", "postDate": "05/18/2017 18:13:02", "content": "I'm not sure you can avoid it completely, but I ended up clustering the images in the train/additional sets and picked one image per cluster to use for fitting / validation. The aim was to try to reduce the amount of in-sample bias. Not ideal, but I'm not sure what other options there are...
", "rawMarkdown": "I'm not sure you can avoid it completely, but I ended up clustering the images in the train/additional sets and picked one image per cluster to use for fitting / validation. The aim was to try to reduce the amount of in-sample bias. Not ideal, but I'm not sure what other options there are...", "votes": null }, { "id": "183645", "postDate": "05/18/2017 20:21:16", "content": "Interesting!! good to know :) ...My score is based on zero leakage, and I have been struggling to catch the top teams...I am suspicious of the 0.47700 score by GRX but would not risk not trying to catch up, can't tell if that score is based on leakage or not.
", "rawMarkdown": "Interesting!! good to know :) ...My score is based on zero leakage, and I have been struggling to catch the top teams...I am suspicious of the 0.47700 score by GRX but would not risk not trying to catch up, can't tell if that score is based on leakage or not.", "votes": null }, { "id": "183665", "postDate": "05/18/2017 21:53:35", "content": "Is this the one you guys are talking about?https://www.kaggle.com/c/intel-mobileodt-cervical-cancer-screening/discussion/32427
", "rawMarkdown": "Is this the one you guys are talking about?https://www.kaggle.com/c/intel-mobileodt-cervical-cancer-screening/discussion/32427", "votes": null }, { "id": "183669", "postDate": "05/18/2017 22:01:42", "content": "You can only avoid leakage by not touching the additional images at all, and that's probably going to hurt your performance too. Just don't try to handle the leakage yourselves and know that your Stage 2 score is going to be lower than the Stage 1 one.
", "rawMarkdown": "You can only avoid leakage by not touching the additional images at all, and that's probably going to hurt your performance too. Just don't try to handle the leakage yourselves and know that your Stage 2 score is going to be lower than the Stage 1 one.", "votes": null }, { "id": "183671", "postDate": "05/18/2017 22:07:42", "content": "Surprisingly, I see a limited impact of the 'leakage'.
\n\nThe models trained on the full (train + additional) datasets, will score around 0.65-0.7 on my validation dataset, while those trained on the train + additional which are not similar / duplicates of my validation set, will score in the 0.7-0.75. There is a difference, but it feels small, compared to the importance of the information (ie, extremely similar examples).
", "rawMarkdown": "Surprisingly, I see a limited impact of the 'leakage'.\n\nThe models trained on the full (train + additional) datasets, will score around 0.65-0.7 on my validation dataset, while those trained on the train + additional which are not similar / duplicates of my validation set, will score in the 0.7-0.75. There is a difference, but it feels small, compared to the importance of the information (ie, extremely similar examples).", "votes": null }, { "id": "183673", "postDate": "05/18/2017 22:30:59", "content": "I think it depends on how weak your underlying models are. For example, the score that is showing 0.7 for me is actually a rubbish model that I just added the leakage to - it gave about a 0.2 improvement. Since then I have kind of abandoned the leaderboard and am trusting local cv. I don't think I could get anywhere near a 0.2 improvement if I added the leakage to the strongest set of models that I have currently.... Which leads me to believe that the leaderboard could be very misleading - a mixture of very good models with very weak ones which are adjusted. It makes it hard to decipher where individual teams stand.
", "rawMarkdown": "I think it depends on how weak your underlying models are. For example, the score that is showing 0.7 for me is actually a rubbish model that I just added the leakage to - it gave about a 0.2 improvement. Since then I have kind of abandoned the leaderboard and am trusting local cv. I don't think I could get anywhere near a 0.2 improvement if I added the leakage to the strongest set of models that I have currently.... Which leads me to believe that the leaderboard could be very misleading - a mixture of very good models with very weak ones which are adjusted. It makes it hard to decipher where individual teams stand.", "votes": null }, { "id": "183884", "postDate": "05/19/2017 15:56:37", "content": "Interesting.
", "rawMarkdown": "Interesting.", "votes": null }, { "id": "184562", "postDate": "05/22/2017 11:03:08", "content": "I just joined the competition around two weeks ago and saw there was an earlier data set which was removed or updated... was there leak in the earlier data set ? or the leak still exists in the current data set ? I see about 20 or so images in train&additional are the same or very similar test...
", "rawMarkdown": "I just joined the competition around two weeks ago and saw there was an earlier data set which was removed or updated... was there leak in the earlier data set ? or the leak still exists in the current data set ? I see about 20 or so images in train&additional are the same or very similar test...", "votes": null }, { "id": "184595", "postDate": "05/22/2017 13:07:10", "content": "I think the earlier additional dataset contained images that were misclassified (people identified images that were the same as those in the train set, but that had different labels). These were corrected in the current additional dataset. The leak still exists. See https://www.kaggle.com/c/intel-mobileodt-cervical-cancer-screening/discussion/32427
", "rawMarkdown": "I think the earlier additional dataset contained images that were misclassified (people identified images that were the same as those in the train set, but that had different labels). These were corrected in the current additional dataset. The leak still exists. See https://www.kaggle.com/c/intel-mobileodt-cervical-cancer-screening/discussion/32427", "votes": null }, { "id": "185189", "postDate": "05/24/2017 12:10:59", "content": "But @Alchemist, I used your test_duplicates_v3.csv to remove images from the additional set and I see quite an impact of leakage!
\n\nI can certainly say that the public leaderboard score goes down as the number of duplicate images are removed from the additional set. The impact is profound, with a loss of as high as 0.1 in log loss score on the leaderboard for the same architecture/model. But the local score improves (most of the times)
\n\nAlso, for some reason validation log-loss really fluctuates with the subset of data used for train/val. I am not able to conjure up why this is happening, but my guess would be that additional images are the culprit.
\n\nIt seems that the more \"theoretically and pragmatic\" of an approach I think of, the more my performance drops :D. Either I have lost my mind or this competition really wants to give me a run for the money!
", "rawMarkdown": "But @Alchemist, I used your test_duplicates_v3.csv to remove images from the additional set and I see quite an impact of leakage!\n\nI can certainly say that the public leaderboard score goes down as the number of duplicate images are removed from the additional set. The impact is **profound**, with a loss of as high as 0.1 in log loss score on the leaderboard for the same architecture/model. But the local score improves (most of the times)\n\nAlso, for some reason validation log-loss really fluctuates with the subset of data used for train/val. I am not able to conjure up why this is happening, but my guess would be that additional images are the culprit.\n\nIt seems that the more \"theoretically and pragmatic\" of an approach I think of, the more my performance drops :D. Either I have lost my mind or this competition really wants to give me a run for the money!", "votes": null }, { "id": "185274", "postDate": "05/24/2017 18:34:06", "content": "@SarthakYadav - that's roughly what I see also (0.1 log-loss loss).
\n\nHowever, I would expect the bonus from having duplicate images much more important than that. I thought that at one point the net will just overfit perfectly one those duplicates / near duplicates but it doesn't seem to happen
", "rawMarkdown": "SarthakYadav - that's roughly what I see also (0.1 log-loss loss). \n\nHowever, I would expect the bonus from having duplicate images much more important than that. I thought that at one point the net will just overfit perfectly one those duplicates / near duplicates but it doesn't seem to happen", "votes": null }, { "id": "186639", "postDate": "05/28/2017 19:44:27", "content": "I can \"attest\" to the fact that my top submission has used the massive amount of leakage present in the data. I don't know what others are doing, but I haven't improved on the lb score for a long time now. I reached my current best score after around 2-3 weeks into the competition.\nAnd it has been downhill from there. The more \"refined\" of an approach I try to apply, that seems to be theoretically correct to me, the larger the drop in my performance.
\n\nI'll continue to trust myself on this one. From reaching as high as rank 3 to dropping down to the 40's, doesn't matter which way the result goes, there will be a lot of learning involved on this one!
", "rawMarkdown": "I can \"attest\" to the fact that my top submission has used the massive amount of leakage present in the data. I don't know what others are doing, but I haven't improved on the lb score for a long time now. I reached my current best score after around 2-3 weeks into the competition.\nAnd it has been downhill from there. The more \"refined\" of an approach I try to apply, that seems to be theoretically correct to me, the larger the drop in my performance.\n\nI'll continue to trust myself on this one. From reaching as high as rank 3 to dropping down to the 40's, doesn't matter which way the result goes, there will be a lot of learning involved on this one!", "votes": null } ], "comments": [ { "id": 183596, "author_name": "buabua", "author_url": "", "post_date": "05/18/2017 17:30:29", "content": "The same women occur in the train and stage 1 test data, also, numbers of images in particular classes are similarly distributed in both datasets. What do you do to not use this leak?
", "votes": null, "replies": [ { "id": 183607, "author_name": "fergusoci", "author_url": "", "post_date": "05/18/2017 18:13:02", "content": "I'm not sure you can avoid it completely, but I ended up clustering the images in the train/additional sets and picked one image per cluster to use for fitting / validation. The aim was to try to reduce the amount of in-sample bias. Not ideal, but I'm not sure what other options there are...
", "votes": null, "replies": [] } ] }, { "id": 183605, "author_name": "fergusoci", "author_url": "", "post_date": "05/18/2017 18:10:04", "content": "I was just asked what I meant explicitly by information leakage - I meant that you can improve your score by taking your submission and changing the classification for the images that are matched to the additional or test sets to 1-0-0 for a Type 1 match, etc. Obviously this isn't of any use when it comes to Stage 2, it just makes it more difficult to gauge what a good score is at the moment.
", "votes": null, "replies": [] }, { "id": 183645, "author_name": "godaibo", "author_url": "", "post_date": "05/18/2017 20:21:16", "content": "Interesting!! good to know :) ...My score is based on zero leakage, and I have been struggling to catch the top teams...I am suspicious of the 0.47700 score by GRX but would not risk not trying to catch up, can't tell if that score is based on leakage or not.
", "votes": null, "replies": [ { "id": 184562, "author_name": "darraghdog", "author_url": "", "post_date": "05/22/2017 11:03:08", "content": "I just joined the competition around two weeks ago and saw there was an earlier data set which was removed or updated... was there leak in the earlier data set ? or the leak still exists in the current data set ? I see about 20 or so images in train&additional are the same or very similar test...
", "votes": null, "replies": [] }, { "id": 184595, "author_name": "fergusoci", "author_url": "", "post_date": "05/22/2017 13:07:10", "content": "I think the earlier additional dataset contained images that were misclassified (people identified images that were the same as those in the train set, but that had different labels). These were corrected in the current additional dataset. The leak still exists. See https://www.kaggle.com/c/intel-mobileodt-cervical-cancer-screening/discussion/32427
", "votes": null, "replies": [] } ] }, { "id": 183665, "author_name": "chuijh", "author_url": "", "post_date": "05/18/2017 21:53:35", "content": "Is this the one you guys are talking about?https://www.kaggle.com/c/intel-mobileodt-cervical-cancer-screening/discussion/32427
", "votes": null, "replies": [] }, { "id": 183669, "author_name": "mumech", "author_url": "", "post_date": "05/18/2017 22:01:42", "content": "You can only avoid leakage by not touching the additional images at all, and that's probably going to hurt your performance too. Just don't try to handle the leakage yourselves and know that your Stage 2 score is going to be lower than the Stage 1 one.
", "votes": null, "replies": [] }, { "id": 183671, "author_name": "alchemist", "author_url": "", "post_date": "05/18/2017 22:07:42", "content": "Surprisingly, I see a limited impact of the 'leakage'.
\n\nThe models trained on the full (train + additional) datasets, will score around 0.65-0.7 on my validation dataset, while those trained on the train + additional which are not similar / duplicates of my validation set, will score in the 0.7-0.75. There is a difference, but it feels small, compared to the importance of the information (ie, extremely similar examples).
", "votes": null, "replies": [ { "id": 183673, "author_name": "fergusoci", "author_url": "", "post_date": "05/18/2017 22:30:59", "content": "I think it depends on how weak your underlying models are. For example, the score that is showing 0.7 for me is actually a rubbish model that I just added the leakage to - it gave about a 0.2 improvement. Since then I have kind of abandoned the leaderboard and am trusting local cv. I don't think I could get anywhere near a 0.2 improvement if I added the leakage to the strongest set of models that I have currently.... Which leads me to believe that the leaderboard could be very misleading - a mixture of very good models with very weak ones which are adjusted. It makes it hard to decipher where individual teams stand.
", "votes": null, "replies": [] }, { "id": 185189, "author_name": "yadavsarthak", "author_url": "", "post_date": "05/24/2017 12:10:59", "content": "But @Alchemist, I used your test_duplicates_v3.csv to remove images from the additional set and I see quite an impact of leakage!
\n\nI can certainly say that the public leaderboard score goes down as the number of duplicate images are removed from the additional set. The impact is profound, with a loss of as high as 0.1 in log loss score on the leaderboard for the same architecture/model. But the local score improves (most of the times)
\n\nAlso, for some reason validation log-loss really fluctuates with the subset of data used for train/val. I am not able to conjure up why this is happening, but my guess would be that additional images are the culprit.
\n\nIt seems that the more \"theoretically and pragmatic\" of an approach I think of, the more my performance drops :D. Either I have lost my mind or this competition really wants to give me a run for the money!
", "votes": null, "replies": [] }, { "id": 185274, "author_name": "alchemist", "author_url": "", "post_date": "05/24/2017 18:34:06", "content": "@SarthakYadav - that's roughly what I see also (0.1 log-loss loss).
\n\nHowever, I would expect the bonus from having duplicate images much more important than that. I thought that at one point the net will just overfit perfectly one those duplicates / near duplicates but it doesn't seem to happen
", "votes": null, "replies": [] } ] }, { "id": 183884, "author_name": "amrmorsey", "author_url": "", "post_date": "05/19/2017 15:56:37", "content": "Interesting.
", "votes": null, "replies": [] }, { "id": 186639, "author_name": "yadavsarthak", "author_url": "", "post_date": "05/28/2017 19:44:27", "content": "I can \"attest\" to the fact that my top submission has used the massive amount of leakage present in the data. I don't know what others are doing, but I haven't improved on the lb score for a long time now. I reached my current best score after around 2-3 weeks into the competition.\nAnd it has been downhill from there. The more \"refined\" of an approach I try to apply, that seems to be theoretically correct to me, the larger the drop in my performance.
\n\nI'll continue to trust myself on this one. From reaching as high as rank 3 to dropping down to the 40's, doesn't matter which way the result goes, there will be a lot of learning involved on this one!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "183587": "What are people's thoughts on where the leaderboard might end up in Stage 2 when we haven't got a dataset with a huge amount of information leakage? \n\nIs there much information that we can take from the current leaderboard? i.e. what proportion of the top 100 teams are explicitly using the information leakage in their top submissions? \n\nI'm wondering how to gauge what a 'good' model is, relatively speaking, at this point. I'm guessing that a model yielding a score between 0.7 - 0.8 without using any info leaks could put you in the top 50 when a cleaner dataset comes through... Although who knows!\n\nInterested to hear what other people are making of this. Are any of the top performing teams willing to divulge whether they are using the leakage in their scores or not? FWIW my ~0.7 score does have leaked info in it.", "183596": "The same women occur in the train and stage 1 test data, also, numbers of images in particular classes are similarly distributed in both datasets. What do you do to not use this leak?", "183605": "I was just asked what I meant explicitly by information leakage - I meant that you can improve your score by taking your submission and changing the classification for the images that are matched to the additional or test sets to 1-0-0 for a Type 1 match, etc. Obviously this isn't of any use when it comes to Stage 2, it just makes it more difficult to gauge what a good score is at the moment.", "183607": "I'm not sure you can avoid it completely, but I ended up clustering the images in the train/additional sets and picked one image per cluster to use for fitting / validation. The aim was to try to reduce the amount of in-sample bias. Not ideal, but I'm not sure what other options there are...", "183645": "Interesting!! good to know :) ...My score is based on zero leakage, and I have been struggling to catch the top teams...I am suspicious of the 0.47700 score by GRX but would not risk not trying to catch up, can't tell if that score is based on leakage or not.", "183665": "Is this the one you guys are talking about?https://www.kaggle.com/c/intel-mobileodt-cervical-cancer-screening/discussion/32427", "183669": "You can only avoid leakage by not touching the additional images at all, and that's probably going to hurt your performance too. Just don't try to handle the leakage yourselves and know that your Stage 2 score is going to be lower than the Stage 1 one.", "183671": "Surprisingly, I see a limited impact of the 'leakage'.\n\nThe models trained on the full (train + additional) datasets, will score around 0.65-0.7 on my validation dataset, while those trained on the train + additional which are not similar / duplicates of my validation set, will score in the 0.7-0.75. There is a difference, but it feels small, compared to the importance of the information (ie, extremely similar examples).", "183673": "I think it depends on how weak your underlying models are. For example, the score that is showing 0.7 for me is actually a rubbish model that I just added the leakage to - it gave about a 0.2 improvement. Since then I have kind of abandoned the leaderboard and am trusting local cv. I don't think I could get anywhere near a 0.2 improvement if I added the leakage to the strongest set of models that I have currently.... Which leads me to believe that the leaderboard could be very misleading - a mixture of very good models with very weak ones which are adjusted. It makes it hard to decipher where individual teams stand.", "183884": "Interesting.", "184562": "I just joined the competition around two weeks ago and saw there was an earlier data set which was removed or updated... was there leak in the earlier data set ? or the leak still exists in the current data set ? I see about 20 or so images in train&additional are the same or very similar test...", "184595": "I think the earlier additional dataset contained images that were misclassified (people identified images that were the same as those in the train set, but that had different labels). These were corrected in the current additional dataset. The leak still exists. See https://www.kaggle.com/c/intel-mobileodt-cervical-cancer-screening/discussion/32427", "185189": "But @Alchemist, I used your test_duplicates_v3.csv to remove images from the additional set and I see quite an impact of leakage!\n\nI can certainly say that the public leaderboard score goes down as the number of duplicate images are removed from the additional set. The impact is **profound**, with a loss of as high as 0.1 in log loss score on the leaderboard for the same architecture/model. But the local score improves (most of the times)\n\nAlso, for some reason validation log-loss really fluctuates with the subset of data used for train/val. I am not able to conjure up why this is happening, but my guess would be that additional images are the culprit.\n\nIt seems that the more \"theoretically and pragmatic\" of an approach I think of, the more my performance drops :D. Either I have lost my mind or this competition really wants to give me a run for the money!", "185274": "SarthakYadav - that's roughly what I see also (0.1 log-loss loss). \n\nHowever, I would expect the bonus from having duplicate images much more important than that. I thought that at one point the net will just overfit perfectly one those duplicates / near duplicates but it doesn't seem to happen", "186639": "I can \"attest\" to the fact that my top submission has used the massive amount of leakage present in the data. I don't know what others are doing, but I haven't improved on the lb score for a long time now. I reached my current best score after around 2-3 weeks into the competition.\nAnd it has been downhill from there. The more \"refined\" of an approach I try to apply, that seems to be theoretically correct to me, the larger the drop in my performance.\n\nI'll continue to trust myself on this one. From reaching as high as rank 3 to dropping down to the 40's, doesn't matter which way the result goes, there will be a lot of learning involved on this one!" }, "source": "meta" }