Accuracy will always be high because there are only 1% malignant. Therefore if you guess target = 0 which is benign for every image then you have 99% accuracy. You need to use the metric AUC. (Note that AUC is area under the roc curve and it is different than accuracy).
", "rawMarkdown": "Accuracy will always be high because there are only 1% malignant. Therefore if you guess target = 0 which is benign for every image then you have 99% accuracy. You need to use the metric AUC. (Note that AUC is area under the roc curve and it is different than accuracy).", "votes": 6 }, { "id": 921412, "postDate": "2020-07-09T08:55:26.240Z", "content": "Check the top topic in discussions, there's a problem with the data.
\n\nQuoting here,
\n\nDear SIIM/ISIC 2020 challenge participants,
\n\nWe're very grateful for the awesome feedback we have received so far, and especially to the users who have prodded us with information about duplicate images.
\n\nIndeed, we discovered that based on an issue that happened during data ingestion into our archive, several hundred images were indeed stored twice under two internal names (ISIC_ID).
\n\nAttached is a list of images that are these exact duplicates, with both ISIC_IDs and the partition in which they occur in--in all cases both copies are in the same partition! In other words, no leakage occurred in this manner that we are aware of.
\n\nPlease note that several threads by users have also demonstrated duplicates and near duplicates between the 2020 challenge dataset and images prior uploaded to the ISIC Archive. This was, to some extent, inevitable, since the archive doesn't yet support complex image duplicate detection, which is something we will be working on before next year's challenge. We are aware of some of these additional (near-) duplicates, but believe them to be few in number.
\n\nSince the split of exact duplicates across train and test roughly matches the overall dataset split, and the total number of possible leaks is small, we believe that no substantial bias in favor of specific outcomes occurs, and we have agreed with the Kaggle team that the scoring will not be altered--which we assume will also make participation much easier.
\n\nWe would like to apologize that this mistake happened, and we will put in place at least two mechanisms and steps such that future datasets will definitely not have the same problem.
\n\nAnd, again, thank you very much for such fantastic submissions to date, and also to the users who have provided some initial code for the detection of duplicates!
\n\n\n\nhttps://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/161943
", "rawMarkdown": "Check the top topic in discussions, there's a problem with the data.\n\nQuoting here,\n\n\nDear SIIM/ISIC 2020 challenge participants,\n\nWe're very grateful for the awesome feedback we have received so far, and especially to the users who have prodded us with information about duplicate images.\n\nIndeed, we discovered that based on an issue that happened during data ingestion into our archive, several hundred images were indeed stored twice under two internal names (ISIC_ID).\n\nAttached is a list of images that are these exact duplicates, with both ISIC_IDs and the partition in which they occur in--in all cases both copies are in the same partition! In other words, no leakage occurred in this manner that we are aware of.\n\nPlease note that several threads by users have also demonstrated duplicates and near duplicates between the 2020 challenge dataset and images prior uploaded to the ISIC Archive. This was, to some extent, inevitable, since the archive doesn't yet support complex image duplicate detection, which is something we will be working on before next year's challenge. We are aware of some of these additional (near-) duplicates, but believe them to be few in number.\n\nSince the split of exact duplicates across train and test roughly matches the overall dataset split, and the total number of possible leaks is small, we believe that no substantial bias in favor of specific outcomes occurs, and we have agreed with the Kaggle team that the scoring will not be altered--which we assume will also make participation much easier.\n\nWe would like to apologize that this mistake happened, and we will put in place at least two mechanisms and steps such that future datasets will definitely not have the same problem.\n\nAnd, again, thank you very much for such fantastic submissions to date, and also to the users who have provided some initial code for the detection of duplicates!\n\n[Duplicates](https://storage.googleapis.com/kaggle-forum-message-attachments/903372/16145/2020_Challenge_duplicates.csv)\n\nhttps://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/161943" }, { "id": 916405, "postDate": "2020-07-05T15:49:43.453Z", "content": "I am facing a similar situation when I use the AUC as my evaluation metric. Both training and validation accuracy turn out to be 0.98+ but on submitting my solutions I get only 0.83-0.85. Any idea why this is happening?\nIf needed I will make my notebook public and attach the link for it here so that you can tell me if I'm doing anything wrong. I am using a GPU notebook
", "rawMarkdown": "I am facing a similar situation when I use the AUC as my evaluation metric. Both training and validation accuracy turn out to be 0.98+ but on submitting my solutions I get only 0.83-0.85. Any idea why this is happening?\nIf needed I will make my notebook public and attach the link for it here so that you can tell me if I'm doing anything wrong. I am using a GPU notebook\n", "replies": [ { "id": 917560, "postDate": "2020-07-06T15:56:00.987Z", "content": "Are you using any external data. If yes than discard those images which are from external data subset. Validation set should only contain images from the competition. If you are not using any external data than may be there is issue with your splitting method., you could try stratified- KFOLD OR group-k-fold
", "rawMarkdown": "Are you using any external data. If yes than discard those images which are from external data subset. Validation set should only contain images from the competition. If you are not using any external data than may be there is issue with your splitting method., you could try stratified- KFOLD OR group-k-fold", "votes": 1 }, { "id": 917599, "postDate": "2020-07-06T16:19:02.980Z", "rawMarkdown": "", "isDeleted": true }, { "id": 921408, "postDate": "2020-07-09T08:52:48.527Z", "content": "Most probably it looks like a target leakage
", "rawMarkdown": "Most probably it looks like a target leakage" } ] }, { "id": 914667, "postDate": "2020-07-04T06:15:21.627Z", "rawMarkdown": "" }, { "id": 921410, "postDate": "2020-07-09T08:54:29.670Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 914679, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-04T06:23:56.413000", "content": "Accuracy will always be high because there are only 1% malignant. Therefore if you guess target = 0 which is benign for every image then you have 99% accuracy. You need to use the metric AUC. (Note that AUC is area under the roc curve and it is different than accuracy).
", "votes": 6, "replies": [] }, { "id": 921412, "author_name": "Samar Srivastava", "author_url": "", "post_date": "2020-07-09T08:55:26.240000", "content": "Check the top topic in discussions, there's a problem with the data.
\n\nQuoting here,
\n\nDear SIIM/ISIC 2020 challenge participants,
\n\nWe're very grateful for the awesome feedback we have received so far, and especially to the users who have prodded us with information about duplicate images.
\n\nIndeed, we discovered that based on an issue that happened during data ingestion into our archive, several hundred images were indeed stored twice under two internal names (ISIC_ID).
\n\nAttached is a list of images that are these exact duplicates, with both ISIC_IDs and the partition in which they occur in--in all cases both copies are in the same partition! In other words, no leakage occurred in this manner that we are aware of.
\n\nPlease note that several threads by users have also demonstrated duplicates and near duplicates between the 2020 challenge dataset and images prior uploaded to the ISIC Archive. This was, to some extent, inevitable, since the archive doesn't yet support complex image duplicate detection, which is something we will be working on before next year's challenge. We are aware of some of these additional (near-) duplicates, but believe them to be few in number.
\n\nSince the split of exact duplicates across train and test roughly matches the overall dataset split, and the total number of possible leaks is small, we believe that no substantial bias in favor of specific outcomes occurs, and we have agreed with the Kaggle team that the scoring will not be altered--which we assume will also make participation much easier.
\n\nWe would like to apologize that this mistake happened, and we will put in place at least two mechanisms and steps such that future datasets will definitely not have the same problem.
\n\nAnd, again, thank you very much for such fantastic submissions to date, and also to the users who have provided some initial code for the detection of duplicates!
\n\n\n\nhttps://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/161943
", "votes": 0, "replies": [] }, { "id": 916405, "author_name": "Aryan Pandey", "author_url": "", "post_date": "2020-07-05T15:49:43.453000", "content": "I am facing a similar situation when I use the AUC as my evaluation metric. Both training and validation accuracy turn out to be 0.98+ but on submitting my solutions I get only 0.83-0.85. Any idea why this is happening?\nIf needed I will make my notebook public and attach the link for it here so that you can tell me if I'm doing anything wrong. I am using a GPU notebook
", "votes": 0, "replies": [ { "id": 917560, "author_name": "ayush", "author_url": "", "post_date": "2020-07-06T15:56:00.987000", "content": "Are you using any external data. If yes than discard those images which are from external data subset. Validation set should only contain images from the competition. If you are not using any external data than may be there is issue with your splitting method., you could try stratified- KFOLD OR group-k-fold
", "votes": 1, "replies": [] }, { "id": 917599, "author_name": "", "author_url": "", "post_date": "2020-07-06T16:19:02.980000", "content": "", "votes": 0, "replies": [] }, { "id": 921408, "author_name": "Samar Srivastava", "author_url": "", "post_date": "2020-07-09T08:52:48.527000", "content": "Most probably it looks like a target leakage
", "votes": 0, "replies": [] } ] }, { "id": 921410, "author_name": "", "author_url": "", "post_date": "2020-07-09T08:54:29.670000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "914679": "Accuracy will always be high because there are only 1% malignant. Therefore if you guess target = 0 which is benign for every image then you have 99% accuracy. You need to use the metric AUC. (Note that AUC is area under the roc curve and it is different than accuracy).", "921412": "Check the top topic in discussions, there's a problem with the data.\n\nQuoting here,\n\n\nDear SIIM/ISIC 2020 challenge participants,\n\nWe're very grateful for the awesome feedback we have received so far, and especially to the users who have prodded us with information about duplicate images.\n\nIndeed, we discovered that based on an issue that happened during data ingestion into our archive, several hundred images were indeed stored twice under two internal names (ISIC_ID).\n\nAttached is a list of images that are these exact duplicates, with both ISIC_IDs and the partition in which they occur in--in all cases both copies are in the same partition! In other words, no leakage occurred in this manner that we are aware of.\n\nPlease note that several threads by users have also demonstrated duplicates and near duplicates between the 2020 challenge dataset and images prior uploaded to the ISIC Archive. This was, to some extent, inevitable, since the archive doesn't yet support complex image duplicate detection, which is something we will be working on before next year's challenge. We are aware of some of these additional (near-) duplicates, but believe them to be few in number.\n\nSince the split of exact duplicates across train and test roughly matches the overall dataset split, and the total number of possible leaks is small, we believe that no substantial bias in favor of specific outcomes occurs, and we have agreed with the Kaggle team that the scoring will not be altered--which we assume will also make participation much easier.\n\nWe would like to apologize that this mistake happened, and we will put in place at least two mechanisms and steps such that future datasets will definitely not have the same problem.\n\nAnd, again, thank you very much for such fantastic submissions to date, and also to the users who have provided some initial code for the detection of duplicates!\n\n[Duplicates](https://storage.googleapis.com/kaggle-forum-message-attachments/903372/16145/2020_Challenge_duplicates.csv)\n\nhttps://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/161943", "916405": "I am facing a similar situation when I use the AUC as my evaluation metric. Both training and validation accuracy turn out to be 0.98+ but on submitting my solutions I get only 0.83-0.85. Any idea why this is happening?\nIf needed I will make my notebook public and attach the link for it here so that you can tell me if I'm doing anything wrong. I am using a GPU notebook\n", "914667": "", "921410": "" } }