![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3132358%2F8b039fc9631dce853f8915576c39951c%2F2020-06-09%2022-01-49.png?generation=1591711413752119&alt=media\")
Hi everyone!
\n\nI have created kernel with merge external data (including metadata):
\n\nI have got 53351 samples for class 0 and 5106 for class 1.
\n\nMerged dataset you can find here\n- format: jpeg, rgb\n- size: 512x512
\n\nif you need other format of data (tf-records or other image size) you can use my kernel
\n\nI hope it helps you!
\n\nWelcome!
", "messages": [ { "id": 872528, "postDate": "2020-06-03T09:26:06.353Z", "content": "Hi everyone!
\n\nI have created kernel with merge external data (including metadata):
\n\nI have got 53351 samples for class 0 and 5106 for class 1.
\n\nMerged dataset you can find here\n- format: jpeg, rgb\n- size: 512x512
\n\nif you need other format of data (tf-records or other image size) you can use my kernel
\n\nI hope it helps you!
\n\nWelcome!
", "rawMarkdown": "Hi everyone!\n\nI have created [kernel with merge external data](https://www.kaggle.com/shonenkov/merge-external-data) (including metadata):\n\n- [Melanoma Detection Dataset](https://www.kaggle.com/wanderdust/skin-lesion-analysis-toward-melanoma-detection)\n- [Skin Lesion Images for Melanoma Classification](https://www.kaggle.com/andrewmvd/isic-2019)\n- [Skin Cancer MNIST: HAM10000](https://www.kaggle.com/kmader/skin-cancer-mnist-ham10000)\n- [SIIM-ISIC Melanoma Classification](https://www.kaggle.com/c/siim-isic-melanoma-classification/data)\n\nI have got 53351 samples for class 0 and 5106 for class 1. \n\nMerged dataset you can find [here](https://www.kaggle.com/shonenkov/melanoma-merged-external-data-512x512-jpeg)\n- format: jpeg, rgb\n- size: 512x512\n\nif you need other format of data (tf-records or other image size) you can use [my kernel](https://www.kaggle.com/shonenkov/merge-external-data)\n\nI hope it helps you!\n\nWelcome!\n", "votes": 74 }, { "id": 878163, "postDate": "2020-06-08T10:06:50.290Z", "content": "Please be aware of some duplicated images in the marged dataset that may has negative effect on the CV. This is mainly due to the naming of the ISIC 2019 images which some has \"_downsampled\" term in the name. \nExample of duplicate images:\nISIC_0000036.jpg (fold 0)\nISIC_0000036_downsampled.jpg (fold 1)
", "rawMarkdown": "Please be aware of some duplicated images in the marged dataset that may has negative effect on the CV. This is mainly due to the naming of the ISIC 2019 images which some has \"_downsampled\" term in the name. \nExample of duplicate images:\nISIC_0000036.jpg (fold 0)\nISIC_0000036_downsampled.jpg (fold 1)", "votes": 4, "replies": [ { "id": 878178, "postDate": "2020-06-08T10:22:54.760Z", "content": "thank you! Sure, It needs some corrections :)
", "rawMarkdown": "thank you! Sure, It needs some corrections :)", "votes": 1 }, { "id": 878183, "postDate": "2020-06-08T10:35:50.563Z", "content": "I just had a closer look at the datasets. I think it is enough to use the ISIC 2019 dataset (and not use the former datasets (i.e. ISIC 2018, ISIC 2017, ISIC 2016) as there is almost no new images in the former datasets. \nThe only benefit of using former datasets is that they have the images in the original resolutions and not in the downsampled format. Anyway as we may be forced to resize the images for training then even this benefit will not be useful.
", "rawMarkdown": "I just had a closer look at the datasets. I think it is enough to use the ISIC 2019 dataset (and not use the former datasets (i.e. ISIC 2018, ISIC 2017, ISIC 2016) as there is almost no new images in the former datasets. \nThe only benefit of using former datasets is that they have the images in the original resolutions and not in the downsampled format. Anyway as we may be forced to resize the images for training then even this benefit will not be useful. ", "votes": 3, "replies": [ { "id": 969479, "postDate": "2020-08-13T17:55:49.870Z", "content": "Hello ,
\n2019 competition winner have used multiple dataset ( https://arxiv.org/pdf/1910.03910.pdf ) ,
Why these dataset not considered as external dataset ?
\nFrom Paper,
\n A part of the training dataset is the HAM10000 dataset which contains images of
\nsize 600 × 450 that were centered and cropped around the lesion. The dataset
\ncurators applied histogram corrections to some images [16]. Another dataset,
\nBCN 20000, contains images of size 1024 × 1024. This dataset is particularly
\nchallenging as many images are uncropped and lesions in difficult and uncommon
\nlocations are present [5]. Last, the MSK dataset contains images with various
\nsizes.
\nThe dataset also contains meta information about the patient’s age group
\n(in steps of five years), the anatomical site (eight possible sites) and the sex
\n(male/female). The meta data is partially incomplete, i.e., there are missing
\nvalues for some images.
\nIn addition, we make use of external data. We use the 995 dermoscopic images
\nfrom the 7-point dataset [2].
Thanks,
\nAnthony
I have added fixes, thank you!
", "rawMarkdown": "I have added fixes, thank you! ", "votes": 2 } ] }, { "id": 872665, "postDate": "2020-06-03T12:09:40.317Z", "content": "Nice.\nCould you please clarify how you performed resizing? Was it simple resize with a loss of original aspect ratio or was it crop -> resize, keeping an original aspect ratio?
", "rawMarkdown": "Nice.\nCould you please clarify how you performed resizing? Was it simple resize with a loss of original aspect ratio or was it crop -> resize, keeping an original aspect ratio?", "votes": 1, "replies": [ { "id": 872672, "postDate": "2020-06-03T12:16:19.643Z", "content": "Thanks a lot!
\n\nI have used cv2.INTER_AREA without crop and saving ratio, you can see full code here ;) if need you can change resizing method.
\n\nWelcome!
", "rawMarkdown": "Thanks a lot!\n\nI have used cv2.INTER_AREA without crop and saving ratio, you can see full code [here](https://www.kaggle.com/shonenkov/merge-external-data) ;) if need you can change resizing method. \n\nWelcome!", "votes": 1 } ] }, { "id": 872784, "postDate": "2020-06-03T13:59:55.517Z", "content": "great work.
\n\n@shonenkov hi, can you please ensure the data leakage issue. I think it's really important. I've also faced some issues (like other reporters). Thank you.
", "rawMarkdown": "great work. \n\n@shonenkov hi, can you please ensure the data leakage issue. I think it's really important. I've also faced some issues (like other reporters). Thank you.", "votes": 2 }, { "id": 1608565, "postDate": "2021-12-06T12:57:56.477Z", "content": "Can you explain each file？I don't know what these .csv file represents？
", "rawMarkdown": "Can you explain each file？I don't know what these .csv file represents？" }, { "id": 918796, "postDate": "2020-07-07T13:56:53.147Z", "content": "This should be useful for some experimentation. Thank you!
", "rawMarkdown": "This should be useful for some experimentation. Thank you!\n" }, { "id": 878898, "postDate": "2020-06-09T02:08:53.330Z", "content": "I only use ISIC2019 & ISIC2020,but got overfitting,val 0.9269 LB0.887,do you know the reason?
", "rawMarkdown": "I only use ISIC2019 & ISIC2020,but got overfitting,val 0.9269 LB0.887,do you know the reason?", "replies": [ { "id": 878945, "postDate": "2020-06-09T03:58:28.657Z", "content": "now my val score 0.9168 lb score 0.917 ,I only fix the channel bgr ->rgb
", "rawMarkdown": "now my val score 0.9168 lb score 0.917 ,I only fix the channel bgr ->rgb " }, { "id": 879448, "postDate": "2020-06-09T13:31:31.210Z", "content": "For all the images??
", "rawMarkdown": "For all the images?? " }, { "id": 879453, "postDate": "2020-06-09T13:37:56.740Z", "content": "Images are already in RGB according to dataset kernel
", "rawMarkdown": "Images are already in RGB according to dataset kernel" }, { "id": 879485, "postDate": "2020-06-09T14:04:47Z", "content": "ISIC2019 & ISIC2020 the left img is RGB format,but overfit maybe is not the reason.\n
I'm still having trouble getting it. Could you please clarify it...
", "rawMarkdown": "I'm still having trouble getting it. Could you please clarify it..." }, { "id": 880314, "postDate": "2020-06-10T07:11:23.460Z", "content": "when you open the images using cv2.imread, by default, it has the BGR format. So if you need RGB format you need to switch the blue and red channels. However, if you have all the training and test images in BGR format, I assume it will not affect your final score.
I have the same overfit problem, I could try this…but like others said this change of channel should not matter.…
", "rawMarkdown": "I have the same overfit problem, I could try this…but like others said this change of channel should not matter.…" } ] }, { "id": 876168, "postDate": "2020-06-06T13:51:40.040Z", "content": "Please could you share the tf-records for the final dataset? Couldn't find it in the notebook..
\n\nThanks and brilliant job!
", "rawMarkdown": "Please could you share the tf-records for the final dataset? Couldn't find it in the notebook..\n\nThanks and brilliant job!", "replies": [ { "id": 876173, "postDate": "2020-06-06T13:55:02.467Z", "content": "Looks like I found it.
\n\nhttps://www.kaggle.com/cdeotte/how-to-create-tfrecords
\n\nAnyways thanks!
", "rawMarkdown": "Looks like I found it.\n\nhttps://www.kaggle.com/cdeotte/how-to-create-tfrecords\n\nAnyways thanks!" } ] }, { "id": 878304, "postDate": "2020-06-08T12:29:04.373Z", "rawMarkdown": "", "votes": 1, "isDeleted": true, "replies": [ { "id": 878386, "postDate": "2020-06-08T13:44:10.290Z", "content": "@awsaf49ieee I have added folds_08062020.csv with ISIC2019 & ISIC2020. I did't change images and other files to avoid conflicts of versions
Thank you very much!
", "rawMarkdown": "Thank you very much!" } ], "comments": [ { "id": 878163, "author_name": "Amirreza Mahbod", "author_url": "", "post_date": "2020-06-08T10:06:50.290000", "content": "Please be aware of some duplicated images in the marged dataset that may has negative effect on the CV. This is mainly due to the naming of the ISIC 2019 images which some has \"_downsampled\" term in the name. \nExample of duplicate images:\nISIC_0000036.jpg (fold 0)\nISIC_0000036_downsampled.jpg (fold 1)
", "votes": 4, "replies": [ { "id": 878178, "author_name": "Alex Shonenkov", "author_url": "", "post_date": "2020-06-08T10:22:54.760000", "content": "thank you! Sure, It needs some corrections :)
", "votes": 1, "replies": [] }, { "id": 878183, "author_name": "Amirreza Mahbod", "author_url": "", "post_date": "2020-06-08T10:35:50.563000", "content": "I just had a closer look at the datasets. I think it is enough to use the ISIC 2019 dataset (and not use the former datasets (i.e. ISIC 2018, ISIC 2017, ISIC 2016) as there is almost no new images in the former datasets. \nThe only benefit of using former datasets is that they have the images in the original resolutions and not in the downsampled format. Anyway as we may be forced to resize the images for training then even this benefit will not be useful.
", "votes": 3, "replies": [ { "id": 969479, "author_name": "Anthony Leo", "author_url": "", "post_date": "2020-08-13T17:55:49.870000", "content": "Hello ,
\n2019 competition winner have used multiple dataset ( https://arxiv.org/pdf/1910.03910.pdf ) ,
Why these dataset not considered as external dataset ?
\nFrom Paper,
\n A part of the training dataset is the HAM10000 dataset which contains images of
\nsize 600 × 450 that were centered and cropped around the lesion. The dataset
\ncurators applied histogram corrections to some images [16]. Another dataset,
\nBCN 20000, contains images of size 1024 × 1024. This dataset is particularly
\nchallenging as many images are uncropped and lesions in difficult and uncommon
\nlocations are present [5]. Last, the MSK dataset contains images with various
\nsizes.
\nThe dataset also contains meta information about the patient’s age group
\n(in steps of five years), the anatomical site (eight possible sites) and the sex
\n(male/female). The meta data is partially incomplete, i.e., there are missing
\nvalues for some images.
\nIn addition, we make use of external data. We use the 995 dermoscopic images
\nfrom the 7-point dataset [2].
Thanks,
\nAnthony
I have added fixes, thank you!
", "votes": 2, "replies": [] } ] }, { "id": 872665, "author_name": "Roman", "author_url": "", "post_date": "2020-06-03T12:09:40.317000", "content": "Nice.\nCould you please clarify how you performed resizing? Was it simple resize with a loss of original aspect ratio or was it crop -> resize, keeping an original aspect ratio?
", "votes": 1, "replies": [ { "id": 872672, "author_name": "Alex Shonenkov", "author_url": "", "post_date": "2020-06-03T12:16:19.643000", "content": "Thanks a lot!
\n\nI have used cv2.INTER_AREA without crop and saving ratio, you can see full code here ;) if need you can change resizing method.
\n\nWelcome!
", "votes": 1, "replies": [] } ] }, { "id": 872784, "author_name": "Innat", "author_url": "", "post_date": "2020-06-03T13:59:55.517000", "content": "great work.
\n\n@shonenkov hi, can you please ensure the data leakage issue. I think it's really important. I've also faced some issues (like other reporters). Thank you.
", "votes": 2, "replies": [] }, { "id": 1608565, "author_name": "zhujjie", "author_url": "", "post_date": "2021-12-06T12:57:56.477000", "content": "Can you explain each file？I don't know what these .csv file represents？
", "votes": 0, "replies": [] }, { "id": 918796, "author_name": "Harsh Nagouda", "author_url": "", "post_date": "2020-07-07T13:56:53.147000", "content": "This should be useful for some experimentation. Thank you!
", "votes": 0, "replies": [] }, { "id": 878898, "author_name": "xiaopeng", "author_url": "", "post_date": "2020-06-09T02:08:53.330000", "content": "I only use ISIC2019 & ISIC2020,but got overfitting,val 0.9269 LB0.887,do you know the reason?
", "votes": 0, "replies": [ { "id": 878945, "author_name": "Awsaf", "author_url": "", "post_date": "2020-06-09T03:58:28.657000", "content": "now my val score 0.9168 lb score 0.917 ,I only fix the channel bgr ->rgb
", "votes": 0, "replies": [] }, { "id": 879448, "author_name": "Awsaf", "author_url": "", "post_date": "2020-06-09T13:31:31.210000", "content": "For all the images??
", "votes": 0, "replies": [] }, { "id": 879453, "author_name": "Awsaf", "author_url": "", "post_date": "2020-06-09T13:37:56.740000", "content": "Images are already in RGB according to dataset kernel
", "votes": 0, "replies": [] }, { "id": 879485, "author_name": "xiaopeng", "author_url": "", "post_date": "2020-06-09T14:04:47", "content": "ISIC2019 & ISIC2020 the left img is RGB format,but overfit maybe is not the reason.\n
I'm still having trouble getting it. Could you please clarify it...
", "votes": 0, "replies": [] }, { "id": 880314, "author_name": "Amirreza Mahbod", "author_url": "", "post_date": "2020-06-10T07:11:23.460000", "content": "when you open the images using cv2.imread, by default, it has the BGR format. So if you need RGB format you need to switch the blue and red channels. However, if you have all the training and test images in BGR format, I assume it will not affect your final score.
I have the same overfit problem, I could try this…but like others said this change of channel should not matter.…
", "votes": 0, "replies": [] } ] }, { "id": 876168, "author_name": "DeepWilson", "author_url": "", "post_date": "2020-06-06T13:51:40.040000", "content": "Please could you share the tf-records for the final dataset? Couldn't find it in the notebook..
\n\nThanks and brilliant job!
", "votes": 0, "replies": [ { "id": 876173, "author_name": "DeepWilson", "author_url": "", "post_date": "2020-06-06T13:55:02.467000", "content": "Looks like I found it.
\n\nhttps://www.kaggle.com/cdeotte/how-to-create-tfrecords
\n\nAnyways thanks!
", "votes": 0, "replies": [] } ] }, { "id": 878304, "author_name": "", "author_url": "", "post_date": "2020-06-08T12:29:04.373000", "content": "", "votes": 1, "replies": [ { "id": 878386, "author_name": "Alex Shonenkov", "author_url": "", "post_date": "2020-06-08T13:44:10.290000", "content": "@awsaf49ieee I have added folds_08062020.csv with ISIC2019 & ISIC2020. I did't change images and other files to avoid conflicts of versions
Thank you very much!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "872528": "Hi everyone!\n\nI have created [kernel with merge external data](https://www.kaggle.com/shonenkov/merge-external-data) (including metadata):\n\n- [Melanoma Detection Dataset](https://www.kaggle.com/wanderdust/skin-lesion-analysis-toward-melanoma-detection)\n- [Skin Lesion Images for Melanoma Classification](https://www.kaggle.com/andrewmvd/isic-2019)\n- [Skin Cancer MNIST: HAM10000](https://www.kaggle.com/kmader/skin-cancer-mnist-ham10000)\n- [SIIM-ISIC Melanoma Classification](https://www.kaggle.com/c/siim-isic-melanoma-classification/data)\n\nI have got 53351 samples for class 0 and 5106 for class 1. \n\nMerged dataset you can find [here](https://www.kaggle.com/shonenkov/melanoma-merged-external-data-512x512-jpeg)\n- format: jpeg, rgb\n- size: 512x512\n\nif you need other format of data (tf-records or other image size) you can use [my kernel](https://www.kaggle.com/shonenkov/merge-external-data)\n\nI hope it helps you!\n\nWelcome!\n", "878163": "Please be aware of some duplicated images in the marged dataset that may has negative effect on the CV. This is mainly due to the naming of the ISIC 2019 images which some has \"_downsampled\" term in the name. \nExample of duplicate images:\nISIC_0000036.jpg (fold 0)\nISIC_0000036_downsampled.jpg (fold 1)", "872665": "Nice.\nCould you please clarify how you performed resizing? Was it simple resize with a loss of original aspect ratio or was it crop -> resize, keeping an original aspect ratio?", "872784": "great work. \n\n@shonenkov hi, can you please ensure the data leakage issue. I think it's really important. I've also faced some issues (like other reporters). Thank you.", "1608565": "Can you explain each file？I don't know what these .csv file represents？", "918796": "This should be useful for some experimentation. Thank you!\n", "878898": "I only use ISIC2019 & ISIC2020,but got overfitting,val 0.9269 LB0.887,do you know the reason?", "876168": "Please could you share the tf-records for the final dataset? Couldn't find it in the notebook..\n\nThanks and brilliant job!", "878304": "", "876670": "", "918401": "Thank you very much!" } }