\"Could this be?\"
\n\nHere's what I thought - if a model first starts to learn to classify different Dermoscopic images of various categories presented in the ISIC 2019 data particularly - \n1. Melanoma\n2. Melanocytic nevus\n3. Basal cell carcinoma\n4. Actinic keratosis\n5. Benign keratosis (solar lentigo / seborrheic keratosis / lichen planus-like keratosis)\n6. Dermatofibroma\n7. Vascular lesion\n8. Squamous cell carcinoma
\n\nAnd then we use these pretrained to train a Melonama Classifier on ISIC 2020 data, we should get better results right?
\n\nIn my experiments I am getting slightly worse results when starting out with ISIC 2019 pretrained model than simply starting out with pretrained Imagenet weights.
\n\nHave you tried experimenting with this? Does this make sense?
", "messages": [ { "id": "918040", "postDate": "07/06/2020 22:59:51", "content": "\"Could this be?\"
\n\nHere's what I thought - if a model first starts to learn to classify different Dermoscopic images of various categories presented in the ISIC 2019 data particularly - \n1. Melanoma\n2. Melanocytic nevus\n3. Basal cell carcinoma\n4. Actinic keratosis\n5. Benign keratosis (solar lentigo / seborrheic keratosis / lichen planus-like keratosis)\n6. Dermatofibroma\n7. Vascular lesion\n8. Squamous cell carcinoma
\n\nAnd then we use these pretrained to train a Melonama Classifier on ISIC 2020 data, we should get better results right?
\n\nIn my experiments I am getting slightly worse results when starting out with ISIC 2019 pretrained model than simply starting out with pretrained Imagenet weights.
\n\nHave you tried experimenting with this? Does this make sense?
", "rawMarkdown": "\"Could this be?\"\n\nHere's what I thought - if a model first starts to learn to classify different Dermoscopic images of various categories presented in the ISIC 2019 data particularly - \n1. Melanoma\n2. Melanocytic nevus\n3. Basal cell carcinoma\n4. Actinic keratosis\n5. Benign keratosis (solar lentigo / seborrheic keratosis / lichen planus-like keratosis)\n6. Dermatofibroma\n7. Vascular lesion\n8. Squamous cell carcinoma\n\nAnd then we use these pretrained to train a Melonama Classifier on ISIC 2020 data, we should get better results right? \n\nIn my experiments I am getting slightly worse results when starting out with ISIC 2019 pretrained model than simply starting out with pretrained Imagenet weights.\n\nHave you tried experimenting with this? Does this make sense?", "votes": null }, { "id": "918086", "postDate": "07/07/2020 01:13:17", "content": "are you setting pre-trained weights in either case and then just trying your final classifier (freezing the main model) or are you setting the pre-trained weights and then training everything?
", "rawMarkdown": "are you setting pre-trained weights in either case and then just trying your final classifier (freezing the main model) or are you setting the pre-trained weights and then training everything?", "votes": null }, { "id": "918087", "postDate": "07/07/2020 01:14:10", "content": "Starting with pretrained weights and then retraining the whole model (without freezing) again in both cases.
", "rawMarkdown": "Starting with pretrained weights and then retraining the whole model (without freezing) again in both cases.", "votes": null }, { "id": "918386", "postDate": "07/07/2020 08:10:12", "content": "It looks like it make sense. But practically i don't think it will work. As what we are doing is we are focusing it to single class ,so it might a bit of overfit on that class. Yeah , if validate it it will be giving really good performance, but on submission it might result in poor performance. Its just a thought i have about this.\nLets see what our grandmasters say about this.
", "rawMarkdown": "It looks like it make sense. But practically i don't think it will work. As what we are doing is we are focusing it to single class ,so it might a bit of overfit on that class. Yeah , if validate it it will be giving really good performance, but on submission it might result in poor performance. Its just a thought i have about this.\nLets see what our grandmasters say about this.", "votes": null }, { "id": "918937", "postDate": "07/07/2020 15:50:23", "content": "Yes it makes sense, once you fully understand what CNNs do. Training with Imagenet is having millions of images and tons of different patterns and information stored in each layer, different representations of information. If you try to pretrain your model with images of melanomas, you would not have enough diversity in your images to find new patterns and store new information, this will make your net to under develop. Bigger nets can store more information and process new kind of patterns, transfer learning helps us to re use those patterns but for seeking different tasks, that's why we are able to use the same weights from Imagenet, but we just retrain it to use all those pre learned patterns for detecting melanomas. Who knows, maybe the pattern and color scheme for a centipede is really similar to some patterns for melanoma detection. At the end of the day, it is all just a crazy big black box, and we need to accept that.
", "rawMarkdown": "Yes it makes sense, once you fully understand what CNNs do. Training with Imagenet is having millions of images and tons of different patterns and information stored in each layer, different representations of information. If you try to pretrain your model with images of melanomas, you would not have enough diversity in your images to find new patterns and store new information, this will make your net to under develop. Bigger nets can store more information and process new kind of patterns, transfer learning helps us to re use those patterns but for seeking different tasks, that's why we are able to use the same weights from Imagenet, but we just retrain it to use all those pre learned patterns for detecting melanomas. Who knows, maybe the pattern and color scheme for a centipede is really similar to some patterns for melanoma detection. At the end of the day, it is all just a crazy big black box, and we need to accept that.", "votes": null }, { "id": "919318", "postDate": "07/07/2020 19:36:07", "content": "Guillermo, so did you load pretrained weights and then freeze them and just train your classifier layer?
", "rawMarkdown": "Guillermo, so did you load pretrained weights and then freeze them and just train your classifier layer?", "votes": null }, { "id": "919353", "postDate": "07/07/2020 19:59:33", "content": "Using last years 2019 has produced strange results for everyone. If you just mix last year's data together with this years data and train on all 60k images most people have observed lower CV and LB. This is odd because more data should improve CV LB.
\n\nI have published last years 25k images as TFRecords and JPEGs here if people wish to play around with last years data.
", "rawMarkdown": "Using last years 2019 has produced strange results for everyone. If you just mix last year's data together with this years data and train on all 60k images most people have observed lower CV and LB. This is odd because more data should improve CV LB.\n\nI have published last years 25k images as TFRecords and JPEGs [here][1] if people wish to play around with last years data.\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/164910", "votes": null }, { "id": "919360", "postDate": "07/07/2020 20:02:25", "content": "I have also observed that if you just include last years malignant images, your train AUC quickly rises to 1.0. So I think the malignant images from last year are different than all images this year (making it easy for model to pick them out). For example, maybe all images are more zoomed in or use different colors etc.
\n\nSo if we wish to use last year's data, we first need to adjust last year's data so that adversarial validation can not distinguish last year from this year.
", "rawMarkdown": "I have also observed that if you just include last years malignant images, your train AUC quickly rises to 1.0. So I think the malignant images from last year are different than all images this year (making it easy for model to pick them out). For example, maybe all images are more zoomed in or use different colors etc.\n\nSo if we wish to use last year's data, we first need to adjust last year's data so that adversarial validation can not distinguish last year from this year.", "votes": null }, { "id": "919420", "postDate": "07/07/2020 20:52:37", "content": "No i did not freeze the weights, I simply did not try to pretrain it before with ISIC 19 since it will overfit to that data, one thing i tried was combining the images of last year with this year and train on all the set of images, but it lower my score. I'm guessing that the distribution of the images differ so in general we should stick with the ISIC20 images or use ISIC19 images but following the same distribution that ISIC20 (something i have planned to do eventually).
", "rawMarkdown": "No i did not freeze the weights, I simply did not try to pretrain it before with ISIC 19 since it will overfit to that data, one thing i tried was combining the images of last year with this year and train on all the set of images, but it lower my score. I'm guessing that the distribution of the images differ so in general we should stick with the ISIC20 images or use ISIC19 images but following the same distribution that ISIC20 (something i have planned to do eventually).", "votes": null }, { "id": "919427", "postDate": "07/07/2020 21:05:40", "content": "Thanks @cdeotte ! I have observed something similar. Having kept everything else the same, I added Melonama images of 2019 to one of the two models.
\n\nI saw a difference of 1.3 CV AUC for 256x256 sized images between the two and the model with Melonama external 2019 data performed better.
\n\nI have added color constancy to 2019 and 2020 data so the colors look similar to the naked eye and hopefully this will help the model become more robust.
", "rawMarkdown": "Thanks @cdeotte ! I have observed something similar. Having kept everything else the same, I added Melonama images of 2019 to one of the two models. \n\nI saw a difference of 1.3 CV AUC for 256x256 sized images between the two and the model with Melonama external 2019 data performed better. \n\nI have added color constancy to 2019 and 2020 data so the colors look similar to the naked eye and hopefully this will help the model become more robust.", "votes": null }, { "id": "920086", "postDate": "07/08/2020 10:09:29", "content": "Same observation... When I use ISIC 2019 data for training alone while validating on ISIC 2020 data, the AUC was terrible. However, when I blend the training with some ISIC 2020 data as well, there seems to be an improvement over not using any external data. It doesn't make sense to me and wonder why too.
", "rawMarkdown": "Same observation... When I use ISIC 2019 data for training alone while validating on ISIC 2020 data, the AUC was terrible. However, when I blend the training with some ISIC 2020 data as well, there seems to be an improvement over not using any external data. It doesn't make sense to me and wonder why too.", "votes": null }, { "id": "921641", "postDate": "07/09/2020 12:51:02", "content": "It seems like the images in 2019 are different from that of 2020 data. Especially the 1024x1024 size. There’s a separate topic “how to use 2019 data” where this was discussed.
", "rawMarkdown": "It seems like the images in 2019 are different from that of 2020 data. Especially the 1024x1024 size. There’s a separate topic “how to use 2019 data” where this was discussed.", "votes": null } ], "comments": [ { "id": 918086, "author_name": "brianfeeny", "author_url": "", "post_date": "07/07/2020 01:13:17", "content": "are you setting pre-trained weights in either case and then just trying your final classifier (freezing the main model) or are you setting the pre-trained weights and then training everything?
", "votes": null, "replies": [ { "id": 918087, "author_name": "aroraaman", "author_url": "", "post_date": "07/07/2020 01:14:10", "content": "Starting with pretrained weights and then retraining the whole model (without freezing) again in both cases.
", "votes": null, "replies": [] } ] }, { "id": 918386, "author_name": "prashantarorat", "author_url": "", "post_date": "07/07/2020 08:10:12", "content": "It looks like it make sense. But practically i don't think it will work. As what we are doing is we are focusing it to single class ,so it might a bit of overfit on that class. Yeah , if validate it it will be giving really good performance, but on submission it might result in poor performance. Its just a thought i have about this.\nLets see what our grandmasters say about this.
", "votes": null, "replies": [] }, { "id": 918937, "author_name": "guillermocampollo", "author_url": "", "post_date": "07/07/2020 15:50:23", "content": "Yes it makes sense, once you fully understand what CNNs do. Training with Imagenet is having millions of images and tons of different patterns and information stored in each layer, different representations of information. If you try to pretrain your model with images of melanomas, you would not have enough diversity in your images to find new patterns and store new information, this will make your net to under develop. Bigger nets can store more information and process new kind of patterns, transfer learning helps us to re use those patterns but for seeking different tasks, that's why we are able to use the same weights from Imagenet, but we just retrain it to use all those pre learned patterns for detecting melanomas. Who knows, maybe the pattern and color scheme for a centipede is really similar to some patterns for melanoma detection. At the end of the day, it is all just a crazy big black box, and we need to accept that.
", "votes": null, "replies": [ { "id": 919318, "author_name": "brianfeeny", "author_url": "", "post_date": "07/07/2020 19:36:07", "content": "Guillermo, so did you load pretrained weights and then freeze them and just train your classifier layer?
", "votes": null, "replies": [] }, { "id": 919420, "author_name": "guillermocampollo", "author_url": "", "post_date": "07/07/2020 20:52:37", "content": "No i did not freeze the weights, I simply did not try to pretrain it before with ISIC 19 since it will overfit to that data, one thing i tried was combining the images of last year with this year and train on all the set of images, but it lower my score. I'm guessing that the distribution of the images differ so in general we should stick with the ISIC20 images or use ISIC19 images but following the same distribution that ISIC20 (something i have planned to do eventually).
", "votes": null, "replies": [] } ] }, { "id": 919353, "author_name": "cdeotte", "author_url": "", "post_date": "07/07/2020 19:59:33", "content": "Using last years 2019 has produced strange results for everyone. If you just mix last year's data together with this years data and train on all 60k images most people have observed lower CV and LB. This is odd because more data should improve CV LB.
\n\nI have published last years 25k images as TFRecords and JPEGs here if people wish to play around with last years data.
", "votes": null, "replies": [ { "id": 919360, "author_name": "cdeotte", "author_url": "", "post_date": "07/07/2020 20:02:25", "content": "I have also observed that if you just include last years malignant images, your train AUC quickly rises to 1.0. So I think the malignant images from last year are different than all images this year (making it easy for model to pick them out). For example, maybe all images are more zoomed in or use different colors etc.
\n\nSo if we wish to use last year's data, we first need to adjust last year's data so that adversarial validation can not distinguish last year from this year.
", "votes": null, "replies": [] }, { "id": 919427, "author_name": "aroraaman", "author_url": "", "post_date": "07/07/2020 21:05:40", "content": "Thanks @cdeotte ! I have observed something similar. Having kept everything else the same, I added Melonama images of 2019 to one of the two models.
\n\nI saw a difference of 1.3 CV AUC for 256x256 sized images between the two and the model with Melonama external 2019 data performed better.
\n\nI have added color constancy to 2019 and 2020 data so the colors look similar to the naked eye and hopefully this will help the model become more robust.
", "votes": null, "replies": [] } ] }, { "id": 920086, "author_name": "yeeseng", "author_url": "", "post_date": "07/08/2020 10:09:29", "content": "Same observation... When I use ISIC 2019 data for training alone while validating on ISIC 2020 data, the AUC was terrible. However, when I blend the training with some ISIC 2020 data as well, there seems to be an improvement over not using any external data. It doesn't make sense to me and wonder why too.
", "votes": null, "replies": [ { "id": 921641, "author_name": "aroraaman", "author_url": "", "post_date": "07/09/2020 12:51:02", "content": "It seems like the images in 2019 are different from that of 2020 data. Especially the 1024x1024 size. There’s a separate topic “how to use 2019 data” where this was discussed.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "918040": "\"Could this be?\"\n\nHere's what I thought - if a model first starts to learn to classify different Dermoscopic images of various categories presented in the ISIC 2019 data particularly - \n1. Melanoma\n2. Melanocytic nevus\n3. Basal cell carcinoma\n4. Actinic keratosis\n5. Benign keratosis (solar lentigo / seborrheic keratosis / lichen planus-like keratosis)\n6. Dermatofibroma\n7. Vascular lesion\n8. Squamous cell carcinoma\n\nAnd then we use these pretrained to train a Melonama Classifier on ISIC 2020 data, we should get better results right? \n\nIn my experiments I am getting slightly worse results when starting out with ISIC 2019 pretrained model than simply starting out with pretrained Imagenet weights.\n\nHave you tried experimenting with this? Does this make sense?", "918086": "are you setting pre-trained weights in either case and then just trying your final classifier (freezing the main model) or are you setting the pre-trained weights and then training everything?", "918087": "Starting with pretrained weights and then retraining the whole model (without freezing) again in both cases.", "918386": "It looks like it make sense. But practically i don't think it will work. As what we are doing is we are focusing it to single class ,so it might a bit of overfit on that class. Yeah , if validate it it will be giving really good performance, but on submission it might result in poor performance. Its just a thought i have about this.\nLets see what our grandmasters say about this.", "918937": "Yes it makes sense, once you fully understand what CNNs do. Training with Imagenet is having millions of images and tons of different patterns and information stored in each layer, different representations of information. If you try to pretrain your model with images of melanomas, you would not have enough diversity in your images to find new patterns and store new information, this will make your net to under develop. Bigger nets can store more information and process new kind of patterns, transfer learning helps us to re use those patterns but for seeking different tasks, that's why we are able to use the same weights from Imagenet, but we just retrain it to use all those pre learned patterns for detecting melanomas. Who knows, maybe the pattern and color scheme for a centipede is really similar to some patterns for melanoma detection. At the end of the day, it is all just a crazy big black box, and we need to accept that.", "919318": "Guillermo, so did you load pretrained weights and then freeze them and just train your classifier layer?", "919353": "Using last years 2019 has produced strange results for everyone. If you just mix last year's data together with this years data and train on all 60k images most people have observed lower CV and LB. This is odd because more data should improve CV LB.\n\nI have published last years 25k images as TFRecords and JPEGs [here][1] if people wish to play around with last years data.\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/164910", "919360": "I have also observed that if you just include last years malignant images, your train AUC quickly rises to 1.0. So I think the malignant images from last year are different than all images this year (making it easy for model to pick them out). For example, maybe all images are more zoomed in or use different colors etc.\n\nSo if we wish to use last year's data, we first need to adjust last year's data so that adversarial validation can not distinguish last year from this year.", "919420": "No i did not freeze the weights, I simply did not try to pretrain it before with ISIC 19 since it will overfit to that data, one thing i tried was combining the images of last year with this year and train on all the set of images, but it lower my score. I'm guessing that the distribution of the images differ so in general we should stick with the ISIC20 images or use ISIC19 images but following the same distribution that ISIC20 (something i have planned to do eventually).", "919427": "Thanks @cdeotte ! I have observed something similar. Having kept everything else the same, I added Melonama images of 2019 to one of the two models. \n\nI saw a difference of 1.3 CV AUC for 256x256 sized images between the two and the model with Melonama external 2019 data performed better. \n\nI have added color constancy to 2019 and 2020 data so the colors look similar to the naked eye and hopefully this will help the model become more robust.", "920086": "Same observation... When I use ISIC 2019 data for training alone while validating on ISIC 2020 data, the AUC was terrible. However, when I blend the training with some ISIC 2020 data as well, there seems to be an improvement over not using any external data. It doesn't make sense to me and wonder why too.", "921641": "It seems like the images in 2019 are different from that of 2020 data. Especially the 1024x1024 size. There’s a separate topic “how to use 2019 data” where this was discussed." }, "source": "meta" }