{ "id": 156403, "title": "Q. How can I perform Image Augmentation only for the Minority class?", "url": "/competitions/siim-isic-melanoma-classification/discussion/156403", "author_name": "", "post_date": "2020-06-05T21:25:47.486727Z", "votes": null, "comment_count": 7, "views": 0, "content": "

As the provided data is imbalanced, so if I try to perform Image Augmentation using Keras's ImageDataGenerator() it will perform image Augmentation on the whole dataset, which will certainly increase the number of images but the imbalanced class problem will be still there.\nSo, I want to perform Image Augmentation using Keras Specifically in the Malignant images, But I m unable to do that. If anybody can provide any insights or any links on that then it will be much appreciated.🙏🙏🙏

", "messages": [ { "id": "875509", "postDate": "06/05/2020 21:25:47", "content": "

As the provided data is imbalanced, so if I try to perform Image Augmentation using Keras's ImageDataGenerator() it will perform image Augmentation on the whole dataset, which will certainly increase the number of images but the imbalanced class problem will be still there.\nSo, I want to perform Image Augmentation using Keras Specifically in the Malignant images, But I m unable to do that. If anybody can provide any insights or any links on that then it will be much appreciated.🙏🙏🙏

", "rawMarkdown": "As the provided data is imbalanced, so if I try to perform Image Augmentation using `Keras's` `ImageDataGenerator()` it will perform image Augmentation on the whole dataset, which will certainly increase the number of images but the imbalanced class problem will be still there.\n`So, I want to perform Image Augmentation using Keras Specifically in the Malignant images, But I m unable to do that.` If anybody can provide any insights or any links on that then it will be much appreciated.🙏🙏🙏", "votes": null }, { "id": "875656", "postDate": "06/06/2020 03:51:03", "content": "

First thing, you can create a separate DF from the main DF like so:\n\ntrain_malignant = train[train['target']==1]\n

\n\n

Then go ahead!

", "rawMarkdown": "First thing, you can create a separate DF from the main DF like so:\n```\ntrain_malignant = train[train['target']==1]\n```\n\nThen go ahead!", "votes": null }, { "id": "875885", "postDate": "06/06/2020 08:32:12", "content": "

@soumya9977 performing Augmentation or any other transformation only on the Malignant images, is not a very good strategy. Your model will learn to recognize the augmentation and not the melanoma. You can use sampling techniques to solve the imbalance, but whatever you do Do not augment only Malignant images

", "rawMarkdown": "soumya9977 performing Augmentation or any other transformation only on the Malignant images, is not a very good strategy. Your model will learn to recognize the augmentation and not the melanoma. You can use sampling techniques to solve the imbalance, but whatever you do **Do not augment only Malignant images**", "votes": null }, { "id": "877339", "postDate": "06/07/2020 14:15:20", "content": "

so, what should be the approach? @yuval6967

", "rawMarkdown": "so, what should be the approach? @yuval6967", "votes": null }, { "id": "877799", "postDate": "06/08/2020 01:40:52", "content": "

You can create your own Keras data loader. Copy the template in this Melanoma notebook here, or my old notebook here from a different comp. Then inside the data loader, perform augmentation on which ever images you want.

", "rawMarkdown": "You can create your own Keras data loader. Copy the template in this Melanoma notebook [here][1], or my old notebook [here][2] from a different comp. Then inside the data loader, perform augmentation on which ever images you want.\n\n[1]: https://www.kaggle.com/ipythonx/tf-keras-melanoma-classification-starter-gpu\n[2]: https://www.kaggle.com/cdeotte/keras-unet-with-eda", "votes": null }, { "id": "879517", "postDate": "06/09/2020 14:29:24", "content": "

Thank you, sir 🙏

", "rawMarkdown": "Thank you, sir 🙏", "votes": null }, { "id": "884129", "postDate": "06/13/2020 07:48:25", "content": "

@soumya9977

\n\n
    \n

  1. augment everything the same way.

    2. \n

  2. Look for other sources for more data

    3. \n
", "rawMarkdown": "soumya9977 \n\n1. augment everything the same way.\n\n2. Look for other sources for more data", "votes": null }, { "id": "894397", "postDate": "06/20/2020 11:56:36", "content": "

I tried augmenting the melanoma class and saw a very high downfall in the auc score, my average CV dropped from 0.96 to 0.65.

\n\n

Just don't do this in any case just as Yuval said! :)

", "rawMarkdown": "I tried augmenting the melanoma class and saw a very high downfall in the auc score, my average CV dropped from 0.96 to 0.65. \n\nJust don't do this in any case just as Yuval said! :)", "votes": null } ], "comments": [ { "id": 875656, "author_name": "nxrprime", "author_url": "", "post_date": "06/06/2020 03:51:03", "content": "

First thing, you can create a separate DF from the main DF like so:\n\ntrain_malignant = train[train['target']==1]\n

\n\n

Then go ahead!

", "votes": null, "replies": [] }, { "id": 875885, "author_name": "yuval6967", "author_url": "", "post_date": "06/06/2020 08:32:12", "content": "

@soumya9977 performing Augmentation or any other transformation only on the Malignant images, is not a very good strategy. Your model will learn to recognize the augmentation and not the melanoma. You can use sampling techniques to solve the imbalance, but whatever you do Do not augment only Malignant images

", "votes": null, "replies": [ { "id": 877339, "author_name": "soumya9977", "author_url": "", "post_date": "06/07/2020 14:15:20", "content": "

so, what should be the approach? @yuval6967

", "votes": null, "replies": [] }, { "id": 884129, "author_name": "yuval6967", "author_url": "", "post_date": "06/13/2020 07:48:25", "content": "

@soumya9977

\n\n
    \n

  1. augment everything the same way.

    2. \n

  2. Look for other sources for more data

    3. \n
", "votes": null, "replies": [] }, { "id": 894397, "author_name": "sarques", "author_url": "", "post_date": "06/20/2020 11:56:36", "content": "

I tried augmenting the melanoma class and saw a very high downfall in the auc score, my average CV dropped from 0.96 to 0.65.

\n\n

Just don't do this in any case just as Yuval said! :)

", "votes": null, "replies": [] } ] }, { "id": 877799, "author_name": "cdeotte", "author_url": "", "post_date": "06/08/2020 01:40:52", "content": "

You can create your own Keras data loader. Copy the template in this Melanoma notebook here, or my old notebook here from a different comp. Then inside the data loader, perform augmentation on which ever images you want.

", "votes": null, "replies": [ { "id": 879517, "author_name": "soumya9977", "author_url": "", "post_date": "06/09/2020 14:29:24", "content": "

Thank you, sir 🙏

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "875509": "As the provided data is imbalanced, so if I try to perform Image Augmentation using `Keras's` `ImageDataGenerator()` it will perform image Augmentation on the whole dataset, which will certainly increase the number of images but the imbalanced class problem will be still there.\n`So, I want to perform Image Augmentation using Keras Specifically in the Malignant images, But I m unable to do that.` If anybody can provide any insights or any links on that then it will be much appreciated.🙏🙏🙏", "875656": "First thing, you can create a separate DF from the main DF like so:\n```\ntrain_malignant = train[train['target']==1]\n```\n\nThen go ahead!", "875885": "soumya9977 performing Augmentation or any other transformation only on the Malignant images, is not a very good strategy. Your model will learn to recognize the augmentation and not the melanoma. You can use sampling techniques to solve the imbalance, but whatever you do **Do not augment only Malignant images**", "877339": "so, what should be the approach? @yuval6967", "877799": "You can create your own Keras data loader. Copy the template in this Melanoma notebook [here][1], or my old notebook [here][2] from a different comp. Then inside the data loader, perform augmentation on which ever images you want.\n\n[1]: https://www.kaggle.com/ipythonx/tf-keras-melanoma-classification-starter-gpu\n[2]: https://www.kaggle.com/cdeotte/keras-unet-with-eda", "879517": "Thank you, sir 🙏", "884129": "soumya9977 \n\n1. augment everything the same way.\n\n2. Look for other sources for more data", "894397": "I tried augmenting the melanoma class and saw a very high downfall in the auc score, my average CV dropped from 0.96 to 0.65. \n\nJust don't do this in any case just as Yuval said! :)" }, "source": "meta" }