I was experimenting with single model predictions. There is concept of class weights [see this for more info] and got very unexpected results.\nModel: efficientnetb6\nDataset: 384x384 melanoma images [link]
\n\nRESULTS\n* class weights as {0: 0.5089, 1: 28.3613} [using formula] --> 0.887 [NOTEBOOK]\n* class weights as {0: 0.1, 1: 1} --> 0.885 [NOTEBOOK] \n* without class weights --> 0.82 [NOTEBOOK]
\n\nNOTE: No CV, very little Augmentation, no post processing and ensemble
", "messages": [ { "id": "916334", "postDate": "07/05/2020 15:04:45", "content": "I was experimenting with single model predictions. There is concept of class weights [see this for more info] and got very unexpected results.\nModel: efficientnetb6\nDataset: 384x384 melanoma images [link]
\n\nRESULTS\n* class weights as {0: 0.5089, 1: 28.3613} [using formula] --> 0.887 [NOTEBOOK]\n* class weights as {0: 0.1, 1: 1} --> 0.885 [NOTEBOOK] \n* without class weights --> 0.82 [NOTEBOOK]
\n\nNOTE: No CV, very little Augmentation, no post processing and ensemble
", "rawMarkdown": "I was experimenting with single model predictions. There is concept of **_class weights_** [see [this](https://www.tensorflow.org/tutorials/structured_data/imbalanced_data#class_weights) for more info] and got very unexpected results.\n**Model:** efficientnetb6\n**Dataset:** 384x384 melanoma images [[link](https://www.kaggle.com/cdeotte/melanoma-384x384)]\n\n**RESULTS**\n* class weights as {0: 0.5089, 1: 28.3613} [using formula] --> 0.887 [[NOTEBOOK](https://www.kaggle.com/orionpax00/melanoma-detection-single-model-gpu-tpu-yapl?scriptVersionId=38125072)]\n* class weights as {0: 0.1, 1: 1} --> 0.885 [[NOTEBOOK](https://www.kaggle.com/orionpax00/melanoma-detection-single-model-gpu-tpu-yapl?scriptVersionId=38125072)] \n* without class weights --> 0.82 [[NOTEBOOK](https://www.kaggle.com/orionpax00/melanoma-detection-single-model-gpu-tpu-yapl?scriptVersionId=38127955)]\n\n**NOTE:** No CV, very little Augmentation, no post processing and ensemble", "votes": null }, { "id": "916531", "postDate": "07/05/2020 18:24:15", "content": "Due to the imbalance of data I think the class weights are necessary. Depending on the application it may be more desirable to have FP or FN. For example, my research for my Masters is focused on finding abnormal frames in a video of a wireless capsule endoscopy. To the physician its more important to not miss any abnormality than it is to have a few normal frames misclassified. So by varying the weights I can force more of that kind of error so the performance is reduced as far as accuracy is concerned, but for the application it is what's needed.
", "rawMarkdown": "Due to the imbalance of data I think the class weights are necessary. Depending on the application it may be more desirable to have FP or FN. For example, my research for my Masters is focused on finding abnormal frames in a video of a wireless capsule endoscopy. To the physician its more important to not miss any abnormality than it is to have a few normal frames misclassified. So by varying the weights I can force more of that kind of error so the performance is reduced as far as accuracy is concerned, but for the application it is what's needed.", "votes": null }, { "id": "916615", "postDate": "07/05/2020 20:59:36", "content": "Interesting, I have done some experiments with class weights but did not got relevant improvents, I have even tried some different weights configurations.
", "rawMarkdown": "Interesting, I have done some experiments with class weights but did not got relevant improvents, I have even tried some different weights configurations.", "votes": null }, { "id": "916900", "postDate": "07/06/2020 05:25:56", "content": "@dimitreoliveira If you are using class weights try to train it for more number of epochs (around 20 or 30).
", "rawMarkdown": "dimitreoliveira If you are using class weights try to train it for more number of epochs (around 20 or 30).", "votes": null } ], "comments": [ { "id": 916531, "author_name": "hmorera", "author_url": "", "post_date": "07/05/2020 18:24:15", "content": "Due to the imbalance of data I think the class weights are necessary. Depending on the application it may be more desirable to have FP or FN. For example, my research for my Masters is focused on finding abnormal frames in a video of a wireless capsule endoscopy. To the physician its more important to not miss any abnormality than it is to have a few normal frames misclassified. So by varying the weights I can force more of that kind of error so the performance is reduced as far as accuracy is concerned, but for the application it is what's needed.
", "votes": null, "replies": [] }, { "id": 916615, "author_name": "dimitreoliveira", "author_url": "", "post_date": "07/05/2020 20:59:36", "content": "Interesting, I have done some experiments with class weights but did not got relevant improvents, I have even tried some different weights configurations.
", "votes": null, "replies": [ { "id": 916900, "author_name": "orionpax00", "author_url": "", "post_date": "07/06/2020 05:25:56", "content": "@dimitreoliveira If you are using class weights try to train it for more number of epochs (around 20 or 30).
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "916334": "I was experimenting with single model predictions. There is concept of **_class weights_** [see [this](https://www.tensorflow.org/tutorials/structured_data/imbalanced_data#class_weights) for more info] and got very unexpected results.\n**Model:** efficientnetb6\n**Dataset:** 384x384 melanoma images [[link](https://www.kaggle.com/cdeotte/melanoma-384x384)]\n\n**RESULTS**\n* class weights as {0: 0.5089, 1: 28.3613} [using formula] --> 0.887 [[NOTEBOOK](https://www.kaggle.com/orionpax00/melanoma-detection-single-model-gpu-tpu-yapl?scriptVersionId=38125072)]\n* class weights as {0: 0.1, 1: 1} --> 0.885 [[NOTEBOOK](https://www.kaggle.com/orionpax00/melanoma-detection-single-model-gpu-tpu-yapl?scriptVersionId=38125072)] \n* without class weights --> 0.82 [[NOTEBOOK](https://www.kaggle.com/orionpax00/melanoma-detection-single-model-gpu-tpu-yapl?scriptVersionId=38127955)]\n\n**NOTE:** No CV, very little Augmentation, no post processing and ensemble", "916531": "Due to the imbalance of data I think the class weights are necessary. Depending on the application it may be more desirable to have FP or FN. For example, my research for my Masters is focused on finding abnormal frames in a video of a wireless capsule endoscopy. To the physician its more important to not miss any abnormality than it is to have a few normal frames misclassified. So by varying the weights I can force more of that kind of error so the performance is reduced as far as accuracy is concerned, but for the application it is what's needed.", "916615": "Interesting, I have done some experiments with class weights but did not got relevant improvents, I have even tried some different weights configurations.", "916900": "dimitreoliveira If you are using class weights try to train it for more number of epochs (around 20 or 30)." }, "source": "meta" }