![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2534874%2F1abe49ab1f1fa1fcecd1c963b4204d12%2FCCAF50A6-F342-4B0B-BDCE-FB93DF33BECF.png.jpg?generation=1593107241931245&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2534874%2F205177787e8a728947ce7fc2de8ba1b5%2FDF009D6B-ECA4-4F37-BC9A-91E454B8BD8A.png.jpg?generation=1593107240933541&alt=media\")
Hello, I am a CNN newbie and I've been trying to solve my problem of very low validation score problem. I am using EfficentNetB0 and I have augmented my data like this:\n\n
As you can see I have augmented my train (malignant and benign both) dataset and also added an artificial circle to remove vignette. I have also done the same to my test dataset. \nAt training I am using EfficentNetB0 with 158k training parameters and getting a training accuracy of 0.94, training loss of INFINITYY and consistent validation accuracy around 0.52. I am clueless about why am I getting so low validation score
", "messages": [ { "id": "901808", "postDate": "06/25/2020 17:52:40", "content": "Hello, I am a CNN newbie and I've been trying to solve my problem of very low validation score problem. I am using EfficentNetB0 and I have augmented my data like this:\n\n
As you can see I have augmented my train (malignant and benign both) dataset and also added an artificial circle to remove vignette. I have also done the same to my test dataset. \nAt training I am using EfficentNetB0 with 158k training parameters and getting a training accuracy of 0.94, training loss of INFINITYY and consistent validation accuracy around 0.52. I am clueless about why am I getting so low validation score
", "rawMarkdown": "Hello, I am a CNN newbie and I've been trying to solve my problem of very low validation score problem. I am using EfficentNetB0 and I have augmented my data like this:\n\n\n\nAs you can see I have augmented my train (malignant and benign both) dataset and also added an artificial circle to remove vignette. I have also done the same to my test dataset. \nAt training I am using EfficentNetB0 with 158k training parameters and getting a training accuracy of 0.94, training loss of INFINITYY and consistent validation accuracy around 0.52. I am clueless about why am I getting so low validation score", "votes": null }, { "id": "902041", "postDate": "06/25/2020 22:01:03", "content": "Hi, @khizarhussain , try to change your loss function
\n\nPlease read this article , it could change your résulte:\n“Handling Imbalanced Datasets in Deep Learning” de George Seif https://link.medium.com/96nMzcjfC7
", "rawMarkdown": "Hi, @khizarhussain , try to change your loss function\n\nPlease read this article , it could change your résulte:\n“Handling Imbalanced Datasets in Deep Learning” de George Seif https://link.medium.com/96nMzcjfC7", "votes": null }, { "id": "902068", "postDate": "06/25/2020 22:47:38", "content": "Hi, @syphax93 thanks for replying. I read the article thoroughly and used the focal_loss method as mentioned by the author.\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2534874%2Fb153fd7850ea97a9f1696c6cc57e8030%2F9A4816EB-D1B1-4F88-BE38-2303641CE719.png.jpg?generation=1593125148380920&alt=media\")
\nead \nDespite doing this I am still getting 0.5 Validation Accuracy :/\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2534874%2F80b3e37ea40f893341c70c5ba262b056%2F2F064923-347C-4EEA-B7E9-8FE37DA32C2F.png.jpg?generation=1593125255425496&alt=media\")
Also, now I am getting infinite validation loss too
\n\nThis is my entire code:
\n\nfrom imutils import paths\nfrom sklearn.preprocessing import LabelEncoder\nfrom keras.utils import np_utils\nfrom keras.optimizers import Adam\nfrom keras import models, layers, optimizers\nfrom keras import backend as K\nfrom sklearn.utils import shuffle\nimport cv2\nimport numpy as np\nimport os\nimport matplotlib.pyplot as plt\nfrom sklearn.preprocessing import label_binarize\nfrom keras.applications.mobilenet_v2 import MobileNetV2 as mblv2 \nfrom keras.layers import AveragePooling2D\nfrom keras.layers import Dropout\nfrom keras.layers import Flatten\nfrom keras.layers import Dense\nfrom keras.models import Model\nimport tensorflow as tf\nimport efficientnet.keras as efn\nfrom sklearn.model_selection import train_test_split\nfrom keras.applications.mobilenet_v2 import decode_predictions, preprocess_input\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\n\nK.clear_session()\nimage_rows = 256\nimage_columns = 256\nfilter_size = 3\nLR = 1e-4\nBS = 32 \nEPOCHS = 20 \nimagePaths = list(paths.list_images('org_dataset'))\nglobal data\nglobal labels\ndata = []\nlabels = []\n\n\ndef focal_loss(y_true, y_pred):\n gamma = 2.0\n alpha = 0.25\n pt_1 = tf.where(tf.equal(y_true, 1), y_pred, tf.ones_like(y_pred))\n pt_0 = tf.where(tf.equal(y_true, 0), y_pred, tf.zeros_like(y_pred))\n return -K.sum(alpha * K.pow(1. - pt_1, gamma) * K.log(pt_1)) - K.sum(\n (1 - alpha) * K.pow(pt_0, gamma) * K.log(1. - pt_0))\n\n\ndef generate_data_and_labels():\n global data\n global labels\n for imagePath in imagePaths:\n label = imagePath.split(os.path.sep)[-2]\n image = cv2.imread(imagePath)\n image = cv2.resize(image, (256, 256))\n data.append(image)\n labels.append(label)\n le = LabelEncoder()\n labels = le.fit_transform(labels)\n labels = np_utils.to_categorical(labels, 2)\n labels = np.array(labels)\n data = np.array(data)\n\n\ndef construct_model():\n global data\n global labels\n\n base_model = efn.EfficientNetB0(weights='imagenet', include_top=False,\n input_tensor=layers.Input(shape=(256, 256, 3)))\n\n head_model = base_model.output\n head_model = AveragePooling2D(pool_size=(7, 7))(head_model)\n head_model = Flatten(name=\"flatten\")(head_model)\n head_model = Dense(128, activation=\"relu\")(head_model)\n head_model = Dropout(0.5)(head_model)\n head_model = Dense(2, activation=\"softmax\")(head_model)\n\n model = Model(inputs=base_model.input, outputs=head_model)\n\n for layer in base_model.layers:\n layer.trainable = False\n\n model.summary()\n\n adam = Adam(lr=0.0001)\n model.compile(loss=[focal_loss], metrics=[\"accuracy\"], optimizer=adam)\n\n data, labels = shuffle(data, labels)\n (trainX, testX, trainY, testY) = train_test_split(data, labels,\n test_size=0.25, random_state=42)\n H = model.fit(\n x=trainX, y=trainY,\n validation_data=(testX, testY),\n epochs=EPOCHS,\n )\n\n model.save('extended_dataset1_both_aug_vignette_efficentnet.hdf5')\n print('Model saved')\n\n\ndef build_cnn():\n generate_data_and_labels()\n construct_model()\n\n\nbuild_cnn()\nWhen I get 0.5 I always suspect that I have an error that is resulting in a coin flip for validation.
\n\nWill look more at your code later, but would suspect a code error in your shuffle.
", "rawMarkdown": "When I get 0.5 I always suspect that I have an error that is resulting in a coin flip for validation. \n\nWill look more at your code later, but would suspect a code error in your shuffle.", "votes": null }, { "id": "902396", "postDate": "06/26/2020 05:48:05", "content": "I dont think that shufftle is the problem because then i'd be getting really low training accuracy as well...Can you please elaborate ?
", "rawMarkdown": "I dont think that shufftle is the problem because then i'd be getting really low training accuracy as well...Can you please elaborate ?", "votes": null }, { "id": "903419", "postDate": "06/26/2020 20:21:13", "content": "OK - try reducing your dropout value.
", "rawMarkdown": "OK - try reducing your dropout value.", "votes": null }, { "id": "904591", "postDate": "06/27/2020 18:35:43", "content": "Actually the problem in your code is that we use various augmentation during training time as we made our model learn very insightful features using these augs but we do not use these augs during test as the model may not be able to predicts validation images properly the reason behind it is that\n1.)We apply different augmentation according to probability\n2.)You are using validation images like test images to measure performance of your model
", "rawMarkdown": "Actually the problem in your code is that we use various augmentation during training time as we made our model learn very insightful features using these augs but we do not use these augs during test as the model may not be able to predicts validation images properly the reason behind it is that\n1.)We apply different augmentation according to probability\n2.)You are using validation images like test images to measure performance of your model", "votes": null }, { "id": "904842", "postDate": "06/28/2020 02:14:59", "content": "Working on another discussion topic using focal loss - the learning rate needed to be much smaller using focal loss than it was for bce. Try dropping LR by another order or two.
", "rawMarkdown": "Working on another discussion topic using focal loss - the learning rate needed to be much smaller using focal loss than it was for bce. Try dropping LR by another order or two.", "votes": null } ], "comments": [ { "id": 902041, "author_name": "syphax93", "author_url": "", "post_date": "06/25/2020 22:01:03", "content": "Hi, @khizarhussain , try to change your loss function
\n\nPlease read this article , it could change your résulte:\n“Handling Imbalanced Datasets in Deep Learning” de George Seif https://link.medium.com/96nMzcjfC7
", "votes": null, "replies": [ { "id": 902068, "author_name": "khizarhussain", "author_url": "", "post_date": "06/25/2020 22:47:38", "content": "Hi, @syphax93 thanks for replying. I read the article thoroughly and used the focal_loss method as mentioned by the author.\n\nead \nDespite doing this I am still getting 0.5 Validation Accuracy :/\n
Also, now I am getting infinite validation loss too
\n\nThis is my entire code:
\n\nfrom imutils import paths\nfrom sklearn.preprocessing import LabelEncoder\nfrom keras.utils import np_utils\nfrom keras.optimizers import Adam\nfrom keras import models, layers, optimizers\nfrom keras import backend as K\nfrom sklearn.utils import shuffle\nimport cv2\nimport numpy as np\nimport os\nimport matplotlib.pyplot as plt\nfrom sklearn.preprocessing import label_binarize\nfrom keras.applications.mobilenet_v2 import MobileNetV2 as mblv2 \nfrom keras.layers import AveragePooling2D\nfrom keras.layers import Dropout\nfrom keras.layers import Flatten\nfrom keras.layers import Dense\nfrom keras.models import Model\nimport tensorflow as tf\nimport efficientnet.keras as efn\nfrom sklearn.model_selection import train_test_split\nfrom keras.applications.mobilenet_v2 import decode_predictions, preprocess_input\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\n\nK.clear_session()\nimage_rows = 256\nimage_columns = 256\nfilter_size = 3\nLR = 1e-4\nBS = 32 \nEPOCHS = 20 \nimagePaths = list(paths.list_images('org_dataset'))\nglobal data\nglobal labels\ndata = []\nlabels = []\n\n\ndef focal_loss(y_true, y_pred):\n gamma = 2.0\n alpha = 0.25\n pt_1 = tf.where(tf.equal(y_true, 1), y_pred, tf.ones_like(y_pred))\n pt_0 = tf.where(tf.equal(y_true, 0), y_pred, tf.zeros_like(y_pred))\n return -K.sum(alpha * K.pow(1. - pt_1, gamma) * K.log(pt_1)) - K.sum(\n (1 - alpha) * K.pow(pt_0, gamma) * K.log(1. - pt_0))\n\n\ndef generate_data_and_labels():\n global data\n global labels\n for imagePath in imagePaths:\n label = imagePath.split(os.path.sep)[-2]\n image = cv2.imread(imagePath)\n image = cv2.resize(image, (256, 256))\n data.append(image)\n labels.append(label)\n le = LabelEncoder()\n labels = le.fit_transform(labels)\n labels = np_utils.to_categorical(labels, 2)\n labels = np.array(labels)\n data = np.array(data)\n\n\ndef construct_model():\n global data\n global labels\n\n base_model = efn.EfficientNetB0(weights='imagenet', include_top=False,\n input_tensor=layers.Input(shape=(256, 256, 3)))\n\n head_model = base_model.output\n head_model = AveragePooling2D(pool_size=(7, 7))(head_model)\n head_model = Flatten(name=\"flatten\")(head_model)\n head_model = Dense(128, activation=\"relu\")(head_model)\n head_model = Dropout(0.5)(head_model)\n head_model = Dense(2, activation=\"softmax\")(head_model)\n\n model = Model(inputs=base_model.input, outputs=head_model)\n\n for layer in base_model.layers:\n layer.trainable = False\n\n model.summary()\n\n adam = Adam(lr=0.0001)\n model.compile(loss=[focal_loss], metrics=[\"accuracy\"], optimizer=adam)\n\n data, labels = shuffle(data, labels)\n (trainX, testX, trainY, testY) = train_test_split(data, labels,\n test_size=0.25, random_state=42)\n H = model.fit(\n x=trainX, y=trainY,\n validation_data=(testX, testY),\n epochs=EPOCHS,\n )\n\n model.save('extended_dataset1_both_aug_vignette_efficentnet.hdf5')\n print('Model saved')\n\n\ndef build_cnn():\n generate_data_and_labels()\n construct_model()\n\n\nbuild_cnn()\nWhen I get 0.5 I always suspect that I have an error that is resulting in a coin flip for validation.
\n\nWill look more at your code later, but would suspect a code error in your shuffle.
", "votes": null, "replies": [ { "id": 902396, "author_name": "khizarhussain", "author_url": "", "post_date": "06/26/2020 05:48:05", "content": "I dont think that shufftle is the problem because then i'd be getting really low training accuracy as well...Can you please elaborate ?
", "votes": null, "replies": [] }, { "id": 903419, "author_name": "pcjimmmy", "author_url": "", "post_date": "06/26/2020 20:21:13", "content": "OK - try reducing your dropout value.
", "votes": null, "replies": [] }, { "id": 904842, "author_name": "pcjimmmy", "author_url": "", "post_date": "06/28/2020 02:14:59", "content": "Working on another discussion topic using focal loss - the learning rate needed to be much smaller using focal loss than it was for bce. Try dropping LR by another order or two.
", "votes": null, "replies": [] } ] }, { "id": 904591, "author_name": "", "author_url": "", "post_date": "06/27/2020 18:35:43", "content": "Actually the problem in your code is that we use various augmentation during training time as we made our model learn very insightful features using these augs but we do not use these augs during test as the model may not be able to predicts validation images properly the reason behind it is that\n1.)We apply different augmentation according to probability\n2.)You are using validation images like test images to measure performance of your model
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "901808": "Hello, I am a CNN newbie and I've been trying to solve my problem of very low validation score problem. I am using EfficentNetB0 and I have augmented my data like this:\n\n\n\nAs you can see I have augmented my train (malignant and benign both) dataset and also added an artificial circle to remove vignette. I have also done the same to my test dataset. \nAt training I am using EfficentNetB0 with 158k training parameters and getting a training accuracy of 0.94, training loss of INFINITYY and consistent validation accuracy around 0.52. I am clueless about why am I getting so low validation score", "902041": "Hi, @khizarhussain , try to change your loss function\n\nPlease read this article , it could change your résulte:\n“Handling Imbalanced Datasets in Deep Learning” de George Seif https://link.medium.com/96nMzcjfC7", "902068": "Hi, @syphax93 thanks for replying. I read the article thoroughly and used the focal_loss method as mentioned by the author.\n\nead \nDespite doing this I am still getting 0.5 Validation Accuracy :/\n\n\nAlso, now I am getting infinite validation loss too\n\nThis is my entire code:\n\n \n from imutils import paths\n from sklearn.preprocessing import LabelEncoder\n from keras.utils import np_utils\n from keras.optimizers import Adam\n from keras import models, layers, optimizers\n from keras import backend as K\n from sklearn.utils import shuffle\n import cv2\n import numpy as np\n import os\n import matplotlib.pyplot as plt\n from sklearn.preprocessing import label_binarize\n from keras.applications.mobilenet_v2 import MobileNetV2 as mblv2 \n from keras.layers import AveragePooling2D\n from keras.layers import Dropout\n from keras.layers import Flatten\n from keras.layers import Dense\n from keras.models import Model\n import tensorflow as tf\n import efficientnet.keras as efn\n from sklearn.model_selection import train_test_split\n from keras.applications.mobilenet_v2 import decode_predictions, preprocess_input\n from tensorflow.keras.preprocessing.image import ImageDataGenerator\n \n K.clear_session()\n image_rows = 256\n image_columns = 256\n filter_size = 3\n LR = 1e-4\n BS = 32 \n EPOCHS = 20 \n imagePaths = list(paths.list_images('org_dataset'))\n global data\n global labels\n data = []\n labels = []\n \n \n def focal_loss(y_true, y_pred):\n gamma = 2.0\n alpha = 0.25\n pt_1 = tf.where(tf.equal(y_true, 1), y_pred, tf.ones_like(y_pred))\n pt_0 = tf.where(tf.equal(y_true, 0), y_pred, tf.zeros_like(y_pred))\n return -K.sum(alpha * K.pow(1. - pt_1, gamma) * K.log(pt_1)) - K.sum(\n (1 - alpha) * K.pow(pt_0, gamma) * K.log(1. - pt_0))\n \n \n def generate_data_and_labels():\n global data\n global labels\n for imagePath in imagePaths:\n label = imagePath.split(os.path.sep)[-2]\n image = cv2.imread(imagePath)\n image = cv2.resize(image, (256, 256))\n data.append(image)\n labels.append(label)\n le = LabelEncoder()\n labels = le.fit_transform(labels)\n labels = np_utils.to_categorical(labels, 2)\n labels = np.array(labels)\n data = np.array(data)\n \n \n def construct_model():\n global data\n global labels\n \n base_model = efn.EfficientNetB0(weights='imagenet', include_top=False,\n input_tensor=layers.Input(shape=(256, 256, 3)))\n \n head_model = base_model.output\n head_model = AveragePooling2D(pool_size=(7, 7))(head_model)\n head_model = Flatten(name=\"flatten\")(head_model)\n head_model = Dense(128, activation=\"relu\")(head_model)\n head_model = Dropout(0.5)(head_model)\n head_model = Dense(2, activation=\"softmax\")(head_model)\n \n model = Model(inputs=base_model.input, outputs=head_model)\n \n for layer in base_model.layers:\n layer.trainable = False\n \n model.summary()\n \n adam = Adam(lr=0.0001)\n model.compile(loss=[focal_loss], metrics=[\"accuracy\"], optimizer=adam)\n \n data, labels = shuffle(data, labels)\n (trainX, testX, trainY, testY) = train_test_split(data, labels,\n test_size=0.25, random_state=42)\n H = model.fit(\n x=trainX, y=trainY,\n validation_data=(testX, testY),\n epochs=EPOCHS,\n )\n \n model.save('extended_dataset1_both_aug_vignette_efficentnet.hdf5')\n print('Model saved')\n \n \n def build_cnn():\n generate_data_and_labels()\n construct_model()\n \n \n build_cnn()", "902098": "When I get 0.5 I always suspect that I have an error that is resulting in a coin flip for validation. \n\nWill look more at your code later, but would suspect a code error in your shuffle.", "902396": "I dont think that shufftle is the problem because then i'd be getting really low training accuracy as well...Can you please elaborate ?", "903419": "OK - try reducing your dropout value.", "904591": "Actually the problem in your code is that we use various augmentation during training time as we made our model learn very insightful features using these augs but we do not use these augs during test as the model may not be able to predicts validation images properly the reason behind it is that\n1.)We apply different augmentation according to probability\n2.)You are using validation images like test images to measure performance of your model", "904842": "Working on another discussion topic using focal loss - the learning rate needed to be much smaller using focal loss than it was for bce. Try dropping LR by another order or two." }, "source": "meta" }