{
  "id": 74721,
  "title": "Score no better than random, what did I do wrong",
  "url": "/competitions/histopathologic-cancer-detection/discussion/74721",
  "author_name": "Yifeng (Ethan) Zou",
  "post_date": "2018-12-14T19:40:17.360000",
  "votes": 3,
  "comment_count": 6,
  "views": 0,
  "content": "<p>The trained model I used to predict has a ~0.978 acc and ~0.975 val_acc. But the submission score is only ~0.494, just feels like random. I've read loads of posts and no luck. Literally stuck now. I believe it should be something wrong in the way I generate the prediction.</p>\n\n<p>Here's the code I use to generate prediction:\n```python\nimport numpy as np\nimport pandas as pd\nfrom keras.models import load_model\nfrom keras.preprocessing.image import ImageDataGenerator</p>\n\n<p>test_set = pd.read_csv(\"datasets/sample_submission.csv\").drop('label', axis=1)  # load image id\nmodel = load_model(\"checkpoints/e33a194_3/model_epoch002_valacc0.974607.h5\")  # load trained model</p>\n\n<h1>configure data generator</h1>\n\n<p>datagen = ImageDataGenerator()  # no data augmentation\ntest_datagen = datagen.flow_from_dataframe(test_set, directory=\"datasets/test\", has_ext=False, target_size=(96, 96), x_col='id', class_mode=None, batch_size=64, shuffle=False)\ntest_datagen.reset()</p>\n\n<h1>feed data from data generator and predict</h1>\n\n<p>test_result = model.predict_generator(test_datagen, steps=test_datagen.n // 64 + 1, verbose=1)\ntest_result = np.concatenate(test_result)  # flatten test_result</p>\n\n<h1>generate submission</h1>\n\n<p>submission = pd.concat([test_set, pd.Series(test_result, name='label')], axis=1)\nsubmission.to_csv(\"submission.csv\", index=False)\n<code>\nAnd the code I use to train the model, if it helps:\n</code>python\nimport pandas as pd\nfrom keras.callbacks import ModelCheckpoint, TensorBoard\nfrom keras.models import load_model\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom sklearn.model_selection import train_test_split</p>\n\n<p>from model import inception_resnet_v2</p>\n\n<p>rev_num = \"e33a194\"\nrun_num = 5</p>\n\n<p>train_dataset = pd.read_csv(\"datasets/train_labels_mod_75.csv\")  # load modified train_labels dataset\ntrain_set, dev_set = train_test_split(train_dataset, test_size=0.05, random_state=2333)  # split datasets\nif run_num == 0:  # upon first run, load untrained model\n    model = inception_resnet_v2()\nelse:  # otherwise, load previously trained model and continue to train\n    model = load_model(\"checkpoints/\" + rev_num + \"_\" + str(run_num - 1) + \"/model_epoch032.h5\")</p>\n\n<h1>configure data generator</h1>\n\n<p>datagen = ImageDataGenerator(rotation_range=30, horizontal_flip=True, vertical_flip=True)  # real-time data augmentation\ntrain_datagen = datagen.flow_from_dataframe(train_set, directory=\"datasets/train\", x_col='id', y_col='label', has_ext=False, target_size=(96, 96), class_mode='binary', batch_size=64)\ndev_datagen = datagen.flow_from_dataframe(dev_set, directory=\"datasets/train\", x_col='id', y_col='label', has_ext=False, target_size=(96, 96), class_mode='binary', batch_size=64, shuffle=False)</p>\n\n<h1>configure callbacks</h1>\n\n<p>ckpt = ModelCheckpoint(\"checkpoints/\" + rev_num + \"_\" + str(run_num) + \"/model_epoch{epoch:03d}_valacc{val_acc:f}.h5\", monitor='val_acc', verbose=1, save_best_only=True)\ntb = TensorBoard(log_dir=\"logs/\" + rev_num + \"_\" + str(run_num))</p>\n\n<h1>fit model with checkpoint and tensorboard callbacks and data augmentation</h1>\n\n<p>model.fit_generator(train_datagen, steps_per_epoch=train_datagen.n // 64, epochs=32, verbose=2, callbacks=[ckpt, tb], validation_data=dev_datagen, validation_steps=dev_datagen.n // 64)\nmodel.save(\"checkpoints/\" + rev_num + \"_\" + str(run_num) + \"/model_epoch.h5\")\n<code>\nModel, FYI:\n</code>python\nfrom keras.applications.inception_resnet_v2 import InceptionResNetV2\nfrom keras.layers import Input, Dense, Flatten\nfrom keras.models import Model</p>\n\n<p>def inception_resnet_v2(input_shape=(96, 96, 3)):\n    x_input = Input(input_shape)  # define the input as a tensor</p>\n\n<pre><code># add pre-implemented Inception-Resnet-V2 model\nbase_model = InceptionResNetV2(include_top=False, input_shape=input_shape)\nx = base_model(x_input)\n\n# output layer\nx = Flatten()(x)\nx = Dense(1, activation=\"sigmoid\")(x)\n\nmodel = Model(inputs=x_input, outputs=x, name='InceptionResNetV2')  # create model\nmodel.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])  # configure model\n\nreturn model\n</code></pre>\n\n<p>```</p>",
  "messages": [
    {
      "id": 439141,
      "postDate": "2018-12-14T19:40:17.360Z",
      "content": "<p>The trained model I used to predict has a ~0.978 acc and ~0.975 val_acc. But the submission score is only ~0.494, just feels like random. I've read loads of posts and no luck. Literally stuck now. I believe it should be something wrong in the way I generate the prediction.</p>\n\n<p>Here's the code I use to generate prediction:\n```python\nimport numpy as np\nimport pandas as pd\nfrom keras.models import load_model\nfrom keras.preprocessing.image import ImageDataGenerator</p>\n\n<p>test_set = pd.read_csv(\"datasets/sample_submission.csv\").drop('label', axis=1)  # load image id\nmodel = load_model(\"checkpoints/e33a194_3/model_epoch002_valacc0.974607.h5\")  # load trained model</p>\n\n<h1>configure data generator</h1>\n\n<p>datagen = ImageDataGenerator()  # no data augmentation\ntest_datagen = datagen.flow_from_dataframe(test_set, directory=\"datasets/test\", has_ext=False, target_size=(96, 96), x_col='id', class_mode=None, batch_size=64, shuffle=False)\ntest_datagen.reset()</p>\n\n<h1>feed data from data generator and predict</h1>\n\n<p>test_result = model.predict_generator(test_datagen, steps=test_datagen.n // 64 + 1, verbose=1)\ntest_result = np.concatenate(test_result)  # flatten test_result</p>\n\n<h1>generate submission</h1>\n\n<p>submission = pd.concat([test_set, pd.Series(test_result, name='label')], axis=1)\nsubmission.to_csv(\"submission.csv\", index=False)\n<code>\nAnd the code I use to train the model, if it helps:\n</code>python\nimport pandas as pd\nfrom keras.callbacks import ModelCheckpoint, TensorBoard\nfrom keras.models import load_model\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom sklearn.model_selection import train_test_split</p>\n\n<p>from model import inception_resnet_v2</p>\n\n<p>rev_num = \"e33a194\"\nrun_num = 5</p>\n\n<p>train_dataset = pd.read_csv(\"datasets/train_labels_mod_75.csv\")  # load modified train_labels dataset\ntrain_set, dev_set = train_test_split(train_dataset, test_size=0.05, random_state=2333)  # split datasets\nif run_num == 0:  # upon first run, load untrained model\n    model = inception_resnet_v2()\nelse:  # otherwise, load previously trained model and continue to train\n    model = load_model(\"checkpoints/\" + rev_num + \"_\" + str(run_num - 1) + \"/model_epoch032.h5\")</p>\n\n<h1>configure data generator</h1>\n\n<p>datagen = ImageDataGenerator(rotation_range=30, horizontal_flip=True, vertical_flip=True)  # real-time data augmentation\ntrain_datagen = datagen.flow_from_dataframe(train_set, directory=\"datasets/train\", x_col='id', y_col='label', has_ext=False, target_size=(96, 96), class_mode='binary', batch_size=64)\ndev_datagen = datagen.flow_from_dataframe(dev_set, directory=\"datasets/train\", x_col='id', y_col='label', has_ext=False, target_size=(96, 96), class_mode='binary', batch_size=64, shuffle=False)</p>\n\n<h1>configure callbacks</h1>\n\n<p>ckpt = ModelCheckpoint(\"checkpoints/\" + rev_num + \"_\" + str(run_num) + \"/model_epoch{epoch:03d}_valacc{val_acc:f}.h5\", monitor='val_acc', verbose=1, save_best_only=True)\ntb = TensorBoard(log_dir=\"logs/\" + rev_num + \"_\" + str(run_num))</p>\n\n<h1>fit model with checkpoint and tensorboard callbacks and data augmentation</h1>\n\n<p>model.fit_generator(train_datagen, steps_per_epoch=train_datagen.n // 64, epochs=32, verbose=2, callbacks=[ckpt, tb], validation_data=dev_datagen, validation_steps=dev_datagen.n // 64)\nmodel.save(\"checkpoints/\" + rev_num + \"_\" + str(run_num) + \"/model_epoch.h5\")\n<code>\nModel, FYI:\n</code>python\nfrom keras.applications.inception_resnet_v2 import InceptionResNetV2\nfrom keras.layers import Input, Dense, Flatten\nfrom keras.models import Model</p>\n\n<p>def inception_resnet_v2(input_shape=(96, 96, 3)):\n    x_input = Input(input_shape)  # define the input as a tensor</p>\n\n<pre><code># add pre-implemented Inception-Resnet-V2 model\nbase_model = InceptionResNetV2(include_top=False, input_shape=input_shape)\nx = base_model(x_input)\n\n# output layer\nx = Flatten()(x)\nx = Dense(1, activation=\"sigmoid\")(x)\n\nmodel = Model(inputs=x_input, outputs=x, name='InceptionResNetV2')  # create model\nmodel.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])  # configure model\n\nreturn model\n</code></pre>\n\n<p>```</p>",
      "rawMarkdown": "The trained model I used to predict has a ~0.978 acc and ~0.975 val_acc. But the submission score is only ~0.494, just feels like random. I've read loads of posts and no luck. Literally stuck now. I believe it should be something wrong in the way I generate the prediction.\n\nHere's the code I use to generate prediction:\n```python\nimport numpy as np\nimport pandas as pd\nfrom keras.models import load_model\nfrom keras.preprocessing.image import ImageDataGenerator\n\ntest_set = pd.read_csv(\"datasets/sample_submission.csv\").drop('label', axis=1)  # load image id\nmodel = load_model(\"checkpoints/e33a194_3/model_epoch002_valacc0.974607.h5\")  # load trained model\n\n# configure data generator\ndatagen = ImageDataGenerator()  # no data augmentation\ntest_datagen = datagen.flow_from_dataframe(test_set, directory=\"datasets/test\", has_ext=False, target_size=(96, 96), x_col='id', class_mode=None, batch_size=64, shuffle=False)\ntest_datagen.reset()\n\n# feed data from data generator and predict\ntest_result = model.predict_generator(test_datagen, steps=test_datagen.n // 64 + 1, verbose=1)\ntest_result = np.concatenate(test_result)  # flatten test_result\n\n# generate submission\nsubmission = pd.concat([test_set, pd.Series(test_result, name='label')], axis=1)\nsubmission.to_csv(\"submission.csv\", index=False)\n```\nAnd the code I use to train the model, if it helps:\n```python\nimport pandas as pd\nfrom keras.callbacks import ModelCheckpoint, TensorBoard\nfrom keras.models import load_model\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom sklearn.model_selection import train_test_split\n\nfrom model import inception_resnet_v2\n\nrev_num = \"e33a194\"\nrun_num = 5\n\ntrain_dataset = pd.read_csv(\"datasets/train_labels_mod_75.csv\")  # load modified train_labels dataset\ntrain_set, dev_set = train_test_split(train_dataset, test_size=0.05, random_state=2333)  # split datasets\nif run_num == 0:  # upon first run, load untrained model\n    model = inception_resnet_v2()\nelse:  # otherwise, load previously trained model and continue to train\n    model = load_model(\"checkpoints/\" + rev_num + \"_\" + str(run_num - 1) + \"/model_epoch032.h5\")\n\n# configure data generator\ndatagen = ImageDataGenerator(rotation_range=30, horizontal_flip=True, vertical_flip=True)  # real-time data augmentation\ntrain_datagen = datagen.flow_from_dataframe(train_set, directory=\"datasets/train\", x_col='id', y_col='label', has_ext=False, target_size=(96, 96), class_mode='binary', batch_size=64)\ndev_datagen = datagen.flow_from_dataframe(dev_set, directory=\"datasets/train\", x_col='id', y_col='label', has_ext=False, target_size=(96, 96), class_mode='binary', batch_size=64, shuffle=False)\n\n# configure callbacks\nckpt = ModelCheckpoint(\"checkpoints/\" + rev_num + \"_\" + str(run_num) + \"/model_epoch{epoch:03d}_valacc{val_acc:f}.h5\", monitor='val_acc', verbose=1, save_best_only=True)\ntb = TensorBoard(log_dir=\"logs/\" + rev_num + \"_\" + str(run_num))\n\n# fit model with checkpoint and tensorboard callbacks and data augmentation\nmodel.fit_generator(train_datagen, steps_per_epoch=train_datagen.n // 64, epochs=32, verbose=2, callbacks=[ckpt, tb], validation_data=dev_datagen, validation_steps=dev_datagen.n // 64)\nmodel.save(\"checkpoints/\" + rev_num + \"_\" + str(run_num) + \"/model_epoch.h5\")\n```\nModel, FYI:\n```python\nfrom keras.applications.inception_resnet_v2 import InceptionResNetV2\nfrom keras.layers import Input, Dense, Flatten\nfrom keras.models import Model\n\n\ndef inception_resnet_v2(input_shape=(96, 96, 3)):\n    x_input = Input(input_shape)  # define the input as a tensor\n\n    # add pre-implemented Inception-Resnet-V2 model\n    base_model = InceptionResNetV2(include_top=False, input_shape=input_shape)\n    x = base_model(x_input)\n\n    # output layer\n    x = Flatten()(x)\n    x = Dense(1, activation=\"sigmoid\")(x)\n\n    model = Model(inputs=x_input, outputs=x, name='InceptionResNetV2')  # create model\n    model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])  # configure model\n\n    return model\n```",
      "votes": 3
    },
    {
      "id": 470919,
      "postDate": "2019-02-13T19:00:37.660Z",
      "content": "<p>Test generator somehow shuffled the order of testing image. I just predict all images one by one and the problem is solved.</p>",
      "rawMarkdown": "Test generator somehow shuffled the order of testing image. I just predict all images one by one and the problem is solved."
    },
    {
      "id": 442255,
      "postDate": "2018-12-19T17:26:29.540Z",
      "content": "<p>Evaluation score is based on the AUC metric. Have you tried generating it?</p>\n\n<p>You can find it in my <a href=\"https://www.kaggle.com/greg115/histopathologic-cancer-detector-lb-0-958\">kernel</a></p>",
      "rawMarkdown": "Evaluation score is based on the AUC metric. Have you tried generating it?\n\nYou can find it in my [kernel][1]\n\n\n  [1]: https://www.kaggle.com/greg115/histopathologic-cancer-detector-lb-0-958",
      "replies": [
        {
          "id": 443923,
          "postDate": "2018-12-22T18:25:41.510Z",
          "content": "<p>Hi Greg, thanks for your reply! I've done the ROC validation plot as per your kernel. The plot is quite similar to yours and the AUC = 0.990.</p>",
          "rawMarkdown": "Hi Greg, thanks for your reply! I've done the ROC validation plot as per your kernel. The plot is quite similar to yours and the AUC = 0.990."
        },
        {
          "id": 443926,
          "postDate": "2018-12-22T18:37:02.997Z",
          "content": "<p>I think that your testing dataGenerator might have invalid order of elements, thus you get 0.5 random accuracy. I would suggest focusing on that and making sure that your testing set has a correct order. </p>\n\n<p>I would try creating a separate ImageDataGenerator() instance instead of reusing the one used for training.</p>",
          "rawMarkdown": "I think that your testing dataGenerator might have invalid order of elements, thus you get 0.5 random accuracy. I would suggest focusing on that and making sure that your testing set has a correct order. \n\nI would try creating a separate ImageDataGenerator() instance instead of reusing the one used for training."
        },
        {
          "id": 443951,
          "postDate": "2018-12-22T19:21:50.247Z",
          "rawMarkdown": "",
          "isDeleted": true
        }
      ]
    },
    {
      "id": 439678,
      "postDate": "2018-12-16T04:19:58.813Z",
      "content": "<p>Same issue</p>",
      "rawMarkdown": "Same issue"
    }
  ],
  "comments": [
    {
      "id": 470919,
      "author_name": "ivanwang",
      "author_url": "",
      "post_date": "2019-02-13T19:00:37.660000",
      "content": "<p>Test generator somehow shuffled the order of testing image. I just predict all images one by one and the problem is solved.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 442255,
      "author_name": "greg",
      "author_url": "",
      "post_date": "2018-12-19T17:26:29.540000",
      "content": "<p>Evaluation score is based on the AUC metric. Have you tried generating it?</p>\n\n<p>You can find it in my <a href=\"https://www.kaggle.com/greg115/histopathologic-cancer-detector-lb-0-958\">kernel</a></p>",
      "votes": 0,
      "replies": [
        {
          "id": 443923,
          "author_name": "Yifeng (Ethan) Zou",
          "author_url": "",
          "post_date": "2018-12-22T18:25:41.510000",
          "content": "<p>Hi Greg, thanks for your reply! I've done the ROC validation plot as per your kernel. The plot is quite similar to yours and the AUC = 0.990.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 443926,
          "author_name": "greg",
          "author_url": "",
          "post_date": "2018-12-22T18:37:02.997000",
          "content": "<p>I think that your testing dataGenerator might have invalid order of elements, thus you get 0.5 random accuracy. I would suggest focusing on that and making sure that your testing set has a correct order. </p>\n\n<p>I would try creating a separate ImageDataGenerator() instance instead of reusing the one used for training.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 443951,
          "author_name": "",
          "author_url": "",
          "post_date": "2018-12-22T19:21:50.247000",
          "content": "",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 439678,
      "author_name": "Winston Van",
      "author_url": "",
      "post_date": "2018-12-16T04:19:58.813000",
      "content": "<p>Same issue</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "439141": "The trained model I used to predict has a ~0.978 acc and ~0.975 val_acc. But the submission score is only ~0.494, just feels like random. I've read loads of posts and no luck. Literally stuck now. I believe it should be something wrong in the way I generate the prediction.\n\nHere's the code I use to generate prediction:\n```python\nimport numpy as np\nimport pandas as pd\nfrom keras.models import load_model\nfrom keras.preprocessing.image import ImageDataGenerator\n\ntest_set = pd.read_csv(\"datasets/sample_submission.csv\").drop('label', axis=1)  # load image id\nmodel = load_model(\"checkpoints/e33a194_3/model_epoch002_valacc0.974607.h5\")  # load trained model\n\n# configure data generator\ndatagen = ImageDataGenerator()  # no data augmentation\ntest_datagen = datagen.flow_from_dataframe(test_set, directory=\"datasets/test\", has_ext=False, target_size=(96, 96), x_col='id', class_mode=None, batch_size=64, shuffle=False)\ntest_datagen.reset()\n\n# feed data from data generator and predict\ntest_result = model.predict_generator(test_datagen, steps=test_datagen.n // 64 + 1, verbose=1)\ntest_result = np.concatenate(test_result)  # flatten test_result\n\n# generate submission\nsubmission = pd.concat([test_set, pd.Series(test_result, name='label')], axis=1)\nsubmission.to_csv(\"submission.csv\", index=False)\n```\nAnd the code I use to train the model, if it helps:\n```python\nimport pandas as pd\nfrom keras.callbacks import ModelCheckpoint, TensorBoard\nfrom keras.models import load_model\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom sklearn.model_selection import train_test_split\n\nfrom model import inception_resnet_v2\n\nrev_num = \"e33a194\"\nrun_num = 5\n\ntrain_dataset = pd.read_csv(\"datasets/train_labels_mod_75.csv\")  # load modified train_labels dataset\ntrain_set, dev_set = train_test_split(train_dataset, test_size=0.05, random_state=2333)  # split datasets\nif run_num == 0:  # upon first run, load untrained model\n    model = inception_resnet_v2()\nelse:  # otherwise, load previously trained model and continue to train\n    model = load_model(\"checkpoints/\" + rev_num + \"_\" + str(run_num - 1) + \"/model_epoch032.h5\")\n\n# configure data generator\ndatagen = ImageDataGenerator(rotation_range=30, horizontal_flip=True, vertical_flip=True)  # real-time data augmentation\ntrain_datagen = datagen.flow_from_dataframe(train_set, directory=\"datasets/train\", x_col='id', y_col='label', has_ext=False, target_size=(96, 96), class_mode='binary', batch_size=64)\ndev_datagen = datagen.flow_from_dataframe(dev_set, directory=\"datasets/train\", x_col='id', y_col='label', has_ext=False, target_size=(96, 96), class_mode='binary', batch_size=64, shuffle=False)\n\n# configure callbacks\nckpt = ModelCheckpoint(\"checkpoints/\" + rev_num + \"_\" + str(run_num) + \"/model_epoch{epoch:03d}_valacc{val_acc:f}.h5\", monitor='val_acc', verbose=1, save_best_only=True)\ntb = TensorBoard(log_dir=\"logs/\" + rev_num + \"_\" + str(run_num))\n\n# fit model with checkpoint and tensorboard callbacks and data augmentation\nmodel.fit_generator(train_datagen, steps_per_epoch=train_datagen.n // 64, epochs=32, verbose=2, callbacks=[ckpt, tb], validation_data=dev_datagen, validation_steps=dev_datagen.n // 64)\nmodel.save(\"checkpoints/\" + rev_num + \"_\" + str(run_num) + \"/model_epoch.h5\")\n```\nModel, FYI:\n```python\nfrom keras.applications.inception_resnet_v2 import InceptionResNetV2\nfrom keras.layers import Input, Dense, Flatten\nfrom keras.models import Model\n\n\ndef inception_resnet_v2(input_shape=(96, 96, 3)):\n    x_input = Input(input_shape)  # define the input as a tensor\n\n    # add pre-implemented Inception-Resnet-V2 model\n    base_model = InceptionResNetV2(include_top=False, input_shape=input_shape)\n    x = base_model(x_input)\n\n    # output layer\n    x = Flatten()(x)\n    x = Dense(1, activation=\"sigmoid\")(x)\n\n    model = Model(inputs=x_input, outputs=x, name='InceptionResNetV2')  # create model\n    model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])  # configure model\n\n    return model\n```",
    "470919": "Test generator somehow shuffled the order of testing image. I just predict all images one by one and the problem is solved.",
    "442255": "Evaluation score is based on the AUC metric. Have you tried generating it?\n\nYou can find it in my [kernel][1]\n\n\n  [1]: https://www.kaggle.com/greg115/histopathologic-cancer-detector-lb-0-958",
    "439678": "Same issue"
  }
}