{
  "id": 76095,
  "title": "In inception_v3 of keras,Why is it so different in training and predicting?",
  "url": "/competitions/human-protein-atlas-image-classification/discussion/76095",
  "author_name": "",
  "post_date": "2018-12-29T08:51:01.928479900Z",
  "votes": null,
  "comment_count": 2,
  "views": 0,
  "content": "<p>First of all, thank you for answering my questions.\nThis is my first time to use transfer learning of keras,I have a question about training and predicting.Why are the accuracy and f1 score of training and validation so different???</p>\n\n<pre><code>Epoch 4/5\n1500/1500 [==============================] - 23s 16ms/step - loss: 1.3653 - \ncategorical_accuracy: 0.3387 - binary_accuracy: 0.0664 - f1_keras: 0.1578 - val_loss: 3.0188 - \nval_categorical_accuracy: 0.1500 - val_binary_accuracy: 0.0601 - val_f1_keras: 0.0422\nEpoch 5/5\n1500/1500 [==============================] - 23s 15ms/step - loss: 1.1838 - \ncategorical_accuracy: 0.3800 - binary_accuracy: 0.0607 - f1_keras: 0.1995 - val_loss: 3.8084 - \nval_categorical_accuracy: 0.0914 - val_binary_accuracy: 0.0455 - val_f1_keras: 0.0299\n</code></pre>\n\n<p>Even if I use the same small training data set and validation data set, I get a similar result.</p>\n\n<pre><code>model.fit(x=X_train, y=Y_train, batch_size=32, epochs=5, validation_data=(X_train, Y_train))\nEpoch 4/5\n1500/1500 [==============================] - 31s 21ms/step - loss: 1.4078 - \ncategorical_accuracy: 0.3367 - binary_accuracy: 0.0622 - f1_keras: 0.1637 - val_loss: 2.8940 - \nval_categorical_accuracy: 0.2060 - val_binary_accuracy: 0.0954 - val_f1_keras: 0.0691\nEpoch 5/5\n1500/1500 [==============================] - 31s 21ms/step - loss: 1.2215 - \ncategorical_accuracy: 0.3980 - binary_accuracy: 0.0629 - f1_keras: 0.2194 - val_loss: 3.5152 - \nval_categorical_accuracy: 0.0720 - val_binary_accuracy: 0.0700 - val_f1_keras: 0.0556\n</code></pre>",
  "messages": [
    {
      "id": "447161",
      "postDate": "12/29/2018 08:51:01",
      "content": "<p>First of all, thank you for answering my questions.\nThis is my first time to use transfer learning of keras,I have a question about training and predicting.Why are the accuracy and f1 score of training and validation so different???</p>\n\n<pre><code>Epoch 4/5\n1500/1500 [==============================] - 23s 16ms/step - loss: 1.3653 - \ncategorical_accuracy: 0.3387 - binary_accuracy: 0.0664 - f1_keras: 0.1578 - val_loss: 3.0188 - \nval_categorical_accuracy: 0.1500 - val_binary_accuracy: 0.0601 - val_f1_keras: 0.0422\nEpoch 5/5\n1500/1500 [==============================] - 23s 15ms/step - loss: 1.1838 - \ncategorical_accuracy: 0.3800 - binary_accuracy: 0.0607 - f1_keras: 0.1995 - val_loss: 3.8084 - \nval_categorical_accuracy: 0.0914 - val_binary_accuracy: 0.0455 - val_f1_keras: 0.0299\n</code></pre>\n\n<p>Even if I use the same small training data set and validation data set, I get a similar result.</p>\n\n<pre><code>model.fit(x=X_train, y=Y_train, batch_size=32, epochs=5, validation_data=(X_train, Y_train))\nEpoch 4/5\n1500/1500 [==============================] - 31s 21ms/step - loss: 1.4078 - \ncategorical_accuracy: 0.3367 - binary_accuracy: 0.0622 - f1_keras: 0.1637 - val_loss: 2.8940 - \nval_categorical_accuracy: 0.2060 - val_binary_accuracy: 0.0954 - val_f1_keras: 0.0691\nEpoch 5/5\n1500/1500 [==============================] - 31s 21ms/step - loss: 1.2215 - \ncategorical_accuracy: 0.3980 - binary_accuracy: 0.0629 - f1_keras: 0.2194 - val_loss: 3.5152 - \nval_categorical_accuracy: 0.0720 - val_binary_accuracy: 0.0700 - val_f1_keras: 0.0556\n</code></pre>",
      "rawMarkdown": "First of all, thank you for answering my questions.\nThis is my first time to use transfer learning of keras,I have a question about training and predicting.Why are the accuracy and f1 score of training and validation so different???\n\n    Epoch 4/5\n    1500/1500 [==============================] - 23s 16ms/step - loss: 1.3653 - \n    categorical_accuracy: 0.3387 - binary_accuracy: 0.0664 - f1_keras: 0.1578 - val_loss: 3.0188 - \n    val_categorical_accuracy: 0.1500 - val_binary_accuracy: 0.0601 - val_f1_keras: 0.0422\n    Epoch 5/5\n    1500/1500 [==============================] - 23s 15ms/step - loss: 1.1838 - \n    categorical_accuracy: 0.3800 - binary_accuracy: 0.0607 - f1_keras: 0.1995 - val_loss: 3.8084 - \n    val_categorical_accuracy: 0.0914 - val_binary_accuracy: 0.0455 - val_f1_keras: 0.0299\n\nEven if I use the same small training data set and validation data set, I get a similar result.\n\n    model.fit(x=X_train, y=Y_train, batch_size=32, epochs=5, validation_data=(X_train, Y_train))\n    Epoch 4/5\n    1500/1500 [==============================] - 31s 21ms/step - loss: 1.4078 - \n    categorical_accuracy: 0.3367 - binary_accuracy: 0.0622 - f1_keras: 0.1637 - val_loss: 2.8940 - \n    val_categorical_accuracy: 0.2060 - val_binary_accuracy: 0.0954 - val_f1_keras: 0.0691\n    Epoch 5/5\n    1500/1500 [==============================] - 31s 21ms/step - loss: 1.2215 - \n    categorical_accuracy: 0.3980 - binary_accuracy: 0.0629 - f1_keras: 0.2194 - val_loss: 3.5152 - \n    val_categorical_accuracy: 0.0720 - val_binary_accuracy: 0.0700 - val_f1_keras: 0.0556",
      "votes": null
    },
    {
      "id": "447586",
      "postDate": "12/30/2018 05:52:35",
      "content": "<p>I would think that your model overfits a lot, but accuracy 0.07 looks extremely low. Even if I replace the head of the model and train only it, the accuracy already is ~0.93. There can be several issues: some problem with data loading(1), training of entire model with replaced head with using too high learning rate and without freezing the bottom layers(2), small validation dataset and statistical noise (3). Since I do not using Keras, I could not provide more details on the above issues.</p>",
      "rawMarkdown": "I would think that your model overfits a lot, but accuracy 0.07 looks extremely low. Even if I replace the head of the model and train only it, the accuracy already is ~0.93. There can be several issues: some problem with data loading(1), training of entire model with replaced head with using too high learning rate and without freezing the bottom layers(2), small validation dataset and statistical noise (3). Since I do not using Keras, I could not provide more details on the above issues.",
      "votes": null
    },
    {
      "id": "447600",
      "postDate": "12/30/2018 06:39:51",
      "content": "<p>Thank you very much for your reply.I know you.My focal loss is  borrowed from you.I found the  solution to the problem.Due to I froze all the layers of inception_v3,the training argument of BN layer should always be False.But default state is Ture when keras is training.So I need to  force it to be False ,likes the following .</p>\n\n<pre><code>#Set up inference-mode base\nK.set learning phase(0)\ninputs = Input(...)\ninception_model = applications.resnet50.ResNet50(include_top=False, weights='imagenet', input_tensor=inputs)\n#Add training-mode layers\nK.set learning_phase(1)\nx = layerNp1(...)(x) \nx= layerNp2(...)(x)\n</code></pre>",
      "rawMarkdown": "Thank you very much for your reply.I know you.My focal loss is  borrowed from you.I found the  solution to the problem.Due to I froze all the layers of inception_v3,the training argument of BN layer should always be False.But default state is Ture when keras is training.So I need to  force it to be False ,likes the following .\n\n    #Set up inference-mode base\n    K.set learning phase(0)\n    inputs = Input(...)\n    inception_model = applications.resnet50.ResNet50(include_top=False, weights='imagenet', input_tensor=inputs)\n    #Add training-mode layers\n    K.set learning_phase(1)\n    x = layerNp1(...)(x) \n    x= layerNp2(...)(x)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 447586,
      "author_name": "iafoss",
      "author_url": "",
      "post_date": "12/30/2018 05:52:35",
      "content": "<p>I would think that your model overfits a lot, but accuracy 0.07 looks extremely low. Even if I replace the head of the model and train only it, the accuracy already is ~0.93. There can be several issues: some problem with data loading(1), training of entire model with replaced head with using too high learning rate and without freezing the bottom layers(2), small validation dataset and statistical noise (3). Since I do not using Keras, I could not provide more details on the above issues.</p>",
      "votes": null,
      "replies": [
        {
          "id": 447600,
          "author_name": "zjn2ai",
          "author_url": "",
          "post_date": "12/30/2018 06:39:51",
          "content": "<p>Thank you very much for your reply.I know you.My focal loss is  borrowed from you.I found the  solution to the problem.Due to I froze all the layers of inception_v3,the training argument of BN layer should always be False.But default state is Ture when keras is training.So I need to  force it to be False ,likes the following .</p>\n\n<pre><code>#Set up inference-mode base\nK.set learning phase(0)\ninputs = Input(...)\ninception_model = applications.resnet50.ResNet50(include_top=False, weights='imagenet', input_tensor=inputs)\n#Add training-mode layers\nK.set learning_phase(1)\nx = layerNp1(...)(x) \nx= layerNp2(...)(x)\n</code></pre>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "447161": "First of all, thank you for answering my questions.\nThis is my first time to use transfer learning of keras,I have a question about training and predicting.Why are the accuracy and f1 score of training and validation so different???\n\n    Epoch 4/5\n    1500/1500 [==============================] - 23s 16ms/step - loss: 1.3653 - \n    categorical_accuracy: 0.3387 - binary_accuracy: 0.0664 - f1_keras: 0.1578 - val_loss: 3.0188 - \n    val_categorical_accuracy: 0.1500 - val_binary_accuracy: 0.0601 - val_f1_keras: 0.0422\n    Epoch 5/5\n    1500/1500 [==============================] - 23s 15ms/step - loss: 1.1838 - \n    categorical_accuracy: 0.3800 - binary_accuracy: 0.0607 - f1_keras: 0.1995 - val_loss: 3.8084 - \n    val_categorical_accuracy: 0.0914 - val_binary_accuracy: 0.0455 - val_f1_keras: 0.0299\n\nEven if I use the same small training data set and validation data set, I get a similar result.\n\n    model.fit(x=X_train, y=Y_train, batch_size=32, epochs=5, validation_data=(X_train, Y_train))\n    Epoch 4/5\n    1500/1500 [==============================] - 31s 21ms/step - loss: 1.4078 - \n    categorical_accuracy: 0.3367 - binary_accuracy: 0.0622 - f1_keras: 0.1637 - val_loss: 2.8940 - \n    val_categorical_accuracy: 0.2060 - val_binary_accuracy: 0.0954 - val_f1_keras: 0.0691\n    Epoch 5/5\n    1500/1500 [==============================] - 31s 21ms/step - loss: 1.2215 - \n    categorical_accuracy: 0.3980 - binary_accuracy: 0.0629 - f1_keras: 0.2194 - val_loss: 3.5152 - \n    val_categorical_accuracy: 0.0720 - val_binary_accuracy: 0.0700 - val_f1_keras: 0.0556",
    "447586": "I would think that your model overfits a lot, but accuracy 0.07 looks extremely low. Even if I replace the head of the model and train only it, the accuracy already is ~0.93. There can be several issues: some problem with data loading(1), training of entire model with replaced head with using too high learning rate and without freezing the bottom layers(2), small validation dataset and statistical noise (3). Since I do not using Keras, I could not provide more details on the above issues.",
    "447600": "Thank you very much for your reply.I know you.My focal loss is  borrowed from you.I found the  solution to the problem.Due to I froze all the layers of inception_v3,the training argument of BN layer should always be False.But default state is Ture when keras is training.So I need to  force it to be False ,likes the following .\n\n    #Set up inference-mode base\n    K.set learning phase(0)\n    inputs = Input(...)\n    inception_model = applications.resnet50.ResNet50(include_top=False, weights='imagenet', input_tensor=inputs)\n    #Add training-mode layers\n    K.set learning_phase(1)\n    x = layerNp1(...)(x) \n    x= layerNp2(...)(x)"
  },
  "source": "meta"
}