{
  "id": 74825,
  "title": "how to add dropout in Keras for pre-trained stock resnet50",
  "url": "/competitions/human-protein-atlas-image-classification/discussion/74825",
  "author_name": "",
  "post_date": "2018-12-16T08:10:06.483901100Z",
  "votes": 2,
  "comment_count": 3,
  "views": 0,
  "content": "<p>I am encountering the over-fitting glass ceiling again and again lately, so decided to try addign dropout.\nmy code is (based on <a href=\"https://www.kaggle.com/kwentar/two-branches-xception-lb-0-3\">this</a> kernel) :</p>\n\n<pre><code>inp_mask = Input(shape=input_shape)\npretrain_model_mask = ResNet50(input_shape = (512,512,3),\n    include_top=False,\n    weights='imagenet',\n    pooling='avg')\npretrain_model_mask.name='resnet50_mask'\n\nx = pretrain_model_mask(inp_mask)\nx = Dropout(0.5)(x)\nout = Dense(n_out, activation='sigmoid')(x)\nmodel = Model(inputs=inp_mask, outputs=[out])\n</code></pre>\n\n<p>However, i get really bad results. the model f1 and loss hardly progress at all (compared to a model without the dropout line). </p>\n\n<p>Any help will be greatly appreciated!</p>\n\n<p>Just wanted to add here that this is really an amazing competition. People share generously and willingly. I learned more here that in the previous two competitions together. Really enojoying it, so thanks!</p>",
  "messages": [
    {
      "id": "439730",
      "postDate": "12/16/2018 08:10:06",
      "content": "<p>I am encountering the over-fitting glass ceiling again and again lately, so decided to try addign dropout.\nmy code is (based on <a href=\"https://www.kaggle.com/kwentar/two-branches-xception-lb-0-3\">this</a> kernel) :</p>\n\n<pre><code>inp_mask = Input(shape=input_shape)\npretrain_model_mask = ResNet50(input_shape = (512,512,3),\n    include_top=False,\n    weights='imagenet',\n    pooling='avg')\npretrain_model_mask.name='resnet50_mask'\n\nx = pretrain_model_mask(inp_mask)\nx = Dropout(0.5)(x)\nout = Dense(n_out, activation='sigmoid')(x)\nmodel = Model(inputs=inp_mask, outputs=[out])\n</code></pre>\n\n<p>However, i get really bad results. the model f1 and loss hardly progress at all (compared to a model without the dropout line). </p>\n\n<p>Any help will be greatly appreciated!</p>\n\n<p>Just wanted to add here that this is really an amazing competition. People share generously and willingly. I learned more here that in the previous two competitions together. Really enojoying it, so thanks!</p>",
      "rawMarkdown": "I am encountering the over-fitting glass ceiling again and again lately, so decided to try addign dropout.\nmy code is (based on [this][1] kernel) :\n\n    inp_mask = Input(shape=input_shape)\n    pretrain_model_mask = ResNet50(input_shape = (512,512,3),\n        include_top=False,\n        weights='imagenet',\n        pooling='avg')\n    pretrain_model_mask.name='resnet50_mask'\n    \n    x = pretrain_model_mask(inp_mask)\n    x = Dropout(0.5)(x)\n    out = Dense(n_out, activation='sigmoid')(x)\n    model = Model(inputs=inp_mask, outputs=[out])\n\nHowever, i get really bad results. the model f1 and loss hardly progress at all (compared to a model without the dropout line). \n\nAny help will be greatly appreciated!\n\nJust wanted to add here that this is really an amazing competition. People share generously and willingly. I learned more here that in the previous two competitions together. Really enojoying it, so thanks!\n\n\n  [1]: https://www.kaggle.com/kwentar/two-branches-xception-lb-0-3",
      "votes": null
    },
    {
      "id": "439746",
      "postDate": "12/16/2018 08:54:31",
      "content": "<p>I would try adding a dense layer between the pretrained model output and the dropout line, ie make the dropout operate on the added dense layer rather than directly on the pretrained model output.</p>",
      "rawMarkdown": "I would try adding a dense layer between the pretrained model output and the dropout line, ie make the dropout operate on the added dense layer rather than directly on the pretrained model output.",
      "votes": null
    },
    {
      "id": "439751",
      "postDate": "12/16/2018 09:05:09",
      "content": "<p>Oh wow, another Kiwi!\nWill try that, thank you. I saw this being used in a few models but couldn't figure out why it would matter. Will do some more research if it improves the model behavior.</p>",
      "rawMarkdown": "Oh wow, another Kiwi!\nWill try that, thank you. I saw this being used in a few models but couldn't figure out why it would matter. Will do some more research if it improves the model behavior.",
      "votes": null
    },
    {
      "id": "444064",
      "postDate": "12/23/2018 03:52:18",
      "content": "<p>x = pretrain_model(input)\n    #dropout from 0.5 to 0.25?\n    x = Dropout(0.5)(x)\n    x = Dense(1024, activation='relu')(x)\n    x = Dropout(0.5)(x)\n    output = Dense(n_out, activation='sigmoid')(x)\n    model = Model(input_tensor, output)</p>",
      "rawMarkdown": "x = pretrain_model(input)\n    #dropout from 0.5 to 0.25?\n    x = Dropout(0.5)(x)\n    x = Dense(1024, activation='relu')(x)\n    x = Dropout(0.5)(x)\n    output = Dense(n_out, activation='sigmoid')(x)\n    model = Model(input_tensor, output)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 439746,
      "author_name": "timhartill",
      "author_url": "",
      "post_date": "12/16/2018 08:54:31",
      "content": "<p>I would try adding a dense layer between the pretrained model output and the dropout line, ie make the dropout operate on the added dense layer rather than directly on the pretrained model output.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 439751,
      "author_name": "moshel",
      "author_url": "",
      "post_date": "12/16/2018 09:05:09",
      "content": "<p>Oh wow, another Kiwi!\nWill try that, thank you. I saw this being used in a few models but couldn't figure out why it would matter. Will do some more research if it improves the model behavior.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 444064,
      "author_name": "crilinux",
      "author_url": "",
      "post_date": "12/23/2018 03:52:18",
      "content": "<p>x = pretrain_model(input)\n    #dropout from 0.5 to 0.25?\n    x = Dropout(0.5)(x)\n    x = Dense(1024, activation='relu')(x)\n    x = Dropout(0.5)(x)\n    output = Dense(n_out, activation='sigmoid')(x)\n    model = Model(input_tensor, output)</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "439730": "I am encountering the over-fitting glass ceiling again and again lately, so decided to try addign dropout.\nmy code is (based on [this][1] kernel) :\n\n    inp_mask = Input(shape=input_shape)\n    pretrain_model_mask = ResNet50(input_shape = (512,512,3),\n        include_top=False,\n        weights='imagenet',\n        pooling='avg')\n    pretrain_model_mask.name='resnet50_mask'\n    \n    x = pretrain_model_mask(inp_mask)\n    x = Dropout(0.5)(x)\n    out = Dense(n_out, activation='sigmoid')(x)\n    model = Model(inputs=inp_mask, outputs=[out])\n\nHowever, i get really bad results. the model f1 and loss hardly progress at all (compared to a model without the dropout line). \n\nAny help will be greatly appreciated!\n\nJust wanted to add here that this is really an amazing competition. People share generously and willingly. I learned more here that in the previous two competitions together. Really enojoying it, so thanks!\n\n\n  [1]: https://www.kaggle.com/kwentar/two-branches-xception-lb-0-3",
    "439746": "I would try adding a dense layer between the pretrained model output and the dropout line, ie make the dropout operate on the added dense layer rather than directly on the pretrained model output.",
    "439751": "Oh wow, another Kiwi!\nWill try that, thank you. I saw this being used in a few models but couldn't figure out why it would matter. Will do some more research if it improves the model behavior.",
    "444064": "x = pretrain_model(input)\n    #dropout from 0.5 to 0.25?\n    x = Dropout(0.5)(x)\n    x = Dense(1024, activation='relu')(x)\n    x = Dropout(0.5)(x)\n    output = Dense(n_out, activation='sigmoid')(x)\n    model = Model(input_tensor, output)"
  },
  "source": "meta"
}