{
  "id": 74803,
  "title": "Is there a better way to merge to models in Keras",
  "url": "/competitions/quora-insincere-questions-classification/discussion/74803",
  "author_name": "",
  "post_date": "2018-12-16T00:25:14.238795700Z",
  "votes": 1,
  "comment_count": 4,
  "views": 0,
  "content": "<p>I have two models, individually they have validation accuracy of ~ 0.95. But when I concatenate them using functional api, the accuracy drops drastically to 0.06. </p>\n\n<pre><code>model1 = Sequential()\nmodel1.add(Bidirectional(CuDNNGRU(64, return_sequences=True),\n                        input_shape=(30, 300)))\nmodel1.add(Bidirectional(CuDNNGRU(64, return_sequences=True)))\nmodel1.add(GlobalMaxPool1D())\nmodel1.add(Dense(16, activation=\"relu\"))\nmodel1.add(Dropout(0.1))\nmodel1.add(Dense(8, activation=\"relu\"))\nmodel1.add(Dense(1, activation=\"softmax\"))\n\nmodel2 = Sequential()\nmodel2.add(Bidirectional(CuDNNGRU(64, return_sequences=True),\n                        input_shape=(30, 300)))\nmodel2.add(Bidirectional(CuDNNGRU(64, return_sequences=True)))\nmodel2.add(GlobalMaxPool1D())\nmodel2.add(Dense(16, activation=\"relu\"))\nmodel2.add(Dropout(0.1))\nmodel2.add(Dense(8, activation=\"relu\"))\nmodel2.add(Dense(1, activation=\"softmax\"))\n\n\nmodeltmp = concatenate([model1.output, model2.output], axis=-1)\nmodeltmp = Dense(1, activation='softmax')(modeltmp) \nmodel = Model(inputs=[model1.input, model2.input], outputs=modeltmp)\n</code></pre>\n\n<p><img src=\"https://i.imgur.com/G5MnKVW.png\" alt=\"enter image description here\"></p>\n\n<p>I want to understand what could possibly be going wrong with my model. Is there a better way of merging?</p>",
  "messages": [
    {
      "id": "439621",
      "postDate": "12/16/2018 00:25:14",
      "content": "<p>I have two models, individually they have validation accuracy of ~ 0.95. But when I concatenate them using functional api, the accuracy drops drastically to 0.06. </p>\n\n<pre><code>model1 = Sequential()\nmodel1.add(Bidirectional(CuDNNGRU(64, return_sequences=True),\n                        input_shape=(30, 300)))\nmodel1.add(Bidirectional(CuDNNGRU(64, return_sequences=True)))\nmodel1.add(GlobalMaxPool1D())\nmodel1.add(Dense(16, activation=\"relu\"))\nmodel1.add(Dropout(0.1))\nmodel1.add(Dense(8, activation=\"relu\"))\nmodel1.add(Dense(1, activation=\"softmax\"))\n\nmodel2 = Sequential()\nmodel2.add(Bidirectional(CuDNNGRU(64, return_sequences=True),\n                        input_shape=(30, 300)))\nmodel2.add(Bidirectional(CuDNNGRU(64, return_sequences=True)))\nmodel2.add(GlobalMaxPool1D())\nmodel2.add(Dense(16, activation=\"relu\"))\nmodel2.add(Dropout(0.1))\nmodel2.add(Dense(8, activation=\"relu\"))\nmodel2.add(Dense(1, activation=\"softmax\"))\n\n\nmodeltmp = concatenate([model1.output, model2.output], axis=-1)\nmodeltmp = Dense(1, activation='softmax')(modeltmp) \nmodel = Model(inputs=[model1.input, model2.input], outputs=modeltmp)\n</code></pre>\n\n<p><img src=\"https://i.imgur.com/G5MnKVW.png\" alt=\"enter image description here\"></p>\n\n<p>I want to understand what could possibly be going wrong with my model. Is there a better way of merging?</p>",
      "rawMarkdown": "I have two models, individually they have validation accuracy of ~ 0.95. But when I concatenate them using functional api, the accuracy drops drastically to 0.06. \n\n    model1 = Sequential()\n    model1.add(Bidirectional(CuDNNGRU(64, return_sequences=True),\n                            input_shape=(30, 300)))\n    model1.add(Bidirectional(CuDNNGRU(64, return_sequences=True)))\n    model1.add(GlobalMaxPool1D())\n    model1.add(Dense(16, activation=\"relu\"))\n    model1.add(Dropout(0.1))\n    model1.add(Dense(8, activation=\"relu\"))\n    model1.add(Dense(1, activation=\"softmax\"))\n    \n    model2 = Sequential()\n    model2.add(Bidirectional(CuDNNGRU(64, return_sequences=True),\n                            input_shape=(30, 300)))\n    model2.add(Bidirectional(CuDNNGRU(64, return_sequences=True)))\n    model2.add(GlobalMaxPool1D())\n    model2.add(Dense(16, activation=\"relu\"))\n    model2.add(Dropout(0.1))\n    model2.add(Dense(8, activation=\"relu\"))\n    model2.add(Dense(1, activation=\"softmax\"))\n\n    \n    modeltmp = concatenate([model1.output, model2.output], axis=-1)\n    modeltmp = Dense(1, activation='softmax')(modeltmp) \n    model = Model(inputs=[model1.input, model2.input], outputs=modeltmp)\n\n\n![enter image description here](https://i.imgur.com/G5MnKVW.png)\n\n\n\nI want to understand what could possibly be going wrong with my model. Is there a better way of merging?",
      "votes": null
    },
    {
      "id": "439631",
      "postDate": "12/16/2018 00:50:08",
      "content": "<p>Change activation=\"softmax\" to activation=\"sigmoid\" to get an predicted output value in the range (0,1).</p>",
      "rawMarkdown": "Change activation=\"softmax\" to activation=\"sigmoid\" to get an predicted output value in the range (0,1).",
      "votes": null
    },
    {
      "id": "439634",
      "postDate": "12/16/2018 01:00:03",
      "content": "<p>I tried that as well, did not work. </p>",
      "rawMarkdown": "I tried that as well, did not work.",
      "votes": null
    },
    {
      "id": "439647",
      "postDate": "12/16/2018 01:59:46",
      "content": "<p>Thanks, I got it finally working.</p>\n\n<p>model1 = Sequential()\nmodel1.add(Bidirectional(CuDNNGRU(64, return_sequences=True),\n                        input_shape=(30, 300)))\nmodel1.add(Bidirectional(CuDNNGRU(64, return_sequences=True)))\nmodel1.add(GlobalMaxPool1D())\nmodel1.add(Dense(16, activation=\"relu\"))\nmodel1.add(Dropout(0.1))\nmodel1.add(Dense(8, activation=\"relu\"))\nmodel1.add(Dense(1, activation=\"sigmoid\"))</p>\n\n<p>model2 = Sequential()\nmodel2.add(Bidirectional(CuDNNGRU(64, return_sequences=True),\n                        input_shape=(30, 300)))\nmodel2.add(Bidirectional(CuDNNGRU(64, return_sequences=True)))\nmodel2.add(GlobalMaxPool1D())\nmodel2.add(Dense(16, activation=\"relu\"))\nmodel2.add(Dropout(0.1))\nmodel2.add(Dense(8, activation=\"relu\"))\nmodel2.add(Dense(1, activation=\"sigmoid\"))</p>\n\n<p>modeltmp = concatenate([model1.output, model2.output], axis=-1)\nmodeltmp = Dense(1, activation='sigmoid')(modeltmp) \nmodel = Model(inputs=[model1.input, model2.input], outputs=modeltmp)</p>",
      "rawMarkdown": "Thanks, I got it finally working.\n\nmodel1 = Sequential()\nmodel1.add(Bidirectional(CuDNNGRU(64, return_sequences=True),\n                        input_shape=(30, 300)))\nmodel1.add(Bidirectional(CuDNNGRU(64, return_sequences=True)))\nmodel1.add(GlobalMaxPool1D())\nmodel1.add(Dense(16, activation=\"relu\"))\nmodel1.add(Dropout(0.1))\nmodel1.add(Dense(8, activation=\"relu\"))\nmodel1.add(Dense(1, activation=\"sigmoid\"))\n\nmodel2 = Sequential()\nmodel2.add(Bidirectional(CuDNNGRU(64, return_sequences=True),\n                        input_shape=(30, 300)))\nmodel2.add(Bidirectional(CuDNNGRU(64, return_sequences=True)))\nmodel2.add(GlobalMaxPool1D())\nmodel2.add(Dense(16, activation=\"relu\"))\nmodel2.add(Dropout(0.1))\nmodel2.add(Dense(8, activation=\"relu\"))\nmodel2.add(Dense(1, activation=\"sigmoid\"))\n\nmodeltmp = concatenate([model1.output, model2.output], axis=-1)\nmodeltmp = Dense(1, activation='sigmoid')(modeltmp) \nmodel = Model(inputs=[model1.input, model2.input], outputs=modeltmp)",
      "votes": null
    },
    {
      "id": "570219",
      "postDate": "07/08/2019 02:39:28",
      "content": "<p>Why should we  get an predicted output value in the range (0,1) rather than direct 0/1 ? Thx</p>",
      "rawMarkdown": "Why should we  get an predicted output value in the range (0,1) rather than direct 0/1 ? Thx",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 439631,
      "author_name": "maxjeblick",
      "author_url": "",
      "post_date": "12/16/2018 00:50:08",
      "content": "<p>Change activation=\"softmax\" to activation=\"sigmoid\" to get an predicted output value in the range (0,1).</p>",
      "votes": null,
      "replies": [
        {
          "id": 439634,
          "author_name": "puneetsl",
          "author_url": "",
          "post_date": "12/16/2018 01:00:03",
          "content": "<p>I tried that as well, did not work. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 439647,
          "author_name": "puneetsl",
          "author_url": "",
          "post_date": "12/16/2018 01:59:46",
          "content": "<p>Thanks, I got it finally working.</p>\n\n<p>model1 = Sequential()\nmodel1.add(Bidirectional(CuDNNGRU(64, return_sequences=True),\n                        input_shape=(30, 300)))\nmodel1.add(Bidirectional(CuDNNGRU(64, return_sequences=True)))\nmodel1.add(GlobalMaxPool1D())\nmodel1.add(Dense(16, activation=\"relu\"))\nmodel1.add(Dropout(0.1))\nmodel1.add(Dense(8, activation=\"relu\"))\nmodel1.add(Dense(1, activation=\"sigmoid\"))</p>\n\n<p>model2 = Sequential()\nmodel2.add(Bidirectional(CuDNNGRU(64, return_sequences=True),\n                        input_shape=(30, 300)))\nmodel2.add(Bidirectional(CuDNNGRU(64, return_sequences=True)))\nmodel2.add(GlobalMaxPool1D())\nmodel2.add(Dense(16, activation=\"relu\"))\nmodel2.add(Dropout(0.1))\nmodel2.add(Dense(8, activation=\"relu\"))\nmodel2.add(Dense(1, activation=\"sigmoid\"))</p>\n\n<p>modeltmp = concatenate([model1.output, model2.output], axis=-1)\nmodeltmp = Dense(1, activation='sigmoid')(modeltmp) \nmodel = Model(inputs=[model1.input, model2.input], outputs=modeltmp)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 570219,
          "author_name": "zhangyu1223",
          "author_url": "",
          "post_date": "07/08/2019 02:39:28",
          "content": "<p>Why should we  get an predicted output value in the range (0,1) rather than direct 0/1 ? Thx</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "439621": "I have two models, individually they have validation accuracy of ~ 0.95. But when I concatenate them using functional api, the accuracy drops drastically to 0.06. \n\n    model1 = Sequential()\n    model1.add(Bidirectional(CuDNNGRU(64, return_sequences=True),\n                            input_shape=(30, 300)))\n    model1.add(Bidirectional(CuDNNGRU(64, return_sequences=True)))\n    model1.add(GlobalMaxPool1D())\n    model1.add(Dense(16, activation=\"relu\"))\n    model1.add(Dropout(0.1))\n    model1.add(Dense(8, activation=\"relu\"))\n    model1.add(Dense(1, activation=\"softmax\"))\n    \n    model2 = Sequential()\n    model2.add(Bidirectional(CuDNNGRU(64, return_sequences=True),\n                            input_shape=(30, 300)))\n    model2.add(Bidirectional(CuDNNGRU(64, return_sequences=True)))\n    model2.add(GlobalMaxPool1D())\n    model2.add(Dense(16, activation=\"relu\"))\n    model2.add(Dropout(0.1))\n    model2.add(Dense(8, activation=\"relu\"))\n    model2.add(Dense(1, activation=\"softmax\"))\n\n    \n    modeltmp = concatenate([model1.output, model2.output], axis=-1)\n    modeltmp = Dense(1, activation='softmax')(modeltmp) \n    model = Model(inputs=[model1.input, model2.input], outputs=modeltmp)\n\n\n![enter image description here](https://i.imgur.com/G5MnKVW.png)\n\n\n\nI want to understand what could possibly be going wrong with my model. Is there a better way of merging?",
    "439631": "Change activation=\"softmax\" to activation=\"sigmoid\" to get an predicted output value in the range (0,1).",
    "439634": "I tried that as well, did not work.",
    "439647": "Thanks, I got it finally working.\n\nmodel1 = Sequential()\nmodel1.add(Bidirectional(CuDNNGRU(64, return_sequences=True),\n                        input_shape=(30, 300)))\nmodel1.add(Bidirectional(CuDNNGRU(64, return_sequences=True)))\nmodel1.add(GlobalMaxPool1D())\nmodel1.add(Dense(16, activation=\"relu\"))\nmodel1.add(Dropout(0.1))\nmodel1.add(Dense(8, activation=\"relu\"))\nmodel1.add(Dense(1, activation=\"sigmoid\"))\n\nmodel2 = Sequential()\nmodel2.add(Bidirectional(CuDNNGRU(64, return_sequences=True),\n                        input_shape=(30, 300)))\nmodel2.add(Bidirectional(CuDNNGRU(64, return_sequences=True)))\nmodel2.add(GlobalMaxPool1D())\nmodel2.add(Dense(16, activation=\"relu\"))\nmodel2.add(Dropout(0.1))\nmodel2.add(Dense(8, activation=\"relu\"))\nmodel2.add(Dense(1, activation=\"sigmoid\"))\n\nmodeltmp = concatenate([model1.output, model2.output], axis=-1)\nmodeltmp = Dense(1, activation='sigmoid')(modeltmp) \nmodel = Model(inputs=[model1.input, model2.input], outputs=modeltmp)",
    "570219": "Why should we  get an predicted output value in the range (0,1) rather than direct 0/1 ? Thx"
  },
  "source": "meta"
}