{
  "id": 43991,
  "title": "On updating regularizers of a pre-trained model",
  "url": "/competitions/cdiscount-image-classification-challenge/discussion/43991",
  "author_name": "tuna@boston",
  "post_date": "2017-11-22T03:21:00.583000",
  "votes": 0,
  "comment_count": 0,
  "views": 0,
  "content": "<p>I'm having an issue with updating regularizers for a model in Keras and hope I can get some help. </p>\n\n<p>When I looked at a layer in the Xception model, e.g. \"block8_sepconv1\", I could see that both the 'pointwise_regularizer' and 'depthwise_regularizer' are None. Although the Xception paper (<a href=\"https://arxiv.org/abs/1610.02357\">https://arxiv.org/abs/1610.02357</a>) mentioned that weight decay L2 regularization was used (Sec. 4.3, page 4), my understanding was that this information was not saved simply because it is considered \"hyper-parameters\" (please correct me if I'm wrong), so I tried to add my own regularizers to a \"layer\" with the following code:</p>\n\n<pre><code>cf = layer.get_config()\ncf['depthwise_regularizer'] = reg\ncf['pointwise_regularizer'] = reg\nlayer = keras.layers.SeparableConv2D.from_config(cf)\n</code></pre>\n\n<p>With these put in, things seemed to work and I no longer saw big variances during training. However the problem is that I couldn't load these model again for further training: \"load_model\" seemed to run forever without any error messages.</p>\n\n<p>When I examined the saved model file (with HDF5 viewer), the layer \"block8_sepconv_1\" seemed to be \"corrupted\" after I updated the regularizers---I no longer see its properties and the \"weight_names\" becomes null.</p>\n\n<p>Does anyone run into the same issue before and know how to fix it?</p>\n\n<p>Thank you!</p>",
  "messages": [
    {
      "id": 246974,
      "postDate": "2017-11-22T03:21:00.583Z",
      "content": "<p>I'm having an issue with updating regularizers for a model in Keras and hope I can get some help. </p>\n\n<p>When I looked at a layer in the Xception model, e.g. \"block8_sepconv1\", I could see that both the 'pointwise_regularizer' and 'depthwise_regularizer' are None. Although the Xception paper (<a href=\"https://arxiv.org/abs/1610.02357\">https://arxiv.org/abs/1610.02357</a>) mentioned that weight decay L2 regularization was used (Sec. 4.3, page 4), my understanding was that this information was not saved simply because it is considered \"hyper-parameters\" (please correct me if I'm wrong), so I tried to add my own regularizers to a \"layer\" with the following code:</p>\n\n<pre><code>cf = layer.get_config()\ncf['depthwise_regularizer'] = reg\ncf['pointwise_regularizer'] = reg\nlayer = keras.layers.SeparableConv2D.from_config(cf)\n</code></pre>\n\n<p>With these put in, things seemed to work and I no longer saw big variances during training. However the problem is that I couldn't load these model again for further training: \"load_model\" seemed to run forever without any error messages.</p>\n\n<p>When I examined the saved model file (with HDF5 viewer), the layer \"block8_sepconv_1\" seemed to be \"corrupted\" after I updated the regularizers---I no longer see its properties and the \"weight_names\" becomes null.</p>\n\n<p>Does anyone run into the same issue before and know how to fix it?</p>\n\n<p>Thank you!</p>",
      "rawMarkdown": "I'm having an issue with updating regularizers for a model in Keras and hope I can get some help. \n\nWhen I looked at a layer in the Xception model, e.g. \"block8_sepconv1\", I could see that both the 'pointwise_regularizer' and 'depthwise_regularizer' are None. Although the Xception paper (https://arxiv.org/abs/1610.02357) mentioned that weight decay L2 regularization was used (Sec. 4.3, page 4), my understanding was that this information was not saved simply because it is considered \"hyper-parameters\" (please correct me if I'm wrong), so I tried to add my own regularizers to a \"layer\" with the following code:\n\n    cf = layer.get_config()\n    cf['depthwise_regularizer'] = reg\n    cf['pointwise_regularizer'] = reg\n    layer = keras.layers.SeparableConv2D.from_config(cf)\n\nWith these put in, things seemed to work and I no longer saw big variances during training. However the problem is that I couldn't load these model again for further training: \"load_model\" seemed to run forever without any error messages.\n\nWhen I examined the saved model file (with HDF5 viewer), the layer \"block8_sepconv_1\" seemed to be \"corrupted\" after I updated the regularizers---I no longer see its properties and the \"weight_names\" becomes null.\n\nDoes anyone run into the same issue before and know how to fix it?\n\nThank you!\n\n\n\n"
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "246974": "I'm having an issue with updating regularizers for a model in Keras and hope I can get some help. \n\nWhen I looked at a layer in the Xception model, e.g. \"block8_sepconv1\", I could see that both the 'pointwise_regularizer' and 'depthwise_regularizer' are None. Although the Xception paper (https://arxiv.org/abs/1610.02357) mentioned that weight decay L2 regularization was used (Sec. 4.3, page 4), my understanding was that this information was not saved simply because it is considered \"hyper-parameters\" (please correct me if I'm wrong), so I tried to add my own regularizers to a \"layer\" with the following code:\n\n    cf = layer.get_config()\n    cf['depthwise_regularizer'] = reg\n    cf['pointwise_regularizer'] = reg\n    layer = keras.layers.SeparableConv2D.from_config(cf)\n\nWith these put in, things seemed to work and I no longer saw big variances during training. However the problem is that I couldn't load these model again for further training: \"load_model\" seemed to run forever without any error messages.\n\nWhen I examined the saved model file (with HDF5 viewer), the layer \"block8_sepconv_1\" seemed to be \"corrupted\" after I updated the regularizers---I no longer see its properties and the \"weight_names\" becomes null.\n\nDoes anyone run into the same issue before and know how to fix it?\n\nThank you!\n\n\n\n"
  }
}