{
  "id": 89055,
  "title": "(Renewed) Getting CNN trained well",
  "url": "/competitions/freesound-audio-tagging-2019/discussion/89055",
  "author_name": "",
  "post_date": "2019-04-11T12:17:18.725353600Z",
  "votes": 7,
  "comment_count": 1,
  "views": 0,
  "content": "<p><a href=\"http://cs231n.github.io/understanding-cnn/\">CS231n Convolutional Neural Networks for Visual Recognition</a> is always great text to read again and again occasionally, and it reminded me of following notes.</p>\n\n<p><em>\"The weights are useful to visualize because well-trained networks usually display nice and smooth filters without any noisy patterns. Noisy patterns can be an indicator of a network that hasn’t been trained for long enough, or possibly a very low regularization strength that may have led to overfitting.\"</em></p>\n\n<p>The other day I have posted a kernel below to visualize model with CAM, now I updated it to show the first layer also.</p>\n\n<p><a href=\"https://www.kaggle.com/daisukelab/verifying-cnn-models-with-cam-and-etc-fast-ai?scriptVersionId=12789564\">https://www.kaggle.com/daisukelab/verifying-cnn-models-with-cam-and-etc-fast-ai?scriptVersionId=12789564</a></p>\n\n<h2>Training noisy set example</h2>\n\n<p>Here's example of training a CNN model from scratch on noisy set, and how filters grow. Training was held in local server to avoid kernel time limit, it looks like short version of training on ImageNet samples.</p>\n\n<p>I tried this trained model to apply transfer training on curated set, and it shows <strike>very</strike> good performance (not submitted yet).</p>\n\n<p>Regarding training only on curated set from scratch, filters haven't grow well with curated set as I have confirmed at the end part of following kernel. <em>This might imply how to work with training sets for further higher performance.</em></p>\n\n<p><a href=\"https://www.kaggle.com/daisukelab/cnn-2d-basic-2-preprocessed-dataset-noisy\">https://www.kaggle.com/daisukelab/cnn-2d-basic-2-preprocessed-dataset-noisy</a></p>\n\n<h3>Initial filters</h3>\n\n<p><img src=\"https://github.com/daisukelab/dl-image/raw/master/imagenet/images/filter_0.png\" alt=\"init\"></p>\n\n<h3>15 epochs</h3>\n\n<p><img src=\"https://github.com/daisukelab/dl-image/raw/master/imagenet/images/filter_15.png\" alt=\"15\"></p>\n\n<h3>115 epochs</h3>\n\n<p>Filters have almost grown up.</p>\n\n<p><img src=\"https://github.com/daisukelab/dl-image/raw/master/imagenet/images/filter_115.png\" alt=\"115\"></p>\n\n<h3>215 epochs</h3>\n\n<p><img src=\"https://github.com/daisukelab/dl-image/raw/master/imagenet/images/filter_215.png\" alt=\"215\"></p>\n\n<h3>Over 1000 epochs</h3>\n\n<p><img src=\"https://github.com/daisukelab/dl-image/raw/master/imagenet/images/filter_1000+.png\" alt=\"1000\"></p>",
  "messages": [
    {
      "id": "514041",
      "postDate": "04/11/2019 12:17:18",
      "content": "<p><a href=\"http://cs231n.github.io/understanding-cnn/\">CS231n Convolutional Neural Networks for Visual Recognition</a> is always great text to read again and again occasionally, and it reminded me of following notes.</p>\n\n<p><em>\"The weights are useful to visualize because well-trained networks usually display nice and smooth filters without any noisy patterns. Noisy patterns can be an indicator of a network that hasn’t been trained for long enough, or possibly a very low regularization strength that may have led to overfitting.\"</em></p>\n\n<p>The other day I have posted a kernel below to visualize model with CAM, now I updated it to show the first layer also.</p>\n\n<p><a href=\"https://www.kaggle.com/daisukelab/verifying-cnn-models-with-cam-and-etc-fast-ai?scriptVersionId=12789564\">https://www.kaggle.com/daisukelab/verifying-cnn-models-with-cam-and-etc-fast-ai?scriptVersionId=12789564</a></p>\n\n<h2>Training noisy set example</h2>\n\n<p>Here's example of training a CNN model from scratch on noisy set, and how filters grow. Training was held in local server to avoid kernel time limit, it looks like short version of training on ImageNet samples.</p>\n\n<p>I tried this trained model to apply transfer training on curated set, and it shows <strike>very</strike> good performance (not submitted yet).</p>\n\n<p>Regarding training only on curated set from scratch, filters haven't grow well with curated set as I have confirmed at the end part of following kernel. <em>This might imply how to work with training sets for further higher performance.</em></p>\n\n<p><a href=\"https://www.kaggle.com/daisukelab/cnn-2d-basic-2-preprocessed-dataset-noisy\">https://www.kaggle.com/daisukelab/cnn-2d-basic-2-preprocessed-dataset-noisy</a></p>\n\n<h3>Initial filters</h3>\n\n<p><img src=\"https://github.com/daisukelab/dl-image/raw/master/imagenet/images/filter_0.png\" alt=\"init\"></p>\n\n<h3>15 epochs</h3>\n\n<p><img src=\"https://github.com/daisukelab/dl-image/raw/master/imagenet/images/filter_15.png\" alt=\"15\"></p>\n\n<h3>115 epochs</h3>\n\n<p>Filters have almost grown up.</p>\n\n<p><img src=\"https://github.com/daisukelab/dl-image/raw/master/imagenet/images/filter_115.png\" alt=\"115\"></p>\n\n<h3>215 epochs</h3>\n\n<p><img src=\"https://github.com/daisukelab/dl-image/raw/master/imagenet/images/filter_215.png\" alt=\"215\"></p>\n\n<h3>Over 1000 epochs</h3>\n\n<p><img src=\"https://github.com/daisukelab/dl-image/raw/master/imagenet/images/filter_1000+.png\" alt=\"1000\"></p>",
      "rawMarkdown": "[CS231n Convolutional Neural Networks for Visual Recognition](http://cs231n.github.io/understanding-cnn/) is always great text to read again and again occasionally, and it reminded me of following notes.\n\n_\"The weights are useful to visualize because well-trained networks usually display nice and smooth filters without any noisy patterns. Noisy patterns can be an indicator of a network that hasn’t been trained for long enough, or possibly a very low regularization strength that may have led to overfitting.\"_\n\nThe other day I have posted a kernel below to visualize model with CAM, now I updated it to show the first layer also.\n\nhttps://www.kaggle.com/daisukelab/verifying-cnn-models-with-cam-and-etc-fast-ai?scriptVersionId=12789564\n\n## Training noisy set example\n\nHere's example of training a CNN model from scratch on noisy set, and how filters grow. Training was held in local server to avoid kernel time limit, it looks like short version of training on ImageNet samples.\n\nI tried this trained model to apply transfer training on curated set, and it shows <strike>very</strike> good performance (not submitted yet).\n\nRegarding training only on curated set from scratch, filters haven't grow well with curated set as I have confirmed at the end part of following kernel. _This might imply how to work with training sets for further higher performance._\n\nhttps://www.kaggle.com/daisukelab/cnn-2d-basic-2-preprocessed-dataset-noisy\n\n### Initial filters\n\n![init](https://github.com/daisukelab/dl-image/raw/master/imagenet/images/filter_0.png)\n\n### 15 epochs\n\n![15](https://github.com/daisukelab/dl-image/raw/master/imagenet/images/filter_15.png)\n\n### 115 epochs\n\nFilters have almost grown up.\n\n![115](https://github.com/daisukelab/dl-image/raw/master/imagenet/images/filter_115.png)\n\n### 215 epochs\n\n![215](https://github.com/daisukelab/dl-image/raw/master/imagenet/images/filter_215.png)\n\n### Over 1000 epochs\n\n![1000](https://github.com/daisukelab/dl-image/raw/master/imagenet/images/filter_1000+.png)",
      "votes": null
    },
    {
      "id": "515314",
      "postDate": "04/12/2019 12:25:29",
      "content": "<p>Failure example.</p>\n\n<p>I tried to train CNN with unsupervised training fashion, and got this first filters at the first layer.\n<img src=\"https://github.com/daisukelab/dl-image/raw/master/imagenet/images/failed_filters.png\" alt=\"failed filters\"></p>\n\n<p>Then trained with this pretrained model have failed with f_score less than 0.40. Looks smooth but many filters are empty...</p>",
      "rawMarkdown": "Failure example.\n\nI tried to train CNN with unsupervised training fashion, and got this first filters at the first layer.\n![failed filters](https://github.com/daisukelab/dl-image/raw/master/imagenet/images/failed_filters.png)\n\nThen trained with this pretrained model have failed with f_score less than 0.40. Looks smooth but many filters are empty...",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 515314,
      "author_name": "daisukelab",
      "author_url": "",
      "post_date": "04/12/2019 12:25:29",
      "content": "<p>Failure example.</p>\n\n<p>I tried to train CNN with unsupervised training fashion, and got this first filters at the first layer.\n<img src=\"https://github.com/daisukelab/dl-image/raw/master/imagenet/images/failed_filters.png\" alt=\"failed filters\"></p>\n\n<p>Then trained with this pretrained model have failed with f_score less than 0.40. Looks smooth but many filters are empty...</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "514041": "[CS231n Convolutional Neural Networks for Visual Recognition](http://cs231n.github.io/understanding-cnn/) is always great text to read again and again occasionally, and it reminded me of following notes.\n\n_\"The weights are useful to visualize because well-trained networks usually display nice and smooth filters without any noisy patterns. Noisy patterns can be an indicator of a network that hasn’t been trained for long enough, or possibly a very low regularization strength that may have led to overfitting.\"_\n\nThe other day I have posted a kernel below to visualize model with CAM, now I updated it to show the first layer also.\n\nhttps://www.kaggle.com/daisukelab/verifying-cnn-models-with-cam-and-etc-fast-ai?scriptVersionId=12789564\n\n## Training noisy set example\n\nHere's example of training a CNN model from scratch on noisy set, and how filters grow. Training was held in local server to avoid kernel time limit, it looks like short version of training on ImageNet samples.\n\nI tried this trained model to apply transfer training on curated set, and it shows <strike>very</strike> good performance (not submitted yet).\n\nRegarding training only on curated set from scratch, filters haven't grow well with curated set as I have confirmed at the end part of following kernel. _This might imply how to work with training sets for further higher performance._\n\nhttps://www.kaggle.com/daisukelab/cnn-2d-basic-2-preprocessed-dataset-noisy\n\n### Initial filters\n\n![init](https://github.com/daisukelab/dl-image/raw/master/imagenet/images/filter_0.png)\n\n### 15 epochs\n\n![15](https://github.com/daisukelab/dl-image/raw/master/imagenet/images/filter_15.png)\n\n### 115 epochs\n\nFilters have almost grown up.\n\n![115](https://github.com/daisukelab/dl-image/raw/master/imagenet/images/filter_115.png)\n\n### 215 epochs\n\n![215](https://github.com/daisukelab/dl-image/raw/master/imagenet/images/filter_215.png)\n\n### Over 1000 epochs\n\n![1000](https://github.com/daisukelab/dl-image/raw/master/imagenet/images/filter_1000+.png)",
    "515314": "Failure example.\n\nI tried to train CNN with unsupervised training fashion, and got this first filters at the first layer.\n![failed filters](https://github.com/daisukelab/dl-image/raw/master/imagenet/images/failed_filters.png)\n\nThen trained with this pretrained model have failed with f_score less than 0.40. Looks smooth but many filters are empty..."
  },
  "source": "meta"
}