{
  "id": 13109,
  "title": "Weird CNN behavior?",
  "url": "/competitions/diabetic-retinopathy-detection/discussion/13109",
  "author_name": "",
  "post_date": "2015-03-28T14:16:45.367Z",
  "votes": null,
  "comment_count": 9,
  "views": 3185,
  "content": "<p>I've attached a picture of my train vs validation loss for a CNN. Basically the train and validation loss stay pretty much the same for 40 epochs or so and then validation loss starts increasing and training loss sharply declines.</p>\n<p>Does anyone have any ideas why this would happen? I figure that I might be loading the images incorrectly or could it be a parameter thing?&nbsp;</p>",
  "messages": [
    {
      "id": "68694",
      "postDate": "03/28/2015 14:16:45",
      "content": "<p>I've attached a picture of my train vs validation loss for a CNN. Basically the train and validation loss stay pretty much the same for 40 epochs or so and then validation loss starts increasing and training loss sharply declines.</p>\n<p>Does anyone have any ideas why this would happen? I figure that I might be loading the images incorrectly or could it be a parameter thing?&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "68716",
      "postDate": "03/28/2015 16:54:08",
      "content": "<p>overfit</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "68723",
      "postDate": "03/28/2015 17:16:51",
      "content": "<p>I understand that it's overfitting, but it seems pretty extreme. I guess that's the challenge with this dataset.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "68855",
      "postDate": "03/29/2015 16:29:16",
      "content": "<p>Not a lot of info to go on from just that image, but my initial naive guess would be a &quot;complete overfit&quot;. Basically when your network stops trying to understand the data at all, and just starts memorizing the correct answers. Are you using dropout, data augmentation or other regularization techniques?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "68873",
      "postDate": "03/29/2015 19:36:42",
      "content": "<p>Thank you for the feedback. I've been trying different levels of dropout and I am randomly flipping images during training. I haven't been able to make much improvement over that initial graph.&nbsp;</p>\n<p>Could it be that I'm not loading the images correctly, would that cause those kinds of problems?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "68891",
      "postDate": "03/29/2015 20:55:40",
      "content": "<p>It's possible. That would make your network train on wrong / corrupted data, which could explain why it's overfitting so aggressively.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "68910",
      "postDate": "03/29/2015 22:05:56",
      "content": "<p>At least you're getting far enough to overfit; I'm struggling to reduce the training error, let alone the validation!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "68926",
      "postDate": "03/29/2015 23:39:10",
      "content": "<p>I'm using nolearn/lasagne and I'm loading the images with skimage and putting them into a numpy array. So I'm looping through the filenames and putting each image as it's own row like this:</p>\n<p><code> image = imread(os.path.join(path, n))<br> <br> train_image[i, 0:num_features] = np.reshape(image, (1, num_features))</code></p>\n<p>Where num_features would be the 3 * height * width, and then when I load that array for the cnn I do</p>\n<p><code>X_train = X_train.reshape(-1, 3, PIXELS, PIXELS)</code></p>\n<p>Does that seem right?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "68955",
      "postDate": "03/30/2015 01:20:01",
      "content": "<p>I think&nbsp;imread load images in a &nbsp;(PIXELS, PIXELS, 3) matrix.</p>\n<p>May be you have to do&nbsp;&nbsp;X_train = X_train.swapaxes(1,3) to make it (3, PIXELS, PIXELS)&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "68974",
      "postDate": "03/30/2015 04:02:06",
      "content": "<p>I thought that I would have to do the swapaxes thing as well. But I made this script and if I run it on two images it seems like everything should work out the way that I have it with skimage. But something strange is happening when I use PIL.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 68716,
      "author_name": "numb3rs",
      "author_url": "",
      "post_date": "03/28/2015 16:54:08",
      "content": "<p>overfit</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 68723,
      "author_name": "florianm",
      "author_url": "",
      "post_date": "03/28/2015 17:16:51",
      "content": "<p>I understand that it's overfitting, but it seems pretty extreme. I guess that's the challenge with this dataset.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 68855,
      "author_name": "nicolaykorslund",
      "author_url": "",
      "post_date": "03/29/2015 16:29:16",
      "content": "<p>Not a lot of info to go on from just that image, but my initial naive guess would be a &quot;complete overfit&quot;. Basically when your network stops trying to understand the data at all, and just starts memorizing the correct answers. Are you using dropout, data augmentation or other regularization techniques?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 68873,
      "author_name": "florianm",
      "author_url": "",
      "post_date": "03/29/2015 19:36:42",
      "content": "<p>Thank you for the feedback. I've been trying different levels of dropout and I am randomly flipping images during training. I haven't been able to make much improvement over that initial graph.&nbsp;</p>\n<p>Could it be that I'm not loading the images correctly, would that cause those kinds of problems?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 68891,
      "author_name": "nicolaykorslund",
      "author_url": "",
      "post_date": "03/29/2015 20:55:40",
      "content": "<p>It's possible. That would make your network train on wrong / corrupted data, which could explain why it's overfitting so aggressively.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 68910,
      "author_name": "telser",
      "author_url": "",
      "post_date": "03/29/2015 22:05:56",
      "content": "<p>At least you're getting far enough to overfit; I'm struggling to reduce the training error, let alone the validation!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 68926,
      "author_name": "florianm",
      "author_url": "",
      "post_date": "03/29/2015 23:39:10",
      "content": "<p>I'm using nolearn/lasagne and I'm loading the images with skimage and putting them into a numpy array. So I'm looping through the filenames and putting each image as it's own row like this:</p>\n<p><code> image = imread(os.path.join(path, n))<br> <br> train_image[i, 0:num_features] = np.reshape(image, (1, num_features))</code></p>\n<p>Where num_features would be the 3 * height * width, and then when I load that array for the cnn I do</p>\n<p><code>X_train = X_train.reshape(-1, 3, PIXELS, PIXELS)</code></p>\n<p>Does that seem right?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 68955,
      "author_name": "akilaw",
      "author_url": "",
      "post_date": "03/30/2015 01:20:01",
      "content": "<p>I think&nbsp;imread load images in a &nbsp;(PIXELS, PIXELS, 3) matrix.</p>\n<p>May be you have to do&nbsp;&nbsp;X_train = X_train.swapaxes(1,3) to make it (3, PIXELS, PIXELS)&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 68974,
      "author_name": "florianm",
      "author_url": "",
      "post_date": "03/30/2015 04:02:06",
      "content": "<p>I thought that I would have to do the swapaxes thing as well. But I made this script and if I run it on two images it seems like everything should work out the way that I have it with skimage. But something strange is happening when I use PIL.&nbsp;</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "68694": "",
    "68716": "",
    "68723": "",
    "68855": "",
    "68873": "",
    "68891": "",
    "68910": "",
    "68926": "",
    "68955": "",
    "68974": ""
  },
  "source": "meta"
}