{
  "id": 22365,
  "title": "How to fine-tune pre-trained vgg16?",
  "url": "/competitions/state-farm-distracted-driver-detection/discussion/22365",
  "author_name": "",
  "post_date": "2016-07-18T17:23:51.333Z",
  "votes": null,
  "comment_count": 1,
  "views": 883,
  "content": "<p>Hi all,</p>\n\n<p>I saw @Jia Dong's topic on fine tuning the vgg16 to get a LB score of ~0.23 and I saw many people had repeated his work. I have tried adjusting the learning rate from 5e-5 to 1e-3, and tried varying nb_epoch from 2 to 10, but I can only get a ~0.4 validation loss. (I split the train_validation based on drivers, and 0.4 was the average validation loss). I wonder if anyone can give me some suggestions on how to better fine tune the network? Or just recommend some book/paper about this. </p>\n\n<p>Thanks,</p>\n\n<p>Wenxin</p>",
  "messages": [
    {
      "id": "128244",
      "postDate": "07/18/2016 17:23:51",
      "content": "<p>Hi all,</p>\n\n<p>I saw @Jia Dong's topic on fine tuning the vgg16 to get a LB score of ~0.23 and I saw many people had repeated his work. I have tried adjusting the learning rate from 5e-5 to 1e-3, and tried varying nb_epoch from 2 to 10, but I can only get a ~0.4 validation loss. (I split the train_validation based on drivers, and 0.4 was the average validation loss). I wonder if anyone can give me some suggestions on how to better fine tune the network? Or just recommend some book/paper about this. </p>\n\n<p>Thanks,</p>\n\n<p>Wenxin</p>",
      "rawMarkdown": "Hi all,\r\n\r\nI saw @Jia Dong's topic on fine tuning the vgg16 to get a LB score of ~0.23 and I saw many people had repeated his work. I have tried adjusting the learning rate from 5e-5 to 1e-3, and tried varying nb_epoch from 2 to 10, but I can only get a ~0.4 validation loss. (I split the train_validation based on drivers, and 0.4 was the average validation loss). I wonder if anyone can give me some suggestions on how to better fine tune the network? Or just recommend some book/paper about this. \r\n\r\nThanks,\r\n\r\nWenxin",
      "votes": null
    },
    {
      "id": "128289",
      "postDate": "07/19/2016 01:04:29",
      "content": "<p>@Wenxin</p>\n\n<p>Due to the nature of random initialization of weights in neural networks, it is difficult to reproduce the identical results even with the same code. </p>\n\n<p>Basically, Jia Dong has achieved LB score of ~0.23 by ensembling 8 predictions each trained with 15 epochs. I tried to replicate his pipeline with VGG-19, but could not achieve the result as good as his. :( </p>\n\n<p>For me, learning rate of 1e-3 seems bit high for fine-tuning purpose. Perhaps your validation loss is stuck due to high learning rate. </p>\n\n<p>What is your LB score for one with validation loss ~0.4?\nDid you try fine-tuning your model further with lower learning rate?</p>\n\n<p>Chris</p>\n\n<p>[quote=Wenxin;128244]</p>\n\n<p>Hi all,</p>\n\n<p>I saw @Jia Dong's topic on fine tuning the vgg16 to get a LB score of ~0.23 and I saw many people had repeated his work. I have tried adjusting the learning rate from 5e-5 to 1e-3, and tried varying nb_epoch from 2 to 10, but I can only get a ~0.4 validation loss. (I split the train_validation based on drivers, and 0.4 was the average validation loss). I wonder if anyone can give me some suggestions on how to better fine tune the network? Or just recommend some book/paper about this. </p>\n\n<p>Thanks,</p>\n\n<p>Wenxin</p>\n\n<p>[/quote]</p>",
      "rawMarkdown": "Wenxin\r\n\r\nDue to the nature of random initialization of weights in neural networks, it is difficult to reproduce the identical results even with the same code. \r\n\r\nBasically, Jia Dong has achieved LB score of ~0.23 by ensembling 8 predictions each trained with 15 epochs. I tried to replicate his pipeline with VGG-19, but could not achieve the result as good as his. :( \r\n\r\nFor me, learning rate of 1e-3 seems bit high for fine-tuning purpose. Perhaps your validation loss is stuck due to high learning rate. \r\n\r\nWhat is your LB score for one with validation loss ~0.4?\r\nDid you try fine-tuning your model further with lower learning rate?\r\n\r\nChris\r\n\r\n[quote=Wenxin;128244]\r\n\r\nHi all,\r\n\r\nI saw @Jia Dong's topic on fine tuning the vgg16 to get a LB score of ~0.23 and I saw many people had repeated his work. I have tried adjusting the learning rate from 5e-5 to 1e-3, and tried varying nb_epoch from 2 to 10, but I can only get a ~0.4 validation loss. (I split the train_validation based on drivers, and 0.4 was the average validation loss). I wonder if anyone can give me some suggestions on how to better fine tune the network? Or just recommend some book/paper about this. \r\n\r\nThanks,\r\n\r\nWenxin\r\n\r\n\r\n\r\n[/quote]",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 128289,
      "author_name": "kweonwooj",
      "author_url": "",
      "post_date": "07/19/2016 01:04:29",
      "content": "<p>@Wenxin</p>\n\n<p>Due to the nature of random initialization of weights in neural networks, it is difficult to reproduce the identical results even with the same code. </p>\n\n<p>Basically, Jia Dong has achieved LB score of ~0.23 by ensembling 8 predictions each trained with 15 epochs. I tried to replicate his pipeline with VGG-19, but could not achieve the result as good as his. :( </p>\n\n<p>For me, learning rate of 1e-3 seems bit high for fine-tuning purpose. Perhaps your validation loss is stuck due to high learning rate. </p>\n\n<p>What is your LB score for one with validation loss ~0.4?\nDid you try fine-tuning your model further with lower learning rate?</p>\n\n<p>Chris</p>\n\n<p>[quote=Wenxin;128244]</p>\n\n<p>Hi all,</p>\n\n<p>I saw @Jia Dong's topic on fine tuning the vgg16 to get a LB score of ~0.23 and I saw many people had repeated his work. I have tried adjusting the learning rate from 5e-5 to 1e-3, and tried varying nb_epoch from 2 to 10, but I can only get a ~0.4 validation loss. (I split the train_validation based on drivers, and 0.4 was the average validation loss). I wonder if anyone can give me some suggestions on how to better fine tune the network? Or just recommend some book/paper about this. </p>\n\n<p>Thanks,</p>\n\n<p>Wenxin</p>\n\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "128244": "Hi all,\r\n\r\nI saw @Jia Dong's topic on fine tuning the vgg16 to get a LB score of ~0.23 and I saw many people had repeated his work. I have tried adjusting the learning rate from 5e-5 to 1e-3, and tried varying nb_epoch from 2 to 10, but I can only get a ~0.4 validation loss. (I split the train_validation based on drivers, and 0.4 was the average validation loss). I wonder if anyone can give me some suggestions on how to better fine tune the network? Or just recommend some book/paper about this. \r\n\r\nThanks,\r\n\r\nWenxin",
    "128289": "Wenxin\r\n\r\nDue to the nature of random initialization of weights in neural networks, it is difficult to reproduce the identical results even with the same code. \r\n\r\nBasically, Jia Dong has achieved LB score of ~0.23 by ensembling 8 predictions each trained with 15 epochs. I tried to replicate his pipeline with VGG-19, but could not achieve the result as good as his. :( \r\n\r\nFor me, learning rate of 1e-3 seems bit high for fine-tuning purpose. Perhaps your validation loss is stuck due to high learning rate. \r\n\r\nWhat is your LB score for one with validation loss ~0.4?\r\nDid you try fine-tuning your model further with lower learning rate?\r\n\r\nChris\r\n\r\n[quote=Wenxin;128244]\r\n\r\nHi all,\r\n\r\nI saw @Jia Dong's topic on fine tuning the vgg16 to get a LB score of ~0.23 and I saw many people had repeated his work. I have tried adjusting the learning rate from 5e-5 to 1e-3, and tried varying nb_epoch from 2 to 10, but I can only get a ~0.4 validation loss. (I split the train_validation based on drivers, and 0.4 was the average validation loss). I wonder if anyone can give me some suggestions on how to better fine tune the network? Or just recommend some book/paper about this. \r\n\r\nThanks,\r\n\r\nWenxin\r\n\r\n\r\n\r\n[/quote]"
  },
  "source": "meta"
}