{
  "id": 105771,
  "title": "How to train 2015data as a pre-training model of 2019data？",
  "url": "/competitions/aptos2019-blindness-detection/discussion/105771",
  "author_name": "",
  "post_date": "2019-08-26T09:46:57.752909800Z",
  "votes": 2,
  "comment_count": 17,
  "views": 0,
  "content": "<p>My imagesize is 256 and my model is b3. I use 2015data as the training set and 2019 as the val set. The initial value of lr is 0.001, and lr decreases by 10 times every 5 epochs, so my loss remains around 0.4 until the 30epoch. How can I choose the appropriate model as the pre-training model of 2019data?I tried to use the model with the minimum loss equal to 0.4 as the pre-training model of 2019data, and trained 10 epochs, resulting in 0.714 LB.</p>",
  "messages": [
    {
      "id": "608073",
      "postDate": "08/26/2019 09:46:57",
      "content": "<p>My imagesize is 256 and my model is b3. I use 2015data as the training set and 2019 as the val set. The initial value of lr is 0.001, and lr decreases by 10 times every 5 epochs, so my loss remains around 0.4 until the 30epoch. How can I choose the appropriate model as the pre-training model of 2019data?I tried to use the model with the minimum loss equal to 0.4 as the pre-training model of 2019data, and trained 10 epochs, resulting in 0.714 LB.</p>",
      "rawMarkdown": "My imagesize is 256 and my model is b3. I use 2015data as the training set and 2019 as the val set. The initial value of lr is 0.001, and lr decreases by 10 times every 5 epochs, so my loss remains around 0.4 until the 30epoch. How can I choose the appropriate model as the pre-training model of 2019data?I tried to use the model with the minimum loss equal to 0.4 as the pre-training model of 2019data, and trained 10 epochs, resulting in 0.714 LB.",
      "votes": null
    },
    {
      "id": "608211",
      "postDate": "08/26/2019 13:55:20",
      "content": "<p>one key tip in this competition is training longer and pick a better model</p>",
      "rawMarkdown": "one key tip in this competition is training longer and pick a better model",
      "votes": null
    },
    {
      "id": "608577",
      "postDate": "08/27/2019 00:50:43",
      "content": "<p>I found that no matter how low lr ,the loss never drops is when training 2015data （val is 2019data）. I would like to know how many epochs you have trained =）</p>",
      "rawMarkdown": "I found that no matter how low lr ,the loss never drops is when training 2015data （val is 2019data）. I would like to know how many epochs you have trained =）",
      "votes": null
    },
    {
      "id": "609020",
      "postDate": "08/27/2019 10:48:53",
      "content": "<p>pick a reasonable learning rate is quite important, if the loss never drop may you can try the warm up strategy to start training</p>",
      "rawMarkdown": "pick a reasonable learning rate is quite important, if the loss never drop may you can try the warm up strategy to start training",
      "votes": null
    },
    {
      "id": "609022",
      "postDate": "08/27/2019 10:49:36",
      "content": "<p>personally,  i trained 2015data for 8 to 10 epochs</p>",
      "rawMarkdown": "personally,  i trained 2015data for 8 to 10 epochs",
      "votes": null
    },
    {
      "id": "609034",
      "postDate": "08/27/2019 11:06:32",
      "content": "<p>hi, what about the val_loss of 2019data when you pretrained on 2015data?</p>",
      "rawMarkdown": "hi, what about the val_loss of 2019data when you pretrained on 2015data?",
      "votes": null
    },
    {
      "id": "609124",
      "postDate": "08/27/2019 12:31:21",
      "content": "<p><a href=\"/garybios\">@garybios</a> what about your loss? I still dont use 2015</p>",
      "rawMarkdown": "garybios what about your loss? I still dont use 2015",
      "votes": null
    },
    {
      "id": "609172",
      "postDate": "08/27/2019 13:14:37",
      "content": "<p>around 0.42, not a great val loss.</p>",
      "rawMarkdown": "around 0.42, not a great val loss.",
      "votes": null
    },
    {
      "id": "609220",
      "postDate": "08/27/2019 14:01:55",
      "content": "<p><a href=\"/leixiang\">@leixiang</a> thanks！</p>",
      "rawMarkdown": "leixiang thanks！",
      "votes": null
    },
    {
      "id": "609222",
      "postDate": "08/27/2019 14:07:04",
      "content": "<p><a href=\"/garybios\">@garybios</a>  I'm also around 0.4</p>",
      "rawMarkdown": "garybios  I'm also around 0.4",
      "votes": null
    },
    {
      "id": "609264",
      "postDate": "08/27/2019 14:49:22",
      "content": "<p><a href=\"https://www.kaggle.com/valanm\">@Val An</a>why don‘t you use 2015data?</p>",
      "rawMarkdown": "[@Val An](https://www.kaggle.com/valanm)why don‘t you use 2015data?",
      "votes": null
    },
    {
      "id": "609267",
      "postDate": "08/27/2019 14:50:34",
      "content": "<p><a href=\"https://www.kaggle.com/yangfan556677\">@Yangfan</a>彭于晏真帅😄 </p>",
      "rawMarkdown": "[@Yangfan](https://www.kaggle.com/yangfan556677)彭于晏真帅😄",
      "votes": null
    },
    {
      "id": "609276",
      "postDate": "08/27/2019 14:57:29",
      "content": "<p><a href=\"/garybios\">@garybios</a> 没错是我！😁 </p>",
      "rawMarkdown": "garybios 没错是我！😁",
      "votes": null
    },
    {
      "id": "609285",
      "postDate": "08/27/2019 15:02:40",
      "content": "<p>my plan was to do a bunch of experiments with 2019 and then move to 2015 but in the mean time all this mess with GPU limit and broken kernels happened so I am kind of stuck right now (i rely on free GPU only)... </p>",
      "rawMarkdown": "my plan was to do a bunch of experiments with 2019 and then move to 2015 but in the mean time all this mess with GPU limit and broken kernels happened so I am kind of stuck right now (i rely on free GPU only)...",
      "votes": null
    },
    {
      "id": "609304",
      "postDate": "08/27/2019 15:16:00",
      "content": "<p><a href=\"/valanm\">@valanm</a> Good idea. I can't even use kernel now.Unable to submit  answers.</p>",
      "rawMarkdown": "valanm Good idea. I can't even use kernel now.Unable to submit  answers.",
      "votes": null
    },
    {
      "id": "609311",
      "postDate": "08/27/2019 15:24:42",
      "content": "<p>your loss is MSE?</p>",
      "rawMarkdown": "your loss is MSE?",
      "votes": null
    },
    {
      "id": "609321",
      "postDate": "08/27/2019 15:40:44",
      "content": "<p>yes, mse.</p>",
      "rawMarkdown": "yes, mse.",
      "votes": null
    },
    {
      "id": "609517",
      "postDate": "08/27/2019 19:54:20",
      "content": "<p>thanks, what kappa you guys get with that loss? have you tried to submit it directly or you do finetuning on 2019 data? </p>",
      "rawMarkdown": "thanks, what kappa you guys get with that loss? have you tried to submit it directly or you do finetuning on 2019 data?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 608211,
      "author_name": "leixiang",
      "author_url": "",
      "post_date": "08/26/2019 13:55:20",
      "content": "<p>one key tip in this competition is training longer and pick a better model</p>",
      "votes": null,
      "replies": [
        {
          "id": 608577,
          "author_name": "yangfan556677",
          "author_url": "",
          "post_date": "08/27/2019 00:50:43",
          "content": "<p>I found that no matter how low lr ,the loss never drops is when training 2015data （val is 2019data）. I would like to know how many epochs you have trained =）</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 609020,
          "author_name": "leixiang",
          "author_url": "",
          "post_date": "08/27/2019 10:48:53",
          "content": "<p>pick a reasonable learning rate is quite important, if the loss never drop may you can try the warm up strategy to start training</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 609022,
          "author_name": "leixiang",
          "author_url": "",
          "post_date": "08/27/2019 10:49:36",
          "content": "<p>personally,  i trained 2015data for 8 to 10 epochs</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 609034,
          "author_name": "garybios",
          "author_url": "",
          "post_date": "08/27/2019 11:06:32",
          "content": "<p>hi, what about the val_loss of 2019data when you pretrained on 2015data?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 609124,
          "author_name": "valanm",
          "author_url": "",
          "post_date": "08/27/2019 12:31:21",
          "content": "<p><a href=\"/garybios\">@garybios</a> what about your loss? I still dont use 2015</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 609172,
          "author_name": "garybios",
          "author_url": "",
          "post_date": "08/27/2019 13:14:37",
          "content": "<p>around 0.42, not a great val loss.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 609220,
          "author_name": "yangfan556677",
          "author_url": "",
          "post_date": "08/27/2019 14:01:55",
          "content": "<p><a href=\"/leixiang\">@leixiang</a> thanks！</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 609222,
          "author_name": "yangfan556677",
          "author_url": "",
          "post_date": "08/27/2019 14:07:04",
          "content": "<p><a href=\"/garybios\">@garybios</a>  I'm also around 0.4</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 609264,
          "author_name": "garybios",
          "author_url": "",
          "post_date": "08/27/2019 14:49:22",
          "content": "<p><a href=\"https://www.kaggle.com/valanm\">@Val An</a>why don‘t you use 2015data?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 609267,
          "author_name": "garybios",
          "author_url": "",
          "post_date": "08/27/2019 14:50:34",
          "content": "<p><a href=\"https://www.kaggle.com/yangfan556677\">@Yangfan</a>彭于晏真帅😄 </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 609276,
          "author_name": "yangfan556677",
          "author_url": "",
          "post_date": "08/27/2019 14:57:29",
          "content": "<p><a href=\"/garybios\">@garybios</a> 没错是我！😁 </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 609285,
          "author_name": "valanm",
          "author_url": "",
          "post_date": "08/27/2019 15:02:40",
          "content": "<p>my plan was to do a bunch of experiments with 2019 and then move to 2015 but in the mean time all this mess with GPU limit and broken kernels happened so I am kind of stuck right now (i rely on free GPU only)... </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 609304,
          "author_name": "yangfan556677",
          "author_url": "",
          "post_date": "08/27/2019 15:16:00",
          "content": "<p><a href=\"/valanm\">@valanm</a> Good idea. I can't even use kernel now.Unable to submit  answers.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 609311,
          "author_name": "valanm",
          "author_url": "",
          "post_date": "08/27/2019 15:24:42",
          "content": "<p>your loss is MSE?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 609321,
          "author_name": "garybios",
          "author_url": "",
          "post_date": "08/27/2019 15:40:44",
          "content": "<p>yes, mse.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 609517,
          "author_name": "valanm",
          "author_url": "",
          "post_date": "08/27/2019 19:54:20",
          "content": "<p>thanks, what kappa you guys get with that loss? have you tried to submit it directly or you do finetuning on 2019 data? </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "608073": "My imagesize is 256 and my model is b3. I use 2015data as the training set and 2019 as the val set. The initial value of lr is 0.001, and lr decreases by 10 times every 5 epochs, so my loss remains around 0.4 until the 30epoch. How can I choose the appropriate model as the pre-training model of 2019data?I tried to use the model with the minimum loss equal to 0.4 as the pre-training model of 2019data, and trained 10 epochs, resulting in 0.714 LB.",
    "608211": "one key tip in this competition is training longer and pick a better model",
    "608577": "I found that no matter how low lr ,the loss never drops is when training 2015data （val is 2019data）. I would like to know how many epochs you have trained =）",
    "609020": "pick a reasonable learning rate is quite important, if the loss never drop may you can try the warm up strategy to start training",
    "609022": "personally,  i trained 2015data for 8 to 10 epochs",
    "609034": "hi, what about the val_loss of 2019data when you pretrained on 2015data?",
    "609124": "garybios what about your loss? I still dont use 2015",
    "609172": "around 0.42, not a great val loss.",
    "609220": "leixiang thanks！",
    "609222": "garybios  I'm also around 0.4",
    "609264": "[@Val An](https://www.kaggle.com/valanm)why don‘t you use 2015data?",
    "609267": "[@Yangfan](https://www.kaggle.com/yangfan556677)彭于晏真帅😄",
    "609276": "garybios 没错是我！😁",
    "609285": "my plan was to do a bunch of experiments with 2019 and then move to 2015 but in the mean time all this mess with GPU limit and broken kernels happened so I am kind of stuck right now (i rely on free GPU only)...",
    "609304": "valanm Good idea. I can't even use kernel now.Unable to submit  answers.",
    "609311": "your loss is MSE?",
    "609321": "yes, mse.",
    "609517": "thanks, what kappa you guys get with that loss? have you tried to submit it directly or you do finetuning on 2019 data?"
  },
  "source": "meta"
}