{
  "id": 212864,
  "title": "Why the same code some times runs 4 times slower?",
  "url": "/competitions/cassava-leaf-disease-classification/discussion/212864",
  "author_name": "",
  "post_date": "2021-01-20T14:32:02.725530800Z",
  "votes": null,
  "comment_count": 3,
  "views": 0,
  "content": "<p>I noticed that some times my training takes  7-8 minutes per epoch,  sometime takes about 30 minutes per epoch.  The code is the same.  Are there different type of GPUs?   I checked the GPU info, could not see any difference.  Anyone can explain it?</p>",
  "messages": [
    {
      "id": "1161380",
      "postDate": "01/20/2021 14:32:02",
      "content": "<p>I noticed that some times my training takes  7-8 minutes per epoch,  sometime takes about 30 minutes per epoch.  The code is the same.  Are there different type of GPUs?   I checked the GPU info, could not see any difference.  Anyone can explain it?</p>",
      "rawMarkdown": "I noticed that some times my training takes  7-8 minutes per epoch,  sometime takes about 30 minutes per epoch.  The code is the same.  Are there different type of GPUs?   I checked the GPU info, could not see any difference.  Anyone can explain it?",
      "votes": null
    },
    {
      "id": "1161386",
      "postDate": "01/20/2021 14:39:12",
      "content": "<p>code exactly same? no change at all?</p>",
      "rawMarkdown": "code exactly same? no change at all?",
      "votes": null
    },
    {
      "id": "1161433",
      "postDate": "01/20/2021 15:10:40",
      "content": "<p>I noticed that the slower model h5 file is 2-3 times as big as the faster h5 file.  The model configuration is the exactly same. The code used to save the model is always model.save('path_name').  Not sure why the model h5 file changes.   Is there an option to save the model so that next time,  training it will be faster?</p>",
      "rawMarkdown": "I noticed that the slower model h5 file is 2-3 times as big as the faster h5 file.  The model configuration is the exactly same. The code used to save the model is always model.save('path_name').  Not sure why the model h5 file changes.   Is there an option to save the model so that next time,  training it will be faster?",
      "votes": null
    },
    {
      "id": "1161446",
      "postDate": "01/20/2021 15:15:50",
      "content": "<p>yes.  normally save the model, and reload for further training.  I noticed that the slower model h5 file is 2-3 times as big as the faster h5 file.  I wonder if the larger model   disable the mixed pression.  </p>",
      "rawMarkdown": "yes.  normally save the model, and reload for further training.  I noticed that the slower model h5 file is 2-3 times as big as the faster h5 file.  I wonder if the larger model   disable the mixed pression.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1161386,
      "author_name": "mobassir",
      "author_url": "",
      "post_date": "01/20/2021 14:39:12",
      "content": "<p>code exactly same? no change at all?</p>",
      "votes": null,
      "replies": [
        {
          "id": 1161446,
          "author_name": "luqing2",
          "author_url": "",
          "post_date": "01/20/2021 15:15:50",
          "content": "<p>yes.  normally save the model, and reload for further training.  I noticed that the slower model h5 file is 2-3 times as big as the faster h5 file.  I wonder if the larger model   disable the mixed pression.  </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1161433,
      "author_name": "luqing2",
      "author_url": "",
      "post_date": "01/20/2021 15:10:40",
      "content": "<p>I noticed that the slower model h5 file is 2-3 times as big as the faster h5 file.  The model configuration is the exactly same. The code used to save the model is always model.save('path_name').  Not sure why the model h5 file changes.   Is there an option to save the model so that next time,  training it will be faster?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1161380": "I noticed that some times my training takes  7-8 minutes per epoch,  sometime takes about 30 minutes per epoch.  The code is the same.  Are there different type of GPUs?   I checked the GPU info, could not see any difference.  Anyone can explain it?",
    "1161386": "code exactly same? no change at all?",
    "1161433": "I noticed that the slower model h5 file is 2-3 times as big as the faster h5 file.  The model configuration is the exactly same. The code used to save the model is always model.save('path_name').  Not sure why the model h5 file changes.   Is there an option to save the model so that next time,  training it will be faster?",
    "1161446": "yes.  normally save the model, and reload for further training.  I noticed that the slower model h5 file is 2-3 times as big as the faster h5 file.  I wonder if the larger model   disable the mixed pression."
  },
  "source": "meta"
}