{
  "id": 473268,
  "title": "Weird behavior of GPU T4 x 2 (environment problem ?)",
  "url": "/competitions/blood-vessel-segmentation/discussion/473268",
  "author_name": "",
  "post_date": "2024-02-04T06:16:36.618987900Z",
  "votes": 1,
  "comment_count": 1,
  "views": 0,
  "content": "<p>If for example, I fork (\"copy and edit\") the <a href=\"https://www.kaggle.com/code/yoyobar/2-5d-cutting-model-baseline-training/\" target=\"_blank\">YOYOBAR - 2.5d Cutting model baseline [training]</a> notebook (thanks <a href=\"https://www.kaggle.com/yoyobar\" target=\"_blank\">@yoyobar</a> for excellent public notebook)  then \"GPU T4 x 2\" works very fast. But if I create a new blank notebook and copy code in it then 2xT4 works ~8 times slower.</p>\n<p>But P100 not affected</p>\n<p>It's probably due to the launch environment. In the original notebook, when the environment  \"Pin to the original environment (2023-11-15)\" it works fast (in edit mode in the right panel - \"Environment\" option). But if I change to \"Always use latest environment\" it also starts working ~8 times slower.</p>",
  "messages": [
    {
      "id": "2634988",
      "postDate": "02/04/2024 06:16:36",
      "content": "<p>If for example, I fork (\"copy and edit\") the <a href=\"https://www.kaggle.com/code/yoyobar/2-5d-cutting-model-baseline-training/\" target=\"_blank\">YOYOBAR - 2.5d Cutting model baseline [training]</a> notebook (thanks <a href=\"https://www.kaggle.com/yoyobar\" target=\"_blank\">@yoyobar</a> for excellent public notebook)  then \"GPU T4 x 2\" works very fast. But if I create a new blank notebook and copy code in it then 2xT4 works ~8 times slower.</p>\n<p>But P100 not affected</p>\n<p>It's probably due to the launch environment. In the original notebook, when the environment  \"Pin to the original environment (2023-11-15)\" it works fast (in edit mode in the right panel - \"Environment\" option). But if I change to \"Always use latest environment\" it also starts working ~8 times slower.</p>",
      "rawMarkdown": "If for example, I fork (\"copy and edit\") the [YOYOBAR - 2.5d Cutting model baseline [training]](https://www.kaggle.com/code/yoyobar/2-5d-cutting-model-baseline-training/) notebook (thanks @yoyobar for excellent public notebook)  then \"GPU T4 x 2\" works very fast. But if I create a new blank notebook and copy code in it then 2xT4 works ~8 times slower.\n\nBut P100 not affected\n\nIt's probably due to the launch environment. In the original notebook, when the environment  \"Pin to the original environment (2023-11-15)\" it works fast (in edit mode in the right panel - \"Environment\" option). But if I change to \"Always use latest environment\" it also starts working ~8 times slower.",
      "votes": null
    },
    {
      "id": "2635550",
      "postDate": "02/04/2024 13:24:46",
      "content": "<p>That's odd. Maybe check the pytorch versions? You could test when they have the same pytorch versions and that may be telling, then just use the fastest one. I agree it seems odd a newer version wouldn't support multi GPU training as efficiently though.</p>",
      "rawMarkdown": "That's odd. Maybe check the pytorch versions? You could test when they have the same pytorch versions and that may be telling, then just use the fastest one. I agree it seems odd a newer version wouldn't support multi GPU training as efficiently though.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2635550,
      "author_name": "chemdatafarmer",
      "author_url": "",
      "post_date": "02/04/2024 13:24:46",
      "content": "<p>That's odd. Maybe check the pytorch versions? You could test when they have the same pytorch versions and that may be telling, then just use the fastest one. I agree it seems odd a newer version wouldn't support multi GPU training as efficiently though.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2634988": "If for example, I fork (\"copy and edit\") the [YOYOBAR - 2.5d Cutting model baseline [training]](https://www.kaggle.com/code/yoyobar/2-5d-cutting-model-baseline-training/) notebook (thanks @yoyobar for excellent public notebook)  then \"GPU T4 x 2\" works very fast. But if I create a new blank notebook and copy code in it then 2xT4 works ~8 times slower.\n\nBut P100 not affected\n\nIt's probably due to the launch environment. In the original notebook, when the environment  \"Pin to the original environment (2023-11-15)\" it works fast (in edit mode in the right panel - \"Environment\" option). But if I change to \"Always use latest environment\" it also starts working ~8 times slower.",
    "2635550": "That's odd. Maybe check the pytorch versions? You could test when they have the same pytorch versions and that may be telling, then just use the fastest one. I agree it seems odd a newer version wouldn't support multi GPU training as efficiently though."
  },
  "source": "meta"
}