![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F15856017%2Feb0516914782c23ca6f71ce26a4a0f0a%2FScreenshot%202024-12-29%20093405.png?generation=1735445558456464&alt=media\")
Notebook - https://www.kaggle.com/code/vigneshwar472/baseline-residualunetse3d
\nGithub repo (ResidualUNetSE3D implementation) - https://github.com/wolny/pytorch-3dunet/tree/master
Issue - Training starts on 1x P100 GPU but it does not start on 2x T4 GPU
\n
n = len\n\nfor i in range:\n print\n model = ResidualUNetSE3D\n lm = CZIILightningModule\n logger = CSVLogger\n trainer = Trainer\n trainer.fit, \n val_dataloaders=DataLoader)\n del model, lm, logger, trainer\n print\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F15856017%2F9a609bc04602f6749d34bf7be03f903a%2FScreenshot%202024-12-29%20092937.png?generation=1735445583781330&alt=media\")
I have no idea \"why it's not working\".
", "messages": [ { "id": "3083128", "postDate": "12/29/2024 04:14:51", "content": "Notebook - https://www.kaggle.com/code/vigneshwar472/baseline-residualunetse3d
\nGithub repo (ResidualUNetSE3D implementation) - https://github.com/wolny/pytorch-3dunet/tree/master
Issue - Training starts on 1x P100 GPU but it does not start on 2x T4 GPU
\n
n = len\n\nfor i in range:\n print\n model = ResidualUNetSE3D\n lm = CZIILightningModule\n logger = CSVLogger\n trainer = Trainer\n trainer.fit, \n val_dataloaders=DataLoader)\n del model, lm, logger, trainer\n print\n
I have no idea \"why it's not working\".
", "rawMarkdown": "Notebook - https://www.kaggle.com/code/vigneshwar472/baseline-residualunetse3d\nGithub repo (ResidualUNetSE3D implementation) - https://github.com/wolny/pytorch-3dunet/tree/master\n\nIssue - Training starts on 1x P100 GPU but it does not start on 2x T4 GPU\n\n### 1x P100 GPU\n\n\n### 2x T4 GPU (Training does not start and GPUs were not in use)\n```\nn = len(folds)\n\nfor i in range(n):\n print(f'fold {i} started....')\n model = ResidualUNetSE3D(in_channels=1, out_channels=6)\n lm = CZIILightningModule(model=model)\n logger = CSVLogger(save_dir='/kaggle/working/training_results', name=f'fold_{i}')\n trainer = Trainer(accelerator='gpu',\n strategy='ddp_notebook',\n devices=2,\n precision='32',\n gradient_clip_val=None, \n logger=logger,\n max_epochs=15,\n enable_checkpointing=True,\n enable_progress_bar=True,\n enable_model_summary=False,\n inference_mode=True,\n default_root_dir='/kaggle/working/training_results',\n num_sanity_val_steps=0)\n trainer.fit(model=lm, \n train_dataloaders=DataLoader(folds[i][0], batch_size=1, num_workers=4, shuffle=True), \n val_dataloaders=DataLoader(folds[i][1], batch_size=1, num_workers=4, shuffle=False))\n del model, lm, logger, trainer\n print(f'fold {i} completed....')\n```\n\n\nI have no idea \"why it's not working\".", "votes": null }, { "id": "3083277", "postDate": "12/29/2024 08:29:22", "content": "Hi. No familiar with lightning but I'm curious… No error crash, just a never ending pause?
", "rawMarkdown": "Hi. No familiar with lightning but I'm curious... No error crash, just a never ending pause?", "votes": null }, { "id": "3083750", "postDate": "12/30/2024 01:51:42", "content": "@vigneshwar472 it used to be a pain to train notebooks with ddp in pytorch lightning. You might try strategy=\"ddp_notebook\".
", "rawMarkdown": "vigneshwar472 it used to be a pain to train notebooks with ddp in pytorch lightning. You might try [strategy=\"ddp_notebook\"](https://lightning.ai/docs/pytorch/stable/accelerators/gpu_intermediate.html#distributed-data-parallel-in-notebooks).", "votes": null }, { "id": "3083789", "postDate": "12/30/2024 03:11:39", "content": "I am using strategy=\"ddp_notebook\"
", "rawMarkdown": "I am using strategy=\"ddp_notebook\"", "votes": null }, { "id": "3083792", "postDate": "12/30/2024 03:12:43", "content": "I also opened an issue on GitHub.
\nhttps://github.com/Lightning-AI/pytorch-lightning/issues/20523
I tried the notebook you shared, and no errors occurred. That’s quite strange.![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5483160%2F583f4ffce7e8616e814e84401f5d9657%2F_405e8469-f854-43a5-8b90-4efbd0b341ed.png?generation=1735548724169279&alt=media\")
Now run it. You will encounter the situation.
", "rawMarkdown": "1. Turn on 2x T4 GPUs\n2. Uncomment the **\"strategy\"** hook in the trainer \n3. Set **devices = 2** \n\nNow run it. You will encounter the situation.", "votes": null }, { "id": "3084367", "postDate": "12/30/2024 18:15:00", "content": "The times I used T4 x2 I had to add a line like model = nn.DataParallel(model) in my code.
More explicitely:
\nmodel = torch.nn.DataParallel(HMSmodel([EEGmodel,SPECmodel,customSPECmodel]), device_ids = [0,1]).to(device)
Hi. No familiar with lightning but I'm curious… No error crash, just a never ending pause?
", "votes": null, "replies": [] }, { "id": 3083750, "author_name": "sergiosaharovskiy", "author_url": "", "post_date": "12/30/2024 01:51:42", "content": "@vigneshwar472 it used to be a pain to train notebooks with ddp in pytorch lightning. You might try strategy=\"ddp_notebook\".
", "votes": null, "replies": [ { "id": 3083789, "author_name": "vigneshwar472", "author_url": "", "post_date": "12/30/2024 03:11:39", "content": "I am using strategy=\"ddp_notebook\"
", "votes": null, "replies": [] } ] }, { "id": 3083792, "author_name": "vigneshwar472", "author_url": "", "post_date": "12/30/2024 03:12:43", "content": "I also opened an issue on GitHub.
\nhttps://github.com/Lightning-AI/pytorch-lightning/issues/20523
I tried the notebook you shared, and no errors occurred. That’s quite strange.
Now run it. You will encounter the situation.
", "votes": null, "replies": [] } ] }, { "id": 3084367, "author_name": "sacuscreed", "author_url": "", "post_date": "12/30/2024 18:15:00", "content": "The times I used T4 x2 I had to add a line like model = nn.DataParallel(model) in my code.
More explicitely:
\nmodel = torch.nn.DataParallel(HMSmodel([EEGmodel,SPECmodel,customSPECmodel]), device_ids = [0,1]).to(device)