have been running my notebook for almost an hour unaccelerated, how long should the submission normally take? how can i view log traces on the submission itself to see what is taking so long?
\nEDIT:
\nit seems i am unable to utilize GPU resources even though I am setting torch.device(\"cuda:0\") when available and .to(DEVICE) on both model and tensors.
this is my starter notebook https://www.kaggle.com/alelat/start-cuda
\nin the code what takes so long is looping through my dataloader. i know you can play with num_workers on the dataloader but when i do i get this error
\nRuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
\ngiven this open issue on pytorch github
\nhttps://github.com/pytorch/pytorch/issues/40403
\ni tried forcing spawn on multiprocessing
\nmultiprocessing.set_start_method('spawn', force=True)
\nto which i get this error
\nFile \"/opt/conda/lib/python3.7/multiprocessing/spawn.py\", line 105, in spawn_main
\n exitcode = _main(fd)
\n File \"/opt/conda/lib/python3.7/multiprocessing/spawn.py\", line 115, in _main
\n self = reduction.pickle.load(from_parent)
\nAttributeError: Can't get attribute 'FOGDataset' on
Out of the 400+ subs I can imagine many have successfully used GPU's in pytorch esp given that on CPU only my submission on a simple LSTM gives runtime errors.
\ndoes anyone have a better way of using GPU's with Dataloaders and Datasets?
", "messages": [ { "id": "2221051", "postDate": "04/13/2023 23:43:13", "content": "have been running my notebook for almost an hour unaccelerated, how long should the submission normally take? how can i view log traces on the submission itself to see what is taking so long?
\nEDIT:
\nit seems i am unable to utilize GPU resources even though I am setting torch.device(\"cuda:0\") when available and .to(DEVICE) on both model and tensors.
this is my starter notebook https://www.kaggle.com/alelat/start-cuda
\nin the code what takes so long is looping through my dataloader. i know you can play with num_workers on the dataloader but when i do i get this error
\nRuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
\ngiven this open issue on pytorch github
\nhttps://github.com/pytorch/pytorch/issues/40403
\ni tried forcing spawn on multiprocessing
\nmultiprocessing.set_start_method('spawn', force=True)
\nto which i get this error
\nFile \"/opt/conda/lib/python3.7/multiprocessing/spawn.py\", line 105, in spawn_main
\n exitcode = _main(fd)
\n File \"/opt/conda/lib/python3.7/multiprocessing/spawn.py\", line 115, in _main
\n self = reduction.pickle.load(from_parent)
\nAttributeError: Can't get attribute 'FOGDataset' on
Out of the 400+ subs I can imagine many have successfully used GPU's in pytorch esp given that on CPU only my submission on a simple LSTM gives runtime errors.
\ndoes anyone have a better way of using GPU's with Dataloaders and Datasets?
", "rawMarkdown": "have been running my notebook for almost an hour unaccelerated, how long should the submission normally take? how can i view log traces on the submission itself to see what is taking so long?\n\nEDIT:\nit seems i am unable to utilize GPU resources even though I am setting torch.device(\"cuda:0\") when available and .to(DEVICE) on both model and tensors.\n\nthis is my starter notebook https://www.kaggle.com/alelat/start-cuda \n\nin the code what takes so long is looping through my dataloader. i know you can play with num_workers on the dataloader but when i do i get this error\n\nRuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method\n\ngiven this open issue on pytorch github\n\nhttps://github.com/pytorch/pytorch/issues/40403\n\ni tried forcing spawn on multiprocessing \n\nmultiprocessing.set_start_method('spawn', force=True)\n\nto which i get this error\n\n File \"/opt/conda/lib/python3.7/multiprocessing/spawn.py\", line 105, in spawn_main\n exitcode = _main(fd)\n File \"/opt/conda/lib/python3.7/multiprocessing/spawn.py\", line 115, in _main\n self = reduction.pickle.load(from_parent)\nAttributeError: Can't get attribute 'FOGDataset' onIt depends on your code. Your code will be run, and the submission file will be evaluated. It is better if you can run your code on your computer first to make sure it works. Then you submit it in a notebook for evaluation. If you run your notebook on a CPU, whatever time you got from your personal computer, double or triple it for the submission notebook.
", "rawMarkdown": "It depends on your code. Your code will be run, and the submission file will be evaluated. It is better if you can run your code on your computer first to make sure it works. Then you submit it in a notebook for evaluation. If you run your notebook on a CPU, whatever time you got from your personal computer, double or triple it for the submission notebook.", "votes": null }, { "id": "2221767", "postDate": "04/14/2023 14:49:41", "content": "So my notebook takes less than 10 mins to run from start to finish where it outputs a submission.csv file correctly formatted.
\nWith CPU only it took 16 hrs, notebook timed-out without score.
\nI am now 7 hrs into a GPU run submission.
\nQuite confused by the difference as it a very basic LSTM model, reducing mem on datasets, 1000 test batch size. its very basic uncomplicated code that as I said runs all cells in less that 10 mins normally.
\nThe frustration is not being able to see the output. the log trace goes up to 755s or 12m and nothing more. There are no while loops where it can go into infinite recursion. Quite perplexed by this experience as it my first competition sub
", "rawMarkdown": "So my notebook takes less than 10 mins to run from start to finish where it outputs a submission.csv file correctly formatted. \n\nWith CPU only it took 16 hrs, notebook timed-out without score.\n\nI am now 7 hrs into a GPU run submission.\n\nQuite confused by the difference as it a very basic LSTM model, reducing mem on datasets, 1000 test batch size. its very basic uncomplicated code that as I said runs all cells in less that 10 mins normally.\n\nThe frustration is not being able to see the output. the log trace goes up to 755s or 12m and nothing more. There are no while loops where it can go into infinite recursion. Quite perplexed by this experience as it my first competition sub", "votes": null }, { "id": "2221773", "postDate": "04/14/2023 14:55:58", "content": "what is ever more strange is that 755 seconds is actually where the test ends. as in it has run all the code required and saved the submission dataframe like so sub.to_csv('submission.csv', index=False)
\nperhaps the file path is incorrect?
", "rawMarkdown": "what is ever more strange is that 755 seconds is actually where the test ends. as in it has run all the code required and saved the submission dataframe like so sub.to_csv('submission.csv', index=False)\n\nperhaps the file path is incorrect?", "votes": null }, { "id": "2222561", "postDate": "04/15/2023 11:06:47", "content": "One good metric to use is to see how much time it takes to do a prediction on the defog and tdcsfog folders of the train dataset. It could weed out some inefficiencies in you code. For me a single model inference GPU notebook takes about 23-25 minutes to complete with the hidden data set while the visible test data takes 2 min and 31 seconds.
ok i have completely missed the mark here with regards to utilizing GPU resources in pytorch, would you mind helping me out pls.
\nit seems my notebook https://www.kaggle.com/alelat/start-cuda does not utilize GPU resources
\nwhen i am specifying
\nDEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")
\nand then after either model.train() or model.eval() i specify
\nmodel = model.to(DEVICE)
\nannoyingly everytime i make a new tensor I also have to cast it tensor.to(DEVICE) and this is repeated in my custom Dataset class as well as every time i need to convert something into a tensor.
\nwhat is the best way utilize the GPU correctly in pytorch given how i structured my code in the above notebook?
", "rawMarkdown": "ok i have completely missed the mark here with regards to utilizing GPU resources in pytorch, would you mind helping me out pls.\n\nit seems my notebook https://www.kaggle.com/alelat/start-cuda does not utilize GPU resources\n\nwhen i am specifying \n\nDEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n\nand then after either model.train() or model.eval() i specify\n\nmodel = model.to(DEVICE)\n\nannoyingly everytime i make a new tensor I also have to cast it tensor.to(DEVICE) and this is repeated in my custom Dataset class as well as every time i need to convert something into a tensor.\n\nwhat is the best way utilize the GPU correctly in pytorch given how i structured my code in the above notebook?", "votes": null }, { "id": "2222813", "postDate": "04/15/2023 15:26:11", "content": "has anyone been able to train this on CPU only?
", "rawMarkdown": "has anyone been able to train this on CPU only?", "votes": null }, { "id": "2222817", "postDate": "04/15/2023 15:28:58", "content": "currently on CPU only this is training on average 45s per iteration and needs to run through 7000 ish iterations so it would mean 118 hours of training time. is this dataset impossible to train on CPU alone? Or am i being silly inefficient in my code somewhere?
", "rawMarkdown": "currently on CPU only this is training on average 45s per iteration and needs to run through 7000 ish iterations so it would mean 118 hours of training time. is this dataset impossible to train on CPU alone? Or am i being silly inefficient in my code somewhere?", "votes": null }, { "id": "2222827", "postDate": "04/15/2023 15:40:28", "content": "it seems my biggest time sink is the dataloader itself and it takes ages to iterate through each batch. i have tried specifying num_workers=MAX_GPU_WORKERS
\nMAX_GPU_WORKERS = torch.cuda.device_count()
\nbut it is still crazy slow and also doesnt change the fact my GPU usage is 0
", "rawMarkdown": "it seems my biggest time sink is the dataloader itself and it takes ages to iterate through each batch. i have tried specifying num_workers=MAX_GPU_WORKERS \n\nMAX_GPU_WORKERS = torch.cuda.device_count()\n\nbut it is still crazy slow and also doesnt change the fact my GPU usage is 0", "votes": null } ], "comments": [ { "id": 2221696, "author_name": "jdeneva", "author_url": "", "post_date": "04/14/2023 13:30:07", "content": "It depends on your code. Your code will be run, and the submission file will be evaluated. It is better if you can run your code on your computer first to make sure it works. Then you submit it in a notebook for evaluation. If you run your notebook on a CPU, whatever time you got from your personal computer, double or triple it for the submission notebook.
", "votes": null, "replies": [ { "id": 2221767, "author_name": "alelat", "author_url": "", "post_date": "04/14/2023 14:49:41", "content": "So my notebook takes less than 10 mins to run from start to finish where it outputs a submission.csv file correctly formatted.
\nWith CPU only it took 16 hrs, notebook timed-out without score.
\nI am now 7 hrs into a GPU run submission.
\nQuite confused by the difference as it a very basic LSTM model, reducing mem on datasets, 1000 test batch size. its very basic uncomplicated code that as I said runs all cells in less that 10 mins normally.
\nThe frustration is not being able to see the output. the log trace goes up to 755s or 12m and nothing more. There are no while loops where it can go into infinite recursion. Quite perplexed by this experience as it my first competition sub
", "votes": null, "replies": [ { "id": 2221773, "author_name": "alelat", "author_url": "", "post_date": "04/14/2023 14:55:58", "content": "what is ever more strange is that 755 seconds is actually where the test ends. as in it has run all the code required and saved the submission dataframe like so sub.to_csv('submission.csv', index=False)
\nperhaps the file path is incorrect?
", "votes": null, "replies": [] } ] } ] }, { "id": 2222561, "author_name": "coderrkj", "author_url": "", "post_date": "04/15/2023 11:06:47", "content": "One good metric to use is to see how much time it takes to do a prediction on the defog and tdcsfog folders of the train dataset. It could weed out some inefficiencies in you code. For me a single model inference GPU notebook takes about 23-25 minutes to complete with the hidden data set while the visible test data takes 2 min and 31 seconds.
ok i have completely missed the mark here with regards to utilizing GPU resources in pytorch, would you mind helping me out pls.
\nit seems my notebook https://www.kaggle.com/alelat/start-cuda does not utilize GPU resources
\nwhen i am specifying
\nDEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")
\nand then after either model.train() or model.eval() i specify
\nmodel = model.to(DEVICE)
\nannoyingly everytime i make a new tensor I also have to cast it tensor.to(DEVICE) and this is repeated in my custom Dataset class as well as every time i need to convert something into a tensor.
\nwhat is the best way utilize the GPU correctly in pytorch given how i structured my code in the above notebook?
", "votes": null, "replies": [ { "id": 2222817, "author_name": "alelat", "author_url": "", "post_date": "04/15/2023 15:28:58", "content": "currently on CPU only this is training on average 45s per iteration and needs to run through 7000 ish iterations so it would mean 118 hours of training time. is this dataset impossible to train on CPU alone? Or am i being silly inefficient in my code somewhere?
", "votes": null, "replies": [] }, { "id": 2222827, "author_name": "alelat", "author_url": "", "post_date": "04/15/2023 15:40:28", "content": "it seems my biggest time sink is the dataloader itself and it takes ages to iterate through each batch. i have tried specifying num_workers=MAX_GPU_WORKERS
\nMAX_GPU_WORKERS = torch.cuda.device_count()
\nbut it is still crazy slow and also doesnt change the fact my GPU usage is 0
", "votes": null, "replies": [] } ] } ] }, { "id": 2222813, "author_name": "alelat", "author_url": "", "post_date": "04/15/2023 15:26:11", "content": "has anyone been able to train this on CPU only?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2221051": "have been running my notebook for almost an hour unaccelerated, how long should the submission normally take? how can i view log traces on the submission itself to see what is taking so long?\n\nEDIT:\nit seems i am unable to utilize GPU resources even though I am setting torch.device(\"cuda:0\") when available and .to(DEVICE) on both model and tensors.\n\nthis is my starter notebook https://www.kaggle.com/alelat/start-cuda \n\nin the code what takes so long is looping through my dataloader. i know you can play with num_workers on the dataloader but when i do i get this error\n\nRuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method\n\ngiven this open issue on pytorch github\n\nhttps://github.com/pytorch/pytorch/issues/40403\n\ni tried forcing spawn on multiprocessing \n\nmultiprocessing.set_start_method('spawn', force=True)\n\nto which i get this error\n\n File \"/opt/conda/lib/python3.7/multiprocessing/spawn.py\", line 105, in spawn_main\n exitcode = _main(fd)\n File \"/opt/conda/lib/python3.7/multiprocessing/spawn.py\", line 115, in _main\n self = reduction.pickle.load(from_parent)\nAttributeError: Can't get attribute 'FOGDataset' on