![\"\"](\"https://devblogs.nvidia.com/wp-content/uploads/2012/11/C1060Timeline-1024x679.png\")
\n![\"\"](\"https://devblogs.nvidia.com/wp-content/uploads/2012/11/C2050Timeline-1024x670.png\")
https://miro.medium.com/max/700/0*Cqwuv9s1gI_rqHSw\" title=\"\" />
\n\nHere you are:\n - Using inplace=True when using activation functions like ReLu. This disables saving intermediate feature maps in your GPU.\n - Using eval() and with torch.no_grad(): when appropriate. eval() tells the model to switch off the calculation of batch_norm and the dropout. \n - DO NOT USE torch.cuda.emprty_cache() Although this will make the free memory used by torch visible by nvidia-smi, it does not actually reduce any memory. PyTorch did it all for you automatically. Some users even reported a small delay.\n - Careful: when the number of images in EACH GPU is smaller than 8, batch_norm can be unstable, unless you use sync_bn in APEX (described below)\n - Unlike some tutorial, torch.backends.cudnn.deterministic = True does not speed up calculation. It is conformed by multiple users that torch.backends.cudnn.deterministic = False is slower than torch.backends.cudnn.deterministic = True on MNIST dataset. It seems that the actual effect of this setting depends on the dataset. You can set it to True it if you like.\n - Do not take out the output mask when you do not need it for calculation. The flow of data between GPU and CPU cost time and CPU memory.
Described in the other thread.
\n\nSet your train_loader as following:\n - num_worker: the number of CPU threads when loading data. I usually set it to the amount of CPU I have on the machine. It is recommended that value should be greater than or equal to the amount of CPU on the machine.\n - pin_memory: whether to load data into RAM before loading it into GPU. If you are using a server, you can set it to True.\n - drop_last: whether to drop the last batch in an epoch if the last batch has fewer images than your batch_size. This can stabilize your training.
\ndata_loader = data.DataLoader(YOUR_PYTORCH_DATASET,\n num_workers=THE_NUMBER_OF_CPU_I_HAVE,\n pin_memory=True,\n drop_last=True, # Last batch will mess up with batch norm https://github.com/pytorch/pytorch/issues/4534\n ))\n\nIf your pin_memory = True, you can also set non_blocking=True to reduce the waiting time for GPUs. For more info: https://devblogs.nvidia.com/how-overlap-data-transfers-cuda-cc/
```\n\"\"\"Sync Point\"\"\"\nimage = image.cuda(non_blocking=True)\nlabels = labels.cuda(non_blocking=True).float()
\n\n\"\"\"Async Point\"\"\"\nprediction = net(image)\n```
\n\n\n
If you have multiple GPUs, you can use this method.\nVery simple: change torch.nn.parallel.DistributedDataParallel to apex.parallel.DistributedDataParallel
An example provided by APEX: https://github.com/NVIDIA/apex/tree/master/examples/imagenet
\n\nHere are some tricky parts:
\n\nCorrect Version\n\nnvcc --version version should match >>> torch.version.cuda version. Otherwise, it will cause errors.nvcc and pytorch.cuda 10.0 instead of 9.x\n\nGeForce GTX 1650\nGeForce GTX 1660\nGeForce GTX 1660 Ti\nGeForce RTX 2060\nGeForce RTX 2060 Super\nGeForce RTX 2070\nGeForce RTX 2070 Super\nGeForce RTX 2080\nGeForce RTX 2080 Super\nGeForce RTX 2080 Ti\nTitan RTX\nQuadro RTX 4000\nQuadro RTX 5000\nQuadro RTX 6000\nQuadro RTX 8000\nTesla T4\n\nOtherwise, use nvcc and pytorch.cuda 9.2.\nIf you need to re-install pytorch.cuda: HERE\nIf you need to re-install nvcc: NVCC9.2, NVCC10.0\nAnd then you have to re-install APEX all over again..\n.\n.\n(Oh my God. Kaggle has a bug that the whole article will disappear when resizing the browser, and I have to write the entire article all over again =. = And Ctrl + Z does not work either.)\n.\nHope it helps. Please upvote if you learned something here, thanks!\nFeel free to correct me or give us some extra suggestions :)
https://miro.medium.com/max/700/0*Cqwuv9s1gI_rqHSw\" title=\"\" />
\n\nHere you are:\n - Using inplace=True when using activation functions like ReLu. This disables saving intermediate feature maps in your GPU.\n - Using eval() and with torch.no_grad(): when appropriate. eval() tells the model to switch off the calculation of batch_norm and the dropout. \n - DO NOT USE torch.cuda.emprty_cache() Although this will make the free memory used by torch visible by nvidia-smi, it does not actually reduce any memory. PyTorch did it all for you automatically. Some users even reported a small delay.\n - Careful: when the number of images in EACH GPU is smaller than 8, batch_norm can be unstable, unless you use sync_bn in APEX (described below)\n - Unlike some tutorial, torch.backends.cudnn.deterministic = True does not speed up calculation. It is conformed by multiple users that torch.backends.cudnn.deterministic = False is slower than torch.backends.cudnn.deterministic = True on MNIST dataset. It seems that the actual effect of this setting depends on the dataset. You can set it to True it if you like.\n - Do not take out the output mask when you do not need it for calculation. The flow of data between GPU and CPU cost time and CPU memory.
Described in the other thread.
\n\nSet your train_loader as following:\n - num_worker: the number of CPU threads when loading data. I usually set it to the amount of CPU I have on the machine. It is recommended that value should be greater than or equal to the amount of CPU on the machine.\n - pin_memory: whether to load data into RAM before loading it into GPU. If you are using a server, you can set it to True.\n - drop_last: whether to drop the last batch in an epoch if the last batch has fewer images than your batch_size. This can stabilize your training.
\ndata_loader = data.DataLoader(YOUR_PYTORCH_DATASET,\n num_workers=THE_NUMBER_OF_CPU_I_HAVE,\n pin_memory=True,\n drop_last=True, # Last batch will mess up with batch norm https://github.com/pytorch/pytorch/issues/4534\n ))\n\nIf your pin_memory = True, you can also set non_blocking=True to reduce the waiting time for GPUs. For more info: https://devblogs.nvidia.com/how-overlap-data-transfers-cuda-cc/
```\n\"\"\"Sync Point\"\"\"\nimage = image.cuda(non_blocking=True)\nlabels = labels.cuda(non_blocking=True).float()
\n\n\"\"\"Async Point\"\"\"\nprediction = net(image)\n```
\n\n\n
If you have multiple GPUs, you can use this method.\nVery simple: change torch.nn.parallel.DistributedDataParallel to apex.parallel.DistributedDataParallel
An example provided by APEX: https://github.com/NVIDIA/apex/tree/master/examples/imagenet
\n\nHere are some tricky parts:
\n\nCorrect Version\n\nnvcc --version version should match >>> torch.version.cuda version. Otherwise, it will cause errors.nvcc and pytorch.cuda 10.0 instead of 9.x\n\nGeForce GTX 1650\nGeForce GTX 1660\nGeForce GTX 1660 Ti\nGeForce RTX 2060\nGeForce RTX 2060 Super\nGeForce RTX 2070\nGeForce RTX 2070 Super\nGeForce RTX 2080\nGeForce RTX 2080 Super\nGeForce RTX 2080 Ti\nTitan RTX\nQuadro RTX 4000\nQuadro RTX 5000\nQuadro RTX 6000\nQuadro RTX 8000\nTesla T4\n\nOtherwise, use nvcc and pytorch.cuda 9.2.\nIf you need to re-install pytorch.cuda: HERE\nIf you need to re-install nvcc: NVCC9.2, NVCC10.0\nAnd then you have to re-install APEX all over again..\n.\n.\n(Oh my God. Kaggle has a bug that the whole article will disappear when resizing the browser, and I have to write the entire article all over again =. = And Ctrl + Z does not work either.)\n.\nHope it helps. Please upvote if you learned something here, thanks!\nFeel free to correct me or give us some extra suggestions :)
Excellent references, thanks for posting it!!
", "rawMarkdown": "Excellent references, thanks for posting it!!" }, { "id": 913696, "postDate": "2020-07-03T11:07:26.313Z", "content": "When I set num_worker from 0 to 4, it would throw the OOM earlier.
Excellent references, thanks for posting it!!
", "votes": 0, "replies": [] }, { "id": 913696, "author_name": "", "author_url": "", "post_date": "2020-07-03T11:07:26.313000", "content": "When I set num_worker from 0 to 4, it would throw the OOM earlier.