![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1885266%2F73f2dfbe8e0fba77d48477de7d67e3b5%2FScreen%20Shot%202020-06-13%20at%203.48.35%20PM.png?generation=1592078086023616&alt=media\")
So, I was using TensorFlow 2.0, I had enabled the GPU but still, it was not utilizing the GPU. I tried to restart the kernel and start the training again but it is still the same.\nMy, model:\nresnet101,\nnormal imageDataGenerator() and flow_from_dataframe,\nI am using \"focal loss\", as the dataset is imbalanced,\nI also used the weight class approach as the dataset is imbalanced,\nbatch size=32,\nimage size 521,\nmetrics = AUC,\noptimizer = adam\nthis is my notebook link - https://www.kaggle.com/soumya9977/kernel61b65f025f-siim?scriptVersionId=36233725
So, I was using TensorFlow 2.0, I had enabled the GPU but still, it was not utilizing the GPU. I tried to restart the kernel and start the training again but it is still the same.\nMy, model:\nresnet101,\nnormal imageDataGenerator() and flow_from_dataframe,\nI am using \"focal loss\", as the dataset is imbalanced,\nI also used the weight class approach as the dataset is imbalanced,\nbatch size=32,\nimage size 521,\nmetrics = AUC,\noptimizer = adam\nthis is my notebook link - https://www.kaggle.com/soumya9977/kernel61b65f025f-siim?scriptVersionId=36233725
@cdeotte sir can you please 🙏🙏🙏 look into this matter, I really need help. while training a model this thing never happened to me.
", "rawMarkdown": "cdeotte sir can you please 🙏🙏🙏 look into this matter, I really need help. while training a model this thing never happened to me.", "votes": null }, { "id": "885020", "postDate": "06/13/2020 19:55:58", "content": "Hi :)
\n\nI am not an expert nor do I play one on TV but...
\n\nThe notebook you have linked to doesn't have the GPU or the Internet connected so not much will happen anyway. Assuming that you know that, I enabled them and ran your code. I think it is pretty I/O bound reading in the jpeg files and the GPU is working but it doesn't have much to do while waiting for the CPU (which is maxed out).
\n\n
Small but significant typo.\n\ntrain_data_generator = tf.keras.preprocessing.image.ImageDataGenerator(\n rescale=1./255,\n #rescale=1.255, #you had this line\n
@mutantspore sir, thank you for the reply and also thank you for finding that typo.🙇♂️\nI also just found that it is using GPU but sometimes not every time and for a few seconds
\n\n![](\"https://i.imgur.com/Htnq2YM.png\")
As you said, the GPU is working but it doesn't have much to do while waiting for the CPU (which is maxed out)., so how to use the only GPU? Is there any specific way, like installing the GPU version of TensorFlow (like this pip install tensorflow-gpu). \nOr, the scenario is like that-> it is not possible to train that model using GPU(as it is so much time consuming ) and I should try to use TPU.\nAny solution to this problem? how should I approach this? \nActually it is my 1st featured competition so I am lacking in experience.
\n\n\nAs you said, `the GPU is working but it doesn't have much to do while waiting for the CPU (which is maxed out).`, so how to use the only GPU? Is there any specific way, like installing the GPU version of TensorFlow (like this `pip install tensorflow-gpu`). \nOr, the scenario is like that-> it is not possible to train that model using GPU(as it is so much time consuming ) and I should try to use TPU.\nAny solution to this problem? how should I approach this? \nActually it is my 1st featured competition so I am lacking in experience.",
"votes": null
},
{
"id": "885099",
"postDate": "06/13/2020 21:48:48",
"content": "The preprocessing step to resize the image data can be time consuming and is on the CPU. Most Notebooks used pre-processed resized data, so you don't have to resize it for each epoch. There are multiple posts with links to pre-processed image data at various sizes.
\n\nAlso, I find calculating AUC to be time consuming, and I often remove it as a metric. Trade off between speed and information.
", "rawMarkdown": "The preprocessing step to resize the image data can be time consuming and is on the CPU. Most Notebooks used pre-processed resized data, so you don't have to resize it for each epoch. There are multiple posts with links to pre-processed image data at various sizes.\n\nAlso, I find calculating AUC to be time consuming, and I often remove it as a metric. Trade off between speed and information.", "votes": null }, { "id": "885121", "postDate": "06/13/2020 22:50:16", "content": "Ok, sir, I will try that, but is there any way by which I can utilize that GPU by its full potential.
", "rawMarkdown": "Ok, sir, I will try that, but is there any way by which I can utilize that GPU by its full potential.", "votes": null }, { "id": "885134", "postDate": "06/13/2020 23:49:18", "content": "Hi
\n\nAs has already been pointed out you are trying to read in and resize to 512, very large images that are sometimes over 3000 pixels across. This is killing your CPU as it's the CPU that does that aspect. As already mentioned there are datasets graciously provided by others that have been resized to a useful manageable size that you can use. I have included one by Arnaud Roussel in my mod of your notebook. SIIM ISIC - 224x224 images
\n\nWhen you are developing a notebook for the first time it's best to keep it simple, get it working and then try out ideas. So with that in mind, I cut your model back to the basic parts to make it work. The GPU is now used far more often, even maxing out sometimes. I also commented out bits that had bugs or were incomplete as you can now work through it and sort them out. There are higher resolution preprocessed images available when you get this all working properly.
\n\nAt the moment each epoch is taking about 500 seconds which is a little bit better than the 3 1/2 hours it was before.\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1885266%2Fb9a7f38fdf4ef62001957425da3f616b%2FScreen%20Shot%202020-06-13%20at%207.41.59%20PM.png?generation=1592091771221044&alt=media\")
Using the GPU is slower than using a TPU but there are aspects of this problem that seem to be easier or can currently only be done using a GPU.
\n\nI made my version of your notebook public so you can see it.\nhttps://www.kaggle.com/mutantspore/kernel61b65f025f-siim-start
", "rawMarkdown": "Hi\n\nAs has already been pointed out you are trying to read in and resize to 512, very large images that are sometimes over 3000 pixels across. This is killing your CPU as it's the CPU that does that aspect. As already mentioned there are datasets graciously provided by others that have been resized to a useful manageable size that you can use. I have included one by Arnaud Roussel in my mod of your notebook. [SIIM ISIC - 224x224 images](https://www.kaggle.com/arroqc/siic-isic-224x224-images)\n\nWhen you are developing a notebook for the first time it's best to keep it simple, get it working and then try out ideas. So with that in mind, I cut your model back to the basic parts to make it work. The GPU is now used far more often, even maxing out sometimes. I also commented out bits that had bugs or were incomplete as you can now work through it and sort them out. There are higher resolution preprocessed images available when you get this all working properly. \n\nAt the moment each epoch is taking about 500 seconds which is a little bit better than the 3 1/2 hours it was before.\n\n\nUsing the GPU is slower than using a TPU but there are aspects of this problem that seem to be easier or can currently only be done using a GPU. \n\nI made my version of your notebook public so you can see it.\n[https://www.kaggle.com/mutantspore/kernel61b65f025f-siim-start](https://www.kaggle.com/mutantspore/kernel61b65f025f-siim-start)", "votes": null }, { "id": "885824", "postDate": "06/14/2020 14:04:40", "content": "Sorry, for the late response.\nThank you sir @mutantspore for helping me out. I can't thank you more for what you did for me 🙇♂️🙇♂️🙇♂️. thank you for taking some time from daily busy life and investing that to help a newbie. Now it is running.
\n\nI am lacking in experience and because of that I am facing problems, but, I will try my best to overcome this and I will be more active In the competition section.\nLooking forward to learning from you.
", "rawMarkdown": "Sorry, for the late response.\nThank you sir @mutantspore for helping me out. I can't thank you more for what you did for me 🙇♂️🙇♂️🙇♂️. thank you for taking some time from daily busy life and investing that to help a newbie. Now it is running.\n\nI am lacking in experience and because of that I am facing problems, but, I will try my best to overcome this and I will be more active In the competition section.\nLooking forward to learning from you.", "votes": null }, { "id": "885832", "postDate": "06/14/2020 14:09:59", "content": "you were right @richardepstein, sir.\nThank you for pointing that out. Now I know how to deal with these issues 🙏🙏🙏
", "rawMarkdown": "you were right @richardepstein, sir.\nThank you for pointing that out. Now I know how to deal with these issues 🙏🙏🙏", "votes": null }, { "id": "886829", "postDate": "06/15/2020 09:53:35", "content": "the same issue with me too @soumya9977. Thank you all for the solution!
", "rawMarkdown": "the same issue with me too @soumya9977. Thank you all for the solution!", "votes": null }, { "id": "887844", "postDate": "06/15/2020 23:27:13", "content": "The problem was, resizing an image takes a good amount of computation, because of that the training process was taking time. So, Mr.@richardepstein pointed that out and Mr.@mutantspore pointed out and provided the solution by creating another notebook.\nAll thanks to them.
", "rawMarkdown": "The problem was, resizing an image takes a good amount of computation, because of that the training process was taking time. So, Mr.@richardepstein pointed that out and Mr.@mutantspore pointed out and provided the solution by creating another notebook.\nAll thanks to them.", "votes": null } ], "comments": [ { "id": 884977, "author_name": "soumya9977", "author_url": "", "post_date": "06/13/2020 19:17:57", "content": "@cdeotte sir can you please 🙏🙏🙏 look into this matter, I really need help. while training a model this thing never happened to me.
", "votes": null, "replies": [ { "id": 885099, "author_name": "richardepstein", "author_url": "", "post_date": "06/13/2020 21:48:48", "content": "The preprocessing step to resize the image data can be time consuming and is on the CPU. Most Notebooks used pre-processed resized data, so you don't have to resize it for each epoch. There are multiple posts with links to pre-processed image data at various sizes.
\n\nAlso, I find calculating AUC to be time consuming, and I often remove it as a metric. Trade off between speed and information.
", "votes": null, "replies": [] }, { "id": 885121, "author_name": "soumya9977", "author_url": "", "post_date": "06/13/2020 22:50:16", "content": "Ok, sir, I will try that, but is there any way by which I can utilize that GPU by its full potential.
", "votes": null, "replies": [] }, { "id": 885832, "author_name": "soumya9977", "author_url": "", "post_date": "06/14/2020 14:09:59", "content": "you were right @richardepstein, sir.\nThank you for pointing that out. Now I know how to deal with these issues 🙏🙏🙏
", "votes": null, "replies": [] } ] }, { "id": 885020, "author_name": "mutantspore", "author_url": "", "post_date": "06/13/2020 19:55:58", "content": "Hi :)
\n\nI am not an expert nor do I play one on TV but...
\n\nThe notebook you have linked to doesn't have the GPU or the Internet connected so not much will happen anyway. Assuming that you know that, I enabled them and ran your code. I think it is pretty I/O bound reading in the jpeg files and the GPU is working but it doesn't have much to do while waiting for the CPU (which is maxed out).
\n\n
Small but significant typo.\n\ntrain_data_generator = tf.keras.preprocessing.image.ImageDataGenerator(\n rescale=1./255,\n #rescale=1.255, #you had this line\n
@mutantspore sir, thank you for the reply and also thank you for finding that typo.🙇♂️\nI also just found that it is using GPU but sometimes not every time and for a few seconds
\n\n
As you said, the GPU is working but it doesn't have much to do while waiting for the CPU (which is maxed out)., so how to use the only GPU? Is there any specific way, like installing the GPU version of TensorFlow (like this pip install tensorflow-gpu). \nOr, the scenario is like that-> it is not possible to train that model using GPU(as it is so much time consuming ) and I should try to use TPU.\nAny solution to this problem? how should I approach this? \nActually it is my 1st featured competition so I am lacking in experience.
Hi
\n\nAs has already been pointed out you are trying to read in and resize to 512, very large images that are sometimes over 3000 pixels across. This is killing your CPU as it's the CPU that does that aspect. As already mentioned there are datasets graciously provided by others that have been resized to a useful manageable size that you can use. I have included one by Arnaud Roussel in my mod of your notebook. SIIM ISIC - 224x224 images
\n\nWhen you are developing a notebook for the first time it's best to keep it simple, get it working and then try out ideas. So with that in mind, I cut your model back to the basic parts to make it work. The GPU is now used far more often, even maxing out sometimes. I also commented out bits that had bugs or were incomplete as you can now work through it and sort them out. There are higher resolution preprocessed images available when you get this all working properly.
\n\nAt the moment each epoch is taking about 500 seconds which is a little bit better than the 3 1/2 hours it was before.\n
Using the GPU is slower than using a TPU but there are aspects of this problem that seem to be easier or can currently only be done using a GPU.
\n\nI made my version of your notebook public so you can see it.\nhttps://www.kaggle.com/mutantspore/kernel61b65f025f-siim-start
", "votes": null, "replies": [ { "id": 885824, "author_name": "soumya9977", "author_url": "", "post_date": "06/14/2020 14:04:40", "content": "Sorry, for the late response.\nThank you sir @mutantspore for helping me out. I can't thank you more for what you did for me 🙇♂️🙇♂️🙇♂️. thank you for taking some time from daily busy life and investing that to help a newbie. Now it is running.
\n\nI am lacking in experience and because of that I am facing problems, but, I will try my best to overcome this and I will be more active In the competition section.\nLooking forward to learning from you.
", "votes": null, "replies": [] } ] }, { "id": 886829, "author_name": "abhiswain", "author_url": "", "post_date": "06/15/2020 09:53:35", "content": "the same issue with me too @soumya9977. Thank you all for the solution!
", "votes": null, "replies": [ { "id": 887844, "author_name": "soumya9977", "author_url": "", "post_date": "06/15/2020 23:27:13", "content": "The problem was, resizing an image takes a good amount of computation, because of that the training process was taking time. So, Mr.@richardepstein pointed that out and Mr.@mutantspore pointed out and provided the solution by creating another notebook.\nAll thanks to them.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "884971": "So, I was using `TensorFlow 2.0`, I had `enabled the GPU` but still, it was not utilizing the GPU. I tried to `restart the kernel` and start the training again but it is still the same.\nMy, model:\n`resnet101`,\nnormal `imageDataGenerator()` and flow_from_dataframe,\nI am using \"focal loss\", as the dataset is imbalanced,\nI also used the `weight class approach` as the dataset is imbalanced,\nbatch size=32,\nimage size 521,\nmetrics = AUC,\noptimizer = adam\nthis is my notebook link - https://www.kaggle.com/soumya9977/kernel61b65f025f-siim?scriptVersionId=36233725", "884977": "cdeotte sir can you please 🙏🙏🙏 look into this matter, I really need help. while training a model this thing never happened to me.", "885020": "Hi :)\n\nI am not an expert nor do I play one on TV but...\n\nThe notebook you have linked to doesn't have the GPU or the Internet connected so not much will happen anyway. Assuming that you know that, I enabled them and ran your code. I think it is pretty I/O bound reading in the jpeg files and the GPU is working but it doesn't have much to do while waiting for the CPU (which is maxed out).\n\n\n\n\nUPDATE\n---------\nSmall but significant typo.\n```\ntrain_data_generator = tf.keras.preprocessing.image.ImageDataGenerator(\n rescale=1./255,\n #rescale=1.255, #you had this line\n```", "885086": "mutantspore sir, thank you for the reply and also thank you for finding that typo.🙇♂️\nI also just found that it is using GPU but sometimes not every time and for a few seconds\n\n\n\n\n\nAs you said, `the GPU is working but it doesn't have much to do while waiting for the CPU (which is maxed out).`, so how to use the only GPU? Is there any specific way, like installing the GPU version of TensorFlow (like this `pip install tensorflow-gpu`). \nOr, the scenario is like that-> it is not possible to train that model using GPU(as it is so much time consuming ) and I should try to use TPU.\nAny solution to this problem? how should I approach this? \nActually it is my 1st featured competition so I am lacking in experience.",
"885099": "The preprocessing step to resize the image data can be time consuming and is on the CPU. Most Notebooks used pre-processed resized data, so you don't have to resize it for each epoch. There are multiple posts with links to pre-processed image data at various sizes.\n\nAlso, I find calculating AUC to be time consuming, and I often remove it as a metric. Trade off between speed and information.",
"885121": "Ok, sir, I will try that, but is there any way by which I can utilize that GPU by its full potential.",
"885134": "Hi\n\nAs has already been pointed out you are trying to read in and resize to 512, very large images that are sometimes over 3000 pixels across. This is killing your CPU as it's the CPU that does that aspect. As already mentioned there are datasets graciously provided by others that have been resized to a useful manageable size that you can use. I have included one by Arnaud Roussel in my mod of your notebook. [SIIM ISIC - 224x224 images](https://www.kaggle.com/arroqc/siic-isic-224x224-images)\n\nWhen you are developing a notebook for the first time it's best to keep it simple, get it working and then try out ideas. So with that in mind, I cut your model back to the basic parts to make it work. The GPU is now used far more often, even maxing out sometimes. I also commented out bits that had bugs or were incomplete as you can now work through it and sort them out. There are higher resolution preprocessed images available when you get this all working properly. \n\nAt the moment each epoch is taking about 500 seconds which is a little bit better than the 3 1/2 hours it was before.\n\n\nUsing the GPU is slower than using a TPU but there are aspects of this problem that seem to be easier or can currently only be done using a GPU. \n\nI made my version of your notebook public so you can see it.\n[https://www.kaggle.com/mutantspore/kernel61b65f025f-siim-start](https://www.kaggle.com/mutantspore/kernel61b65f025f-siim-start)",
"885824": "Sorry, for the late response.\nThank you sir @mutantspore for helping me out. I can't thank you more for what you did for me 🙇♂️🙇♂️🙇♂️. thank you for taking some time from daily busy life and investing that to help a newbie. Now it is running.\n\nI am lacking in experience and because of that I am facing problems, but, I will try my best to overcome this and I will be more active In the competition section.\nLooking forward to learning from you.",
"885832": "you were right @richardepstein, sir.\nThank you for pointing that out. Now I know how to deal with these issues 🙏🙏🙏",
"886829": "the same issue with me too @soumya9977. Thank you all for the solution!",
"887844": "The problem was, resizing an image takes a good amount of computation, because of that the training process was taking time. So, Mr.@richardepstein pointed that out and Mr.@mutantspore pointed out and provided the solution by creating another notebook.\nAll thanks to them."
},
"source": "meta"
}