Hey, I used to work with GPU and it gave me a decent speed, I was using it to train a model on the data with image size of 256. The time taken per epoch was 4:15 minutes with b0 while 8:45 with b4.
\n\nBut, now, when I am using the GPU with a image size of 384, it is taking about 10:30 minutes per epoch with b1.\nNothing other than the data was changed in my code, and I confirmed that the session metrics was showing 6.5 GB GPU usage of 15 GB.
\n\nWhat might be causing this?
\n\nI could only run two programs this week due to the amount of time taken by the programs. Please give me any pointers, this is really disheartening. :/
", "messages": [ { "id": 931841, "postDate": "2020-07-16T13:42:52.960Z", "content": "This notebook uses TPUs (not my work). For initial working on it, you might not want to run B0-B6. Cut it down to B0 until you get things working. Then experiment.
\n\nhttps://www.kaggle.com/agentauers/incredible-tpus-finetune-effnetb0-b6-at-once
\n\n-Rich
", "rawMarkdown": "This notebook uses TPUs (not my work). For initial working on it, you might not want to run B0-B6. Cut it down to B0 until you get things working. Then experiment.\n\n[https://www.kaggle.com/agentauers/incredible-tpus-finetune-effnetb0-b6-at-once](https://www.kaggle.com/agentauers/incredible-tpus-finetune-effnetb0-b6-at-once)\n\n-Rich\n\n", "votes": 1, "replies": [ { "id": 931900, "postDate": "2020-07-16T14:42:38.653Z", "content": "Hey, thanks!\nAlthough I saw it earlier, I didn't study it because it was in Keras, I know keras but I did all my work from the first day of this competition in pytorch, so will find one with pytorch! \nThanks anyway! :)
", "rawMarkdown": "Hey, thanks!\nAlthough I saw it earlier, I didn't study it because it was in Keras, I know keras but I did all my work from the first day of this competition in pytorch, so will find one with pytorch! \nThanks anyway! :) " } ] }, { "id": 931009, "postDate": "2020-07-15T22:28:41.293Z", "content": "It is really worth the time to switch to TPU and TFRecords. There are some great example notebooks. For example, I can train a B7/B0 hybrid with 600x600 pixel images in about 5 minutes per epoch. My guess is if I had more experience with TPUs, I could improve on that time.
", "rawMarkdown": "It is really worth the time to switch to TPU and TFRecords. There are some great example notebooks. For example, I can train a B7/B0 hybrid with 600x600 pixel images in about 5 minutes per epoch. My guess is if I had more experience with TPUs, I could improve on that time.", "votes": 1 }, { "id": 930729, "postDate": "2020-07-15T17:12:18.957Z", "content": "The increase of training time is expected when working with bigger images. If you want to run faster iterations you can keep running experiments with 256x256 images and then try the promising ones with bigger images. Or invest some time on setting up your notebook to be able to work with kaggle TPUs! They are going to speed your process a lot!
", "rawMarkdown": "The increase of training time is expected when working with bigger images. If you want to run faster iterations you can keep running experiments with 256x256 images and then try the promising ones with bigger images. Or invest some time on setting up your notebook to be able to work with kaggle TPUs! They are going to speed your process a lot!", "votes": 1, "replies": [ { "id": 930770, "postDate": "2020-07-15T17:50:14.500Z", "content": "Yeah, I did think about this but thought that it should not take this much amount of time I was expecting around 6-7 minutes as b1 is also a very lighter model than b4, maybe it is what it is, will try the other options as pointed out by you, thanks! :)
", "rawMarkdown": "Yeah, I did think about this but thought that it should not take this much amount of time I was expecting around 6-7 minutes as b1 is also a very lighter model than b4, maybe it is what it is, will try the other options as pointed out by you, thanks! :)", "votes": 1 } ] }, { "id": 930716, "postDate": "2020-07-15T17:02:37.563Z", "content": "Hey, I used to work with GPU and it gave me a decent speed, I was using it to train a model on the data with image size of 256. The time taken per epoch was 4:15 minutes with b0 while 8:45 with b4.
\n\nBut, now, when I am using the GPU with a image size of 384, it is taking about 10:30 minutes per epoch with b1.\nNothing other than the data was changed in my code, and I confirmed that the session metrics was showing 6.5 GB GPU usage of 15 GB.
\n\nWhat might be causing this?
\n\nI could only run two programs this week due to the amount of time taken by the programs. Please give me any pointers, this is really disheartening. :/
", "rawMarkdown": "Hey, I used to work with GPU and it gave me a decent speed, I was using it to train a model on the data with image size of 256. The time taken per epoch was 4:15 minutes with b0 while 8:45 with b4.\n\nBut, now, when I am using the GPU with a image size of 384, it is taking about 10:30 minutes per epoch with b1.\nNothing other than the data was changed in my code, and I confirmed that the session metrics was showing 6.5 GB GPU usage of 15 GB. \n\nWhat might be causing this? \n\nI could only run two programs this week due to the amount of time taken by the programs. Please give me any pointers, this is really disheartening. :/", "votes": 1 }, { "id": 930835, "postDate": "2020-07-15T18:35:48.750Z", "content": "Usually image loading and augmentations is the bottleneck. In you case, the images are 2.25 times bigger, more or less the size of your performance degradation.
I don't know about tensorflow, but in case of PyTorch it is false as in Abhishek's notebook, he used png image with tpu. Now, since I also want to do the same, I have high hopes to find some info on that. :)
As per TensorFlow, I believe we cant work with jpg images using TPU. AS of now, TPU supports only TFRecords.
", "rawMarkdown": "As per TensorFlow, I believe we cant work with jpg images using TPU. AS of now, TPU supports only TFRecords. " }, { "id": 931769, "postDate": "2020-07-16T12:36:50.503Z", "content": "Hey, can anyone share any tut/notebook or something on how to work with tpu and jpeg images? I found many notebooks but could not understand how to implement this, I want to start using tpu with jpg images as of now.
\n\nI saw Abhishek's Power hour session notebook but seems like it has some glitch in it. :/
", "rawMarkdown": "Hey, can anyone share any tut/notebook or something on how to work with tpu and jpeg images? I found many notebooks but could not understand how to implement this, I want to start using tpu with jpg images as of now.\n\nI saw Abhishek's Power hour session notebook but seems like it has some glitch in it. :/ " }, { "id": 931137, "postDate": "2020-07-16T02:34:44.523Z", "content": "Sure, thanks everyone, will follow the lead given by you all, I did not invest my time on TPU thinking it will take a lot of time just to learn but seems it is worthy of doing so, thanks again! :)
", "rawMarkdown": "Sure, thanks everyone, will follow the lead given by you all, I did not invest my time on TPU thinking it will take a lot of time just to learn but seems it is worthy of doing so, thanks again! :)" }, { "id": 933162, "postDate": "2020-07-17T14:31:16.397Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 931841, "author_name": "quadcore/Richard Epstein", "author_url": "", "post_date": "2020-07-16T13:42:52.960000", "content": "This notebook uses TPUs (not my work). For initial working on it, you might not want to run B0-B6. Cut it down to B0 until you get things working. Then experiment.
\n\nhttps://www.kaggle.com/agentauers/incredible-tpus-finetune-effnetb0-b6-at-once
\n\n-Rich
", "votes": 1, "replies": [ { "id": 931900, "author_name": "Gajendra Saraswat", "author_url": "", "post_date": "2020-07-16T14:42:38.653000", "content": "Hey, thanks!\nAlthough I saw it earlier, I didn't study it because it was in Keras, I know keras but I did all my work from the first day of this competition in pytorch, so will find one with pytorch! \nThanks anyway! :)
", "votes": 0, "replies": [] } ] }, { "id": 931009, "author_name": "quadcore/Richard Epstein", "author_url": "", "post_date": "2020-07-15T22:28:41.293000", "content": "It is really worth the time to switch to TPU and TFRecords. There are some great example notebooks. For example, I can train a B7/B0 hybrid with 600x600 pixel images in about 5 minutes per epoch. My guess is if I had more experience with TPUs, I could improve on that time.
", "votes": 1, "replies": [] }, { "id": 930729, "author_name": "Santiago Viquez", "author_url": "", "post_date": "2020-07-15T17:12:18.957000", "content": "The increase of training time is expected when working with bigger images. If you want to run faster iterations you can keep running experiments with 256x256 images and then try the promising ones with bigger images. Or invest some time on setting up your notebook to be able to work with kaggle TPUs! They are going to speed your process a lot!
", "votes": 1, "replies": [ { "id": 930770, "author_name": "Gajendra Saraswat", "author_url": "", "post_date": "2020-07-15T17:50:14.500000", "content": "Yeah, I did think about this but thought that it should not take this much amount of time I was expecting around 6-7 minutes as b1 is also a very lighter model than b4, maybe it is what it is, will try the other options as pointed out by you, thanks! :)
", "votes": 1, "replies": [] } ] }, { "id": 930835, "author_name": "nosound", "author_url": "", "post_date": "2020-07-15T18:35:48.750000", "content": "Usually image loading and augmentations is the bottleneck. In you case, the images are 2.25 times bigger, more or less the size of your performance degradation.
I don't know about tensorflow, but in case of PyTorch it is false as in Abhishek's notebook, he used png image with tpu. Now, since I also want to do the same, I have high hopes to find some info on that. :)
As per TensorFlow, I believe we cant work with jpg images using TPU. AS of now, TPU supports only TFRecords.
", "votes": 0, "replies": [] }, { "id": 931769, "author_name": "Gajendra Saraswat", "author_url": "", "post_date": "2020-07-16T12:36:50.503000", "content": "Hey, can anyone share any tut/notebook or something on how to work with tpu and jpeg images? I found many notebooks but could not understand how to implement this, I want to start using tpu with jpg images as of now.
\n\nI saw Abhishek's Power hour session notebook but seems like it has some glitch in it. :/
", "votes": 0, "replies": [] }, { "id": 931137, "author_name": "Gajendra Saraswat", "author_url": "", "post_date": "2020-07-16T02:34:44.523000", "content": "Sure, thanks everyone, will follow the lead given by you all, I did not invest my time on TPU thinking it will take a lot of time just to learn but seems it is worthy of doing so, thanks again! :)
", "votes": 0, "replies": [] }, { "id": 933162, "author_name": "", "author_url": "", "post_date": "2020-07-17T14:31:16.397000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "931841": "This notebook uses TPUs (not my work). For initial working on it, you might not want to run B0-B6. Cut it down to B0 until you get things working. Then experiment.\n\n[https://www.kaggle.com/agentauers/incredible-tpus-finetune-effnetb0-b6-at-once](https://www.kaggle.com/agentauers/incredible-tpus-finetune-effnetb0-b6-at-once)\n\n-Rich\n\n", "931009": "It is really worth the time to switch to TPU and TFRecords. There are some great example notebooks. For example, I can train a B7/B0 hybrid with 600x600 pixel images in about 5 minutes per epoch. My guess is if I had more experience with TPUs, I could improve on that time.", "930729": "The increase of training time is expected when working with bigger images. If you want to run faster iterations you can keep running experiments with 256x256 images and then try the promising ones with bigger images. Or invest some time on setting up your notebook to be able to work with kaggle TPUs! They are going to speed your process a lot!", "930716": "Hey, I used to work with GPU and it gave me a decent speed, I was using it to train a model on the data with image size of 256. The time taken per epoch was 4:15 minutes with b0 while 8:45 with b4.\n\nBut, now, when I am using the GPU with a image size of 384, it is taking about 10:30 minutes per epoch with b1.\nNothing other than the data was changed in my code, and I confirmed that the session metrics was showing 6.5 GB GPU usage of 15 GB. \n\nWhat might be causing this? \n\nI could only run two programs this week due to the amount of time taken by the programs. Please give me any pointers, this is really disheartening. :/", "930835": "Usually image loading and augmentations is the bottleneck. In you case, the images are `2.25` times bigger, more or less the size of your performance degradation.", "933163": "I don't know about tensorflow, but in case of PyTorch it is false as in Abhishek's notebook, he used `png` image with tpu. Now, since I also want to do the same, I have high hopes to find some info on that. :)\n", "933007": "As per TensorFlow, I believe we cant work with jpg images using TPU. AS of now, TPU supports only TFRecords. ", "931769": "Hey, can anyone share any tut/notebook or something on how to work with tpu and jpeg images? I found many notebooks but could not understand how to implement this, I want to start using tpu with jpg images as of now.\n\nI saw Abhishek's Power hour session notebook but seems like it has some glitch in it. :/ ", "931137": "Sure, thanks everyone, will follow the lead given by you all, I did not invest my time on TPU thinking it will take a lot of time just to learn but seems it is worthy of doing so, thanks again! :)", "933162": "" } }