![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3307250%2F69f3b3bbf34a77b45999363a93749737%2Fgpu%20error.png?generation=1591363505081184&alt=media\")
I tried to connect to gpu with this code below and also by changing the accelarator\nprint('Setting GPU')\n K.clear_session()\n config = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=1,inter_op_parallelism_threads=1)\n graph = tf.compat.v1.get_default_graph()\n sess = tf.compat.v1.Session(graph=graph,config=config)\n tf.compat.v1.keras.backend.set_session(sess)
And i simply used the Keras ResNet50 model to train the jpeg images\nBut by looking at the below image. Im still confused why even though i connnected to gpu\nmy training is slower.Could someone help me through this\n
EDIT : And this slow training is happening in google colab also with Tesla P-100 GPU
", "messages": [ { "id": 875031, "postDate": "2020-06-05T13:25:30.190Z", "content": "I tried to connect to gpu with this code below and also by changing the accelarator\nprint('Setting GPU')\n K.clear_session()\n config = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=1,inter_op_parallelism_threads=1)\n graph = tf.compat.v1.get_default_graph()\n sess = tf.compat.v1.Session(graph=graph,config=config)\n tf.compat.v1.keras.backend.set_session(sess)
And i simply used the Keras ResNet50 model to train the jpeg images\nBut by looking at the below image. Im still confused why even though i connnected to gpu\nmy training is slower.Could someone help me through this\n
EDIT : And this slow training is happening in google colab also with Tesla P-100 GPU
", "rawMarkdown": "I tried to connect to gpu with this code below and also by changing the accelarator\n`print('Setting GPU')`\n `K.clear_session()`\n `config = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=1,inter_op_parallelism_threads=1)`\n `graph = tf.compat.v1.get_default_graph()`\n `sess = tf.compat.v1.Session(graph=graph,config=config)`\n `tf.compat.v1.keras.backend.set_session(sess)`\n\nAnd i simply used the Keras ResNet50 model to train the jpeg images\nBut by looking at the below image. Im still confused why even though i connnected to gpu\nmy training is slower.Could someone help me through this\n\n\nEDIT : And this slow training is happening in google colab also with Tesla P-100 GPU", "votes": 2 }, { "id": 876387, "postDate": "2020-06-06T16:46:55.003Z", "content": "@cdeotte any updates on my error chris???
", "rawMarkdown": "@cdeotte any updates on my error chris???" }, { "id": 875155, "postDate": "2020-06-05T14:54:53.660Z", "content": "Fred posted some code here. His training is also slow using tensorflow imagedatagenerator class. When i get time, I'll play with his code and see what is slowing it down. I think if we can speed up his code, it would speed up your code.
I will share my simple custom datagenerator code here. Please check with this generator also\nI have run out of my kaggle and colab GPU sessions
\n\nclass DataGenerator(Sequence):\n def __init__(self,df,image_path,batch_size,dim,n_channels,to_fit):\n self.df = df\n self.image_path = image_path\n self.batch = batch_size\n self.dim = dim\n self.n_channels = n_channels\n self.to_fit = to_fit
`def __len__(self):\n return int(np.floor(self.df.shape[0])/self.batch)\n\ndef __getitem__(self, index):\n list_IDs = self.df['image_name'].values[index*self.batch : (index+1)*self.batch]\n X = self._generate_X(list_IDs)\n if self.to_fit:\n target_y = self._generate_y(list_IDs)\n return np.array(X), np.array(target_y)\n return np.array(X)\n\ndef _generate_X(self,list_IDs):\n X = Parallel(n_jobs=self.batch)(delayed(self._load_image)(i) for i in list_IDs)\n return X\n\ndef _generate_y(self,list_IDs):\n target_y = []\n for i, ids in enumerate(list_IDs):\n target_y.append(self.df[self.df['image_name'] == ids]['target'].values[0])\n return target_y\n\ndef _load_image(self,file_):\n img = cv2.imread(os.path.join(self.image_path,file_)+'.jpg')\n img = cv2.cvtColor(img,cv2.COLOR_BGR2RGB)\n return img`\n `\nI see your data generator resizes images. What is original size on Disk and what size do you resize to?
", "rawMarkdown": "I see your data generator resizes images. What is original size on Disk and what size do you resize to?" }, { "id": 875177, "postDate": "2020-06-05T15:05:20.067Z", "content": "Sorry i forgoted to comment that.But im really not using\nIm just reading and changing the channels
", "rawMarkdown": "Sorry i forgoted to comment that.But im really not using\nIm just reading and changing the channels", "votes": 1 } ] }, { "id": 875128, "postDate": "2020-06-05T14:42:46.657Z", "content": "Are you reading in 1024x1024 sized images and then resizing to 224x224? That is what Fred is doing here and that is slow. It is better to resize all the images and save them to smaller files on disk. Then load the smaller files and don't spend time resizing them during training.
", "rawMarkdown": "Are you reading in 1024x1024 sized images and then resizing to 224x224? That is what Fred is doing [here][1] and that is slow. It is better to resize all the images and save them to smaller files on disk. Then load the smaller files and don't spend time resizing them during training.\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/156303", "replies": [ { "id": 875144, "postDate": "2020-06-05T14:49:28.533Z", "content": "No, I saved the 300x300 jpeg files by resizing to 224x224 by some preprocessing.\nI am just reading the files from the the 224x224 folder
\n\nUPDATE : Now im doing the prediction and it took 2-1/2 hours and still continuing
", "rawMarkdown": "No, I saved the 300x300 jpeg files by resizing to 224x224 by some preprocessing.\nI am just reading the files from the the 224x224 folder\n\nUPDATE : Now im doing the prediction and it took 2-1/2 hours and still continuing" } ] }, { "id": 875084, "postDate": "2020-06-05T14:07:31.717Z", "content": "What size images are you using? I suggest doing experiments with smaller images like 256x256, 192x192, or 128x128 if you're only using one GPU P100. With more or faster GPUs, you can use larger images.
", "rawMarkdown": "What size images are you using? I suggest doing experiments with smaller images like 256x256, 192x192, or 128x128 if you're only using one GPU P100. With more or faster GPUs, you can use larger images.", "replies": [ { "id": 875086, "postDate": "2020-06-05T14:09:12.897Z", "content": "I was just using 224x224 images with a hair removal technique\nThe gpus i only have is from kaggle and colab only(Not Colab Pro!!)
\n\nThe above code is right way to connect to GPUs right??\nor did i miss something??
", "rawMarkdown": "I was just using 224x224 images with a hair removal technique\nThe gpus i only have is from kaggle and colab only(Not Colab Pro!!)\n\nThe above code is right way to connect to GPUs right??\nor did i miss something??" }, { "id": 875099, "postDate": "2020-06-05T14:14:31.563Z", "content": "Ok, something is wrong then. Maybe you are running on CPU? Or perhaps your dataloader is slow. Also i'm not familiar if these settings affect GPU
\n\ntf.compat.v1.ConfigProto(intra_op_parallelism_threads=1,inter_op_parallelism_threads=1)\ngraph = tf.compat.v1.get_default_graph()\nsess = tf.compat.v1.Session(graph=graph,config=config)\ntf.compat.v1.keras.backend.set_session(sess)\nOnly having 1 intra and 1 inter 1 thread sounds slow. Can you run your code without these configurations?
", "rawMarkdown": "Ok, something is wrong then. Maybe you are running on CPU? Or perhaps your dataloader is slow. Also i'm not familiar if these settings affect GPU\n\n tf.compat.v1.ConfigProto(intra_op_parallelism_threads=1,inter_op_parallelism_threads=1)\n graph = tf.compat.v1.get_default_graph()\n sess = tf.compat.v1.Session(graph=graph,config=config)\n tf.compat.v1.keras.backend.set_session(sess)\n\nOnly having 1 intra and 1 inter 1 thread sounds slow. Can you run your code without these configurations?" }, { "id": 875100, "postDate": "2020-06-05T14:15:30.790Z", "content": "These codes i picked up from the Ion-Switching competition where they used wavenet model to run on gpus..remember??
", "rawMarkdown": "These codes i picked up from the Ion-Switching competition where they used wavenet model to run on gpus..remember??" }, { "id": 875104, "postDate": "2020-06-05T14:20:04.103Z", "content": "Im using the tensorflow imagedatagenerator class
", "rawMarkdown": "Im using the tensorflow imagedatagenerator class " }, { "id": 875106, "postDate": "2020-06-05T14:20:41.590Z", "content": "Yes, you are right. They were use in Ion Comp and didn't cause problems. So they're probably not the problem. (But I'm not sure how they affect GPU and I never use those lines).
", "rawMarkdown": "Yes, you are right. They were use in Ion Comp and didn't cause problems. So they're probably not the problem. (But I'm not sure how they affect GPU and I never use those lines)." }, { "id": 875107, "postDate": "2020-06-05T14:21:56.947Z", "rawMarkdown": "", "isDeleted": true }, { "id": 875108, "postDate": "2020-06-05T14:22:41.997Z", "content": "Hey i could share my entire notebook to you.Could you help me to check those errors.
", "rawMarkdown": "Hey i could share my entire notebook to you.Could you help me to check those errors." } ] } ], "comments": [ { "id": 876387, "author_name": "MhdSharuk", "author_url": "", "post_date": "2020-06-06T16:46:55.003000", "content": "@cdeotte any updates on my error chris???
", "votes": 0, "replies": [] }, { "id": 875155, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-06-05T14:54:53.660000", "content": "Fred posted some code here. His training is also slow using tensorflow imagedatagenerator class. When i get time, I'll play with his code and see what is slowing it down. I think if we can speed up his code, it would speed up your code.
I will share my simple custom datagenerator code here. Please check with this generator also\nI have run out of my kaggle and colab GPU sessions
\n\nclass DataGenerator(Sequence):\n def __init__(self,df,image_path,batch_size,dim,n_channels,to_fit):\n self.df = df\n self.image_path = image_path\n self.batch = batch_size\n self.dim = dim\n self.n_channels = n_channels\n self.to_fit = to_fit
`def __len__(self):\n return int(np.floor(self.df.shape[0])/self.batch)\n\ndef __getitem__(self, index):\n list_IDs = self.df['image_name'].values[index*self.batch : (index+1)*self.batch]\n X = self._generate_X(list_IDs)\n if self.to_fit:\n target_y = self._generate_y(list_IDs)\n return np.array(X), np.array(target_y)\n return np.array(X)\n\ndef _generate_X(self,list_IDs):\n X = Parallel(n_jobs=self.batch)(delayed(self._load_image)(i) for i in list_IDs)\n return X\n\ndef _generate_y(self,list_IDs):\n target_y = []\n for i, ids in enumerate(list_IDs):\n target_y.append(self.df[self.df['image_name'] == ids]['target'].values[0])\n return target_y\n\ndef _load_image(self,file_):\n img = cv2.imread(os.path.join(self.image_path,file_)+'.jpg')\n img = cv2.cvtColor(img,cv2.COLOR_BGR2RGB)\n return img`\n `\nI see your data generator resizes images. What is original size on Disk and what size do you resize to?
", "votes": 0, "replies": [] }, { "id": 875177, "author_name": "MhdSharuk", "author_url": "", "post_date": "2020-06-05T15:05:20.067000", "content": "Sorry i forgoted to comment that.But im really not using\nIm just reading and changing the channels
", "votes": 1, "replies": [] } ] }, { "id": 875128, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-06-05T14:42:46.657000", "content": "Are you reading in 1024x1024 sized images and then resizing to 224x224? That is what Fred is doing here and that is slow. It is better to resize all the images and save them to smaller files on disk. Then load the smaller files and don't spend time resizing them during training.
", "votes": 0, "replies": [ { "id": 875144, "author_name": "MhdSharuk", "author_url": "", "post_date": "2020-06-05T14:49:28.533000", "content": "No, I saved the 300x300 jpeg files by resizing to 224x224 by some preprocessing.\nI am just reading the files from the the 224x224 folder
\n\nUPDATE : Now im doing the prediction and it took 2-1/2 hours and still continuing
", "votes": 0, "replies": [] } ] }, { "id": 875084, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-06-05T14:07:31.717000", "content": "What size images are you using? I suggest doing experiments with smaller images like 256x256, 192x192, or 128x128 if you're only using one GPU P100. With more or faster GPUs, you can use larger images.
", "votes": 0, "replies": [ { "id": 875086, "author_name": "MhdSharuk", "author_url": "", "post_date": "2020-06-05T14:09:12.897000", "content": "I was just using 224x224 images with a hair removal technique\nThe gpus i only have is from kaggle and colab only(Not Colab Pro!!)
\n\nThe above code is right way to connect to GPUs right??\nor did i miss something??
", "votes": 0, "replies": [] }, { "id": 875099, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-06-05T14:14:31.563000", "content": "Ok, something is wrong then. Maybe you are running on CPU? Or perhaps your dataloader is slow. Also i'm not familiar if these settings affect GPU
\n\ntf.compat.v1.ConfigProto(intra_op_parallelism_threads=1,inter_op_parallelism_threads=1)\ngraph = tf.compat.v1.get_default_graph()\nsess = tf.compat.v1.Session(graph=graph,config=config)\ntf.compat.v1.keras.backend.set_session(sess)\nOnly having 1 intra and 1 inter 1 thread sounds slow. Can you run your code without these configurations?
", "votes": 0, "replies": [] }, { "id": 875100, "author_name": "MhdSharuk", "author_url": "", "post_date": "2020-06-05T14:15:30.790000", "content": "These codes i picked up from the Ion-Switching competition where they used wavenet model to run on gpus..remember??
", "votes": 0, "replies": [] }, { "id": 875104, "author_name": "MhdSharuk", "author_url": "", "post_date": "2020-06-05T14:20:04.103000", "content": "Im using the tensorflow imagedatagenerator class
", "votes": 0, "replies": [] }, { "id": 875106, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-06-05T14:20:41.590000", "content": "Yes, you are right. They were use in Ion Comp and didn't cause problems. So they're probably not the problem. (But I'm not sure how they affect GPU and I never use those lines).
", "votes": 0, "replies": [] }, { "id": 875107, "author_name": "", "author_url": "", "post_date": "2020-06-05T14:21:56.947000", "content": "", "votes": 0, "replies": [] }, { "id": 875108, "author_name": "MhdSharuk", "author_url": "", "post_date": "2020-06-05T14:22:41.997000", "content": "Hey i could share my entire notebook to you.Could you help me to check those errors.
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "875031": "I tried to connect to gpu with this code below and also by changing the accelarator\n`print('Setting GPU')`\n `K.clear_session()`\n `config = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=1,inter_op_parallelism_threads=1)`\n `graph = tf.compat.v1.get_default_graph()`\n `sess = tf.compat.v1.Session(graph=graph,config=config)`\n `tf.compat.v1.keras.backend.set_session(sess)`\n\nAnd i simply used the Keras ResNet50 model to train the jpeg images\nBut by looking at the below image. Im still confused why even though i connnected to gpu\nmy training is slower.Could someone help me through this\n\n\nEDIT : And this slow training is happening in google colab also with Tesla P-100 GPU", "876387": "@cdeotte any updates on my error chris???", "875155": "Fred posted some code [here][1]. His training is also slow using `tensorflow imagedatagenerator class`. When i get time, I'll play with his code and see what is slowing it down. I think if we can speed up his code, it would speed up your code.\n\n[1]: https://www.kaggle.com/fredericods/melanoma-classification/", "875128": "Are you reading in 1024x1024 sized images and then resizing to 224x224? That is what Fred is doing [here][1] and that is slow. It is better to resize all the images and save them to smaller files on disk. Then load the smaller files and don't spend time resizing them during training.\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/156303", "875084": "What size images are you using? I suggest doing experiments with smaller images like 256x256, 192x192, or 128x128 if you're only using one GPU P100. With more or faster GPUs, you can use larger images." } }