Hello I am a student trying to learn about MaskRCNN, I was trying to run the starter jupyter code and when I am running the training I got the error.
\n\nI have tensorflow cpu installed, any ideas?
\n\nStarting at epoch 0. LR=0.001\n\nCheckpoint Path: C:\\Users\\deepfailure\\Desktop\\ml-lessons-master\\lesson3-data\\logs\\pneumonia20181005T0246\\mask_rcnn_pneumonia_{epoch:04d}.h5\nSelecting layers to train\nconv1 (Conv2D)\nbn_conv1 (BatchNorm)\nres2a_branch2a (Conv2D)\nbn2a_branch2a (BatchNorm)\nres2a_branch2b (Conv2D)\nbn2a_branch2b (BatchNorm)\nres2a_branch2c (Conv2D)\nres2a_branch1 (Conv2D)\nbn2a_branch2c (BatchNorm)\nbn2a_branch1 (BatchNorm)\nres2b_branch2a (Conv2D)\nbn2b_branch2a (BatchNorm)\nres2b_branch2b (Conv2D)\nbn2b_branch2b (BatchNorm)\nres2b_branch2c (Conv2D)\nbn2b_branch2c (BatchNorm)\nres2c_branch2a (Conv2D)\nbn2c_branch2a (BatchNorm)\nres2c_branch2b (Conv2D)\nbn2c_branch2b (BatchNorm)\nres2c_branch2c (Conv2D)\nbn2c_branch2c (BatchNorm)\nres3a_branch2a (Conv2D)\nbn3a_branch2a (BatchNorm)\nres3a_branch2b (Conv2D)\nbn3a_branch2b (BatchNorm)\nres3a_branch2c (Conv2D)\nres3a_branch1 (Conv2D)\nbn3a_branch2c (BatchNorm)\nbn3a_branch1 (BatchNorm)\nres3b_branch2a (Conv2D)\nbn3b_branch2a (BatchNorm)\nres3b_branch2b (Conv2D)\nbn3b_branch2b (BatchNorm)\nres3b_branch2c (Conv2D)\nbn3b_branch2c (BatchNorm)\nres3c_branch2a (Conv2D)\nbn3c_branch2a (BatchNorm)\nres3c_branch2b (Conv2D)\nbn3c_branch2b (BatchNorm)\nres3c_branch2c (Conv2D)\nbn3c_branch2c (BatchNorm)\nres3d_branch2a (Conv2D)\nbn3d_branch2a (BatchNorm)\nres3d_branch2b (Conv2D)\nbn3d_branch2b (BatchNorm)\nres3d_branch2c (Conv2D)\nbn3d_branch2c (BatchNorm)\nres4a_branch2a (Conv2D)\nbn4a_branch2a (BatchNorm)\nres4a_branch2b (Conv2D)\nbn4a_branch2b (BatchNorm)\nres4a_branch2c (Conv2D)\nres4a_branch1 (Conv2D)\nbn4a_branch2c (BatchNorm)\nbn4a_branch1 (BatchNorm)\nres4b_branch2a (Conv2D)\nbn4b_branch2a (BatchNorm)\nres4b_branch2b (Conv2D)\nbn4b_branch2b (BatchNorm)\nres4b_branch2c (Conv2D)\nbn4b_branch2c (BatchNorm)\nres4c_branch2a (Conv2D)\nbn4c_branch2a (BatchNorm)\nres4c_branch2b (Conv2D)\nbn4c_branch2b (BatchNorm)\nres4c_branch2c (Conv2D)\nbn4c_branch2c (BatchNorm)\nres4d_branch2a (Conv2D)\nbn4d_branch2a (BatchNorm)\nres4d_branch2b (Conv2D)\nbn4d_branch2b (BatchNorm)\nres4d_branch2c (Conv2D)\nbn4d_branch2c (BatchNorm)\nres4e_branch2a (Conv2D)\nbn4e_branch2a (BatchNorm)\nres4e_branch2b (Conv2D)\nbn4e_branch2b (BatchNorm)\nres4e_branch2c (Conv2D)\nbn4e_branch2c (BatchNorm)\nres4f_branch2a (Conv2D)\nbn4f_branch2a (BatchNorm)\nres4f_branch2b (Conv2D)\nbn4f_branch2b (BatchNorm)\nres4f_branch2c (Conv2D)\nbn4f_branch2c (BatchNorm)\nres5a_branch2a (Conv2D)\nbn5a_branch2a (BatchNorm)\nres5a_branch2b (Conv2D)\nbn5a_branch2b (BatchNorm)\nres5a_branch2c (Conv2D)\nres5a_branch1 (Conv2D)\nbn5a_branch2c (BatchNorm)\nbn5a_branch1 (BatchNorm)\nres5b_branch2a (Conv2D)\nbn5b_branch2a (BatchNorm)\nres5b_branch2b (Conv2D)\nbn5b_branch2b (BatchNorm)\nres5b_branch2c (Conv2D)\nbn5b_branch2c (BatchNorm)\nres5c_branch2a (Conv2D)\nbn5c_branch2a (BatchNorm)\nres5c_branch2b (Conv2D)\nbn5c_branch2b (BatchNorm)\nres5c_branch2c (Conv2D)\nbn5c_branch2c (BatchNorm)\nfpn_c5p5 (Conv2D)\nfpn_c4p4 (Conv2D)\nfpn_c3p3 (Conv2D)\nfpn_c2p2 (Conv2D)\nfpn_p5 (Conv2D)\nfpn_p2 (Conv2D)\nfpn_p3 (Conv2D)\nfpn_p4 (Conv2D)\nIn model: rpn_model\n rpn_conv_shared (Conv2D)\n rpn_class_raw (Conv2D)\n rpn_bbox_pred (Conv2D)\nmrcnn_mask_conv1 (TimeDistributed)\nmrcnn_mask_bn1 (TimeDistributed)\nmrcnn_mask_conv2 (TimeDistributed)\nmrcnn_mask_bn2 (TimeDistributed)\nmrcnn_class_conv1 (TimeDistributed)\nmrcnn_class_bn1 (TimeDistributed)\nmrcnn_mask_conv3 (TimeDistributed)\nmrcnn_mask_bn3 (TimeDistributed)\nmrcnn_class_conv2 (TimeDistributed)\nmrcnn_class_bn2 (TimeDistributed)\nmrcnn_mask_conv4 (TimeDistributed)\nmrcnn_mask_bn4 (TimeDistributed)\nmrcnn_bbox_fc (TimeDistributed)\nmrcnn_mask_deconv (TimeDistributed)\nmrcnn_class_logits (TimeDistributed)\nmrcnn_mask (TimeDistributed)\nEpoch 1/1\n---------------------------------------------------------------------------\nInvalidArgumentError Traceback (most recent call last)\n<ipython-input-23-76a0a00b6f11> in <module>\n 8 epochs=NUM_EPOCHS,\n 9 layers='all',\n---> 10 augmentation=augmentation)\n\n~\\Desktop\\ml-lessons-master\\lesson3-data\\Mask_RCNN\\mrcnn\\model.py in train(self, train_dataset, val_dataset, learning_rate, epochs, layers, augmentation, custom_callbacks, no_augmentation_sources)\n 2372 max_queue_size=100,\n 2373 workers=workers,\n-> 2374 use_multiprocessing=True,\n 2375 )\n 2376 self.epoch = max(self.epoch, epochs)\n\nc:\\users\\deepfailure\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\keras\\legacy\\interfaces.py in wrapper(*args, **kwargs)\n 89 warnings.warn('Update your `' + object_name + '` call to the ' +\n 90 'Keras 2 API: ' + signature, stacklevel=2)\n---> 91 return func(*args, **kwargs)\n 92 wrapper._original_function = func\n 93 return wrapper\n\nc:\\users\\deepfailure\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\keras\\engine\\training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)\n 1416 use_multiprocessing=use_multiprocessing,\n 1417 shuffle=shuffle,\n-> 1418 initial_epoch=initial_epoch)\n 1419 \n 1420 @interfaces.legacy_generator_methods_support\n\nc:\\users\\deepfailure\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\keras\\engine\\training_generator.py in fit_generator(model, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)\n 215 outs = model.train_on_batch(x, y,\n 216 sample_weight=sample_weight,\n--> 217 class_weight=class_weight)\n 218 \n 219 outs = to_list(outs)\n\nc:\\users\\deepfailure\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\keras\\engine\\training.py in train_on_batch(self, x, y, sample_weight, class_weight)\n 1215 ins = x + y + sample_weights\n 1216 self._make_train_function()\n-> 1217 outputs = self.train_function(ins)\n 1218 return unpack_singleton(outputs)\n 1219 \n\nc:\\users\\deepfailure\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py in __call__(self, inputs)\n 2713 return self._legacy_call(inputs)\n 2714 \n-> 2715 return self._call(inputs)\n 2716 else:\n 2717 if py_any(is_tensor(x) for x in inputs):\n\nc:\\users\\deepfailure\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py in _call(self, inputs)\n 2673 fetched = self._callable_fn(*array_vals, run_metadata=self.run_metadata)\n 2674 else:\n-> 2675 fetched = self._callable_fn(*array_vals)\n 2676 return fetched[:len(self.outputs)]\n 2677 \n\nc:\\users\\deepfailure\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\tensorflow\\python\\client\\session.py in __call__(self, *args, **kwargs)\n 1397 ret = tf_session.TF_SessionRunCallable(\n 1398 self._session._session, self._handle, args, status,\n-> 1399 run_metadata_ptr)\n 1400 if run_metadata:\n 1401 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)\n\nc:\\users\\deepfailure\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\tensorflow\\python\\framework\\errors_impl.py in __exit__(self, type_arg, value_arg, traceback_arg)\n 524 None, None,\n 525 compat.as_text(c_api.TF_Message(self.status.status)),\n--> 526 c_api.TF_GetCode(self.status.status))\n 527 # Delete the underlying status object from memory otherwise it stays alive\n 528 # as there is a reference to status from this from the traceback due to\n\nInvalidArgumentError: indices[394] = 963 is not in [0, 960)\n [[{{node ROI/GatherV2_16}} = GatherV2[Taxis=DT_INT32, Tindices=DT_INT32, Tparams=DT_FLOAT, _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"](ROI/strided_slice_34, ROI/strided_slice_35, training_1/SGD/gradients/roi_align_classifier/concat_grad/mod)]]\nHello I am a student trying to learn about MaskRCNN, I was trying to run the starter jupyter code and when I am running the training I got the error.
\n\nI have tensorflow cpu installed, any ideas?
\n\nStarting at epoch 0. LR=0.001\n\nCheckpoint Path: C:\\Users\\deepfailure\\Desktop\\ml-lessons-master\\lesson3-data\\logs\\pneumonia20181005T0246\\mask_rcnn_pneumonia_{epoch:04d}.h5\nSelecting layers to train\nconv1 (Conv2D)\nbn_conv1 (BatchNorm)\nres2a_branch2a (Conv2D)\nbn2a_branch2a (BatchNorm)\nres2a_branch2b (Conv2D)\nbn2a_branch2b (BatchNorm)\nres2a_branch2c (Conv2D)\nres2a_branch1 (Conv2D)\nbn2a_branch2c (BatchNorm)\nbn2a_branch1 (BatchNorm)\nres2b_branch2a (Conv2D)\nbn2b_branch2a (BatchNorm)\nres2b_branch2b (Conv2D)\nbn2b_branch2b (BatchNorm)\nres2b_branch2c (Conv2D)\nbn2b_branch2c (BatchNorm)\nres2c_branch2a (Conv2D)\nbn2c_branch2a (BatchNorm)\nres2c_branch2b (Conv2D)\nbn2c_branch2b (BatchNorm)\nres2c_branch2c (Conv2D)\nbn2c_branch2c (BatchNorm)\nres3a_branch2a (Conv2D)\nbn3a_branch2a (BatchNorm)\nres3a_branch2b (Conv2D)\nbn3a_branch2b (BatchNorm)\nres3a_branch2c (Conv2D)\nres3a_branch1 (Conv2D)\nbn3a_branch2c (BatchNorm)\nbn3a_branch1 (BatchNorm)\nres3b_branch2a (Conv2D)\nbn3b_branch2a (BatchNorm)\nres3b_branch2b (Conv2D)\nbn3b_branch2b (BatchNorm)\nres3b_branch2c (Conv2D)\nbn3b_branch2c (BatchNorm)\nres3c_branch2a (Conv2D)\nbn3c_branch2a (BatchNorm)\nres3c_branch2b (Conv2D)\nbn3c_branch2b (BatchNorm)\nres3c_branch2c (Conv2D)\nbn3c_branch2c (BatchNorm)\nres3d_branch2a (Conv2D)\nbn3d_branch2a (BatchNorm)\nres3d_branch2b (Conv2D)\nbn3d_branch2b (BatchNorm)\nres3d_branch2c (Conv2D)\nbn3d_branch2c (BatchNorm)\nres4a_branch2a (Conv2D)\nbn4a_branch2a (BatchNorm)\nres4a_branch2b (Conv2D)\nbn4a_branch2b (BatchNorm)\nres4a_branch2c (Conv2D)\nres4a_branch1 (Conv2D)\nbn4a_branch2c (BatchNorm)\nbn4a_branch1 (BatchNorm)\nres4b_branch2a (Conv2D)\nbn4b_branch2a (BatchNorm)\nres4b_branch2b (Conv2D)\nbn4b_branch2b (BatchNorm)\nres4b_branch2c (Conv2D)\nbn4b_branch2c (BatchNorm)\nres4c_branch2a (Conv2D)\nbn4c_branch2a (BatchNorm)\nres4c_branch2b (Conv2D)\nbn4c_branch2b (BatchNorm)\nres4c_branch2c (Conv2D)\nbn4c_branch2c (BatchNorm)\nres4d_branch2a (Conv2D)\nbn4d_branch2a (BatchNorm)\nres4d_branch2b (Conv2D)\nbn4d_branch2b (BatchNorm)\nres4d_branch2c (Conv2D)\nbn4d_branch2c (BatchNorm)\nres4e_branch2a (Conv2D)\nbn4e_branch2a (BatchNorm)\nres4e_branch2b (Conv2D)\nbn4e_branch2b (BatchNorm)\nres4e_branch2c (Conv2D)\nbn4e_branch2c (BatchNorm)\nres4f_branch2a (Conv2D)\nbn4f_branch2a (BatchNorm)\nres4f_branch2b (Conv2D)\nbn4f_branch2b (BatchNorm)\nres4f_branch2c (Conv2D)\nbn4f_branch2c (BatchNorm)\nres5a_branch2a (Conv2D)\nbn5a_branch2a (BatchNorm)\nres5a_branch2b (Conv2D)\nbn5a_branch2b (BatchNorm)\nres5a_branch2c (Conv2D)\nres5a_branch1 (Conv2D)\nbn5a_branch2c (BatchNorm)\nbn5a_branch1 (BatchNorm)\nres5b_branch2a (Conv2D)\nbn5b_branch2a (BatchNorm)\nres5b_branch2b (Conv2D)\nbn5b_branch2b (BatchNorm)\nres5b_branch2c (Conv2D)\nbn5b_branch2c (BatchNorm)\nres5c_branch2a (Conv2D)\nbn5c_branch2a (BatchNorm)\nres5c_branch2b (Conv2D)\nbn5c_branch2b (BatchNorm)\nres5c_branch2c (Conv2D)\nbn5c_branch2c (BatchNorm)\nfpn_c5p5 (Conv2D)\nfpn_c4p4 (Conv2D)\nfpn_c3p3 (Conv2D)\nfpn_c2p2 (Conv2D)\nfpn_p5 (Conv2D)\nfpn_p2 (Conv2D)\nfpn_p3 (Conv2D)\nfpn_p4 (Conv2D)\nIn model: rpn_model\n rpn_conv_shared (Conv2D)\n rpn_class_raw (Conv2D)\n rpn_bbox_pred (Conv2D)\nmrcnn_mask_conv1 (TimeDistributed)\nmrcnn_mask_bn1 (TimeDistributed)\nmrcnn_mask_conv2 (TimeDistributed)\nmrcnn_mask_bn2 (TimeDistributed)\nmrcnn_class_conv1 (TimeDistributed)\nmrcnn_class_bn1 (TimeDistributed)\nmrcnn_mask_conv3 (TimeDistributed)\nmrcnn_mask_bn3 (TimeDistributed)\nmrcnn_class_conv2 (TimeDistributed)\nmrcnn_class_bn2 (TimeDistributed)\nmrcnn_mask_conv4 (TimeDistributed)\nmrcnn_mask_bn4 (TimeDistributed)\nmrcnn_bbox_fc (TimeDistributed)\nmrcnn_mask_deconv (TimeDistributed)\nmrcnn_class_logits (TimeDistributed)\nmrcnn_mask (TimeDistributed)\nEpoch 1/1\n---------------------------------------------------------------------------\nInvalidArgumentError Traceback (most recent call last)\n<ipython-input-23-76a0a00b6f11> in <module>\n 8 epochs=NUM_EPOCHS,\n 9 layers='all',\n---> 10 augmentation=augmentation)\n\n~\\Desktop\\ml-lessons-master\\lesson3-data\\Mask_RCNN\\mrcnn\\model.py in train(self, train_dataset, val_dataset, learning_rate, epochs, layers, augmentation, custom_callbacks, no_augmentation_sources)\n 2372 max_queue_size=100,\n 2373 workers=workers,\n-> 2374 use_multiprocessing=True,\n 2375 )\n 2376 self.epoch = max(self.epoch, epochs)\n\nc:\\users\\deepfailure\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\keras\\legacy\\interfaces.py in wrapper(*args, **kwargs)\n 89 warnings.warn('Update your `' + object_name + '` call to the ' +\n 90 'Keras 2 API: ' + signature, stacklevel=2)\n---> 91 return func(*args, **kwargs)\n 92 wrapper._original_function = func\n 93 return wrapper\n\nc:\\users\\deepfailure\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\keras\\engine\\training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)\n 1416 use_multiprocessing=use_multiprocessing,\n 1417 shuffle=shuffle,\n-> 1418 initial_epoch=initial_epoch)\n 1419 \n 1420 @interfaces.legacy_generator_methods_support\n\nc:\\users\\deepfailure\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\keras\\engine\\training_generator.py in fit_generator(model, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)\n 215 outs = model.train_on_batch(x, y,\n 216 sample_weight=sample_weight,\n--> 217 class_weight=class_weight)\n 218 \n 219 outs = to_list(outs)\n\nc:\\users\\deepfailure\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\keras\\engine\\training.py in train_on_batch(self, x, y, sample_weight, class_weight)\n 1215 ins = x + y + sample_weights\n 1216 self._make_train_function()\n-> 1217 outputs = self.train_function(ins)\n 1218 return unpack_singleton(outputs)\n 1219 \n\nc:\\users\\deepfailure\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py in __call__(self, inputs)\n 2713 return self._legacy_call(inputs)\n 2714 \n-> 2715 return self._call(inputs)\n 2716 else:\n 2717 if py_any(is_tensor(x) for x in inputs):\n\nc:\\users\\deepfailure\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py in _call(self, inputs)\n 2673 fetched = self._callable_fn(*array_vals, run_metadata=self.run_metadata)\n 2674 else:\n-> 2675 fetched = self._callable_fn(*array_vals)\n 2676 return fetched[:len(self.outputs)]\n 2677 \n\nc:\\users\\deepfailure\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\tensorflow\\python\\client\\session.py in __call__(self, *args, **kwargs)\n 1397 ret = tf_session.TF_SessionRunCallable(\n 1398 self._session._session, self._handle, args, status,\n-> 1399 run_metadata_ptr)\n 1400 if run_metadata:\n 1401 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)\n\nc:\\users\\deepfailure\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\tensorflow\\python\\framework\\errors_impl.py in __exit__(self, type_arg, value_arg, traceback_arg)\n 524 None, None,\n 525 compat.as_text(c_api.TF_Message(self.status.status)),\n--> 526 c_api.TF_GetCode(self.status.status))\n 527 # Delete the underlying status object from memory otherwise it stays alive\n 528 # as there is a reference to status from this from the traceback due to\n\nInvalidArgumentError: indices[394] = 963 is not in [0, 960)\n [[{{node ROI/GatherV2_16}} = GatherV2[Taxis=DT_INT32, Tindices=DT_INT32, Tparams=DT_FLOAT, _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"](ROI/strided_slice_34, ROI/strided_slice_35, training_1/SGD/gradients/roi_align_classifier/concat_grad/mod)]]\nHi deepfailure (lol nice username ;)
\n\nWhere did you get your code from? (lesson3?)\nI try getting an updated version of keras and cloning the Mask_RCNN from github like in https://www.kaggle.com/hmendonca/mask-rcnn-and-coco-transfer-learning-lb-0-155/notebook
", "rawMarkdown": "Hi deepfailure (lol nice username ;)\n\nWhere did you get your code from? (lesson3?)\nI try getting an updated version of keras and cloning the Mask_RCNN from github like in https://www.kaggle.com/hmendonca/mask-rcnn-and-coco-transfer-learning-lb-0-155/notebook\n", "votes": 2, "replies": [ { "id": 399412, "postDate": "2018-10-05T18:23:06.630Z", "content": "I am now getting a similar error now on code that used to work, even in an environment where I just cloned Mask_RCNN this morning.
", "rawMarkdown": "I am now getting a similar error now on code that used to work, even in an environment where I just cloned Mask_RCNN this morning." }, { "id": 399432, "postDate": "2018-10-05T19:14:58.503Z", "content": "Apparently my old code still works on the old GCP instance that used Tensorflow 1.10.1. and Keras 2.2.3. On the new instance it is TF 1.11.0 and Keras 2.2.4, and it doesn't work. Maybe the new version breaks Mask_RCNN. It also seems to work on a Colab instance using TF 1.11.0 and Keras 2.1.6, so maybe I'll try to downgrade Keras on the new GCP instance.
", "rawMarkdown": "Apparently my old code still works on the old GCP instance that used Tensorflow 1.10.1. and Keras 2.2.3. On the new instance it is TF 1.11.0 and Keras 2.2.4, and it doesn't work. Maybe the new version breaks Mask_RCNN. It also seems to work on a Colab instance using TF 1.11.0 and Keras 2.1.6, so maybe I'll try to downgrade Keras on the new GCP instance." }, { "id": 399440, "postDate": "2018-10-05T19:30:50.383Z", "content": "No, downgrading Keras doesn't solve the problem.
", "rawMarkdown": "No, downgrading Keras doesn't solve the problem." }, { "id": 399465, "postDate": "2018-10-05T20:51:07.057Z", "content": "OK, I think I've finally solved my problem, but I don't know if my solution fixes the OP's problem. There seem to be two separate issues here. The first is that MRCNN doesn't work properly with the CPU version of Tensorflow, for reasons that are unclear to me. The second is that when MRCNN installs on a GCP Tensorflow deep learning instance, it upgrades tensorflow-gpu to a version that doesn't work with Google's deep learning image, so after installing MRCNN, when I imported tensorflow, I got the CPU version (about which see the previous sentence). So I had to uninstall tensorflow-gpu and reinstall the version that my instance likes. (The requirement for an image-specific version of Tensorflow is described in the text that appears when you log in to the instance.) In my case the reinstall command was:
\n\n!pip3 install /opt/deeplearning/binaries/tensorflow/tensorflow_gpu-1.10.1-cp35-cp35m-linux_x86_64.whl\nbut this will of course differ depending on the details of the image you're using.
", "rawMarkdown": "OK, I think I've finally solved my problem, but I don't know if my solution fixes the OP's problem. There seem to be two separate issues here. The first is that MRCNN doesn't work properly with the CPU version of Tensorflow, for reasons that are unclear to me. The second is that when MRCNN installs on a GCP Tensorflow deep learning instance, it upgrades tensorflow-gpu to a version that doesn't work with Google's deep learning image, so after installing MRCNN, when I imported tensorflow, I got the CPU version (about which see the previous sentence). So I had to uninstall tensorflow-gpu and reinstall the version that my instance likes. (The requirement for an image-specific version of Tensorflow is described in the text that appears when you log in to the instance.) In my case the reinstall command was:\n\n !pip3 install /opt/deeplearning/binaries/tensorflow/tensorflow_gpu-1.10.1-cp35-cp35m-linux_x86_64.whl\n\nbut this will of course differ depending on the details of the image you're using.", "votes": 2 }, { "id": 399483, "postDate": "2018-10-05T21:39:08.780Z", "rawMarkdown": "", "isDeleted": true }, { "id": 399501, "postDate": "2018-10-05T23:09:39.087Z", "content": "Yeah, I'm seeing the same error when running a kaggle kernel :/
\n\nhttps://www.kaggle.com/hmendonca/mask-rcnn-and-coco-transfer-learning-lb-0-155/notebook
", "rawMarkdown": "Yeah, I'm seeing the same error when running a kaggle kernel :/\n\nhttps://www.kaggle.com/hmendonca/mask-rcnn-and-coco-transfer-learning-lb-0-155/notebook" }, { "id": 399752, "postDate": "2018-10-06T17:14:18.960Z", "content": "I think your kaggle kernel error is a different one.
", "rawMarkdown": "I think your kaggle kernel error is a different one." }, { "id": 399797, "postDate": "2018-10-06T19:25:17.027Z", "content": "Yeah, you're right Andy. My error seems to be related to the dataset selection itself. Lame me!
", "rawMarkdown": "Yeah, you're right Andy. My error seems to be related to the dataset selection itself. Lame me!" }, { "id": 400561, "postDate": "2018-10-08T14:48:34.537Z", "content": "Hello Henrique, the first code I used was from lesson-3 and I installed most of the stuff following an online tutorial: https://www.youtube.com/watch?v=2TikTv6PWDw&t=468s (in the tutorial they installed tensorflow gpu but I just install the requirements seen inhttps://github.com/matterport/Mask_RCNN also I clone matterpot not the other github in the tutorial.)
\n\nNow I tried to follow your code with a different computer but I still get an error on the same place but it is a different one. (Error below)
\n\nCould it be that there is an error when installing MaskRCNN? When I install it this warning appears:
\n\nWARNING:root:Fail load requirements file, so using default ones.\nzip_safe flag not set; analyzing archive contents...
\n\nThats all I get when installing MaskRCNN so I am not sure if I installed it well.
\n\nThe thing is I have checked and now pip does not have pip.req but pip._internal.req but I can't make it work with pip_internal.req
\n\nIn any case I have all the requirements installed so If the function parse_requirements only checks for requirements it should not be a problem
\n\nBy the way I am using Windows 10, installed tensorflow with pip.
\n\nThanks for everything guys!
\n\nError I get now:
\n\nStarting at epoch 0. LR=0.01\n\nCheckpoint Path: C:/Users/tom_9/OneDrive/Escritorio/ml-lessons-master/kaggle/working\\pneumonia20181008T1620\\mask_rcnn_pneumonia_{epoch:04d}.h5\nSelecting layers to train\nfpn_c5p5 (Conv2D)\nfpn_c4p4 (Conv2D)\nfpn_c3p3 (Conv2D)\nfpn_c2p2 (Conv2D)\nfpn_p5 (Conv2D)\nfpn_p2 (Conv2D)\nfpn_p3 (Conv2D)\nfpn_p4 (Conv2D)\nIn model: rpn_model\n rpn_conv_shared (Conv2D)\n rpn_class_raw (Conv2D)\n rpn_bbox_pred (Conv2D)\nmrcnn_mask_conv1 (TimeDistributed)\nmrcnn_mask_bn1 (TimeDistributed)\nmrcnn_mask_conv2 (TimeDistributed)\nmrcnn_mask_bn2 (TimeDistributed)\nmrcnn_class_conv1 (TimeDistributed)\nmrcnn_class_bn1 (TimeDistributed)\nmrcnn_mask_conv3 (TimeDistributed)\nmrcnn_mask_bn3 (TimeDistributed)\nmrcnn_class_conv2 (TimeDistributed)\nmrcnn_class_bn2 (TimeDistributed)\nmrcnn_mask_conv4 (TimeDistributed)\nmrcnn_mask_bn4 (TimeDistributed)\nmrcnn_bbox_fc (TimeDistributed)\nmrcnn_mask_deconv (TimeDistributed)\nmrcnn_class_logits (TimeDistributed)\nmrcnn_mask (TimeDistributed)\nEpoch 1/1\n---------------------------------------------------------------------------\nResourceExhaustedError Traceback (most recent call last)\n<timed exec> in <module>\n\n~\\OneDrive\\Escritorio\\ml-lessons-master\\kaggle\\Mask_RCNN\\mrcnn\\model.py in train(self, train_dataset, val_dataset, learning_rate, epochs, layers, augmentation, custom_callbacks, no_augmentation_sources)\n 2372 max_queue_size=100,\n 2373 workers=workers,\n-> 2374 use_multiprocessing=True,\n 2375 )\n 2376 self.epoch = max(self.epoch, epochs)\n\nc:\\users\\tom_9\\appdata\\local\\continuum\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\keras\\legacy\\interfaces.py in wrapper(*args, **kwargs)\n 89 warnings.warn('Update your `' + object_name + '` call to the ' +\n 90 'Keras 2 API: ' + signature, stacklevel=2)\n---> 91 return func(*args, **kwargs)\n 92 wrapper._original_function = func\n 93 return wrapper\n\nc:\\users\\tom_9\\appdata\\local\\continuum\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\keras\\engine\\training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)\n 1416 use_multiprocessing=use_multiprocessing,\n 1417 shuffle=shuffle,\n-> 1418 initial_epoch=initial_epoch)\n 1419 \n 1420 @interfaces.legacy_generator_methods_support\n\nc:\\users\\tom_9\\appdata\\local\\continuum\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\keras\\engine\\training_generator.py in fit_generator(model, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)\n 215 outs = model.train_on_batch(x, y,\n 216 sample_weight=sample_weight,\n--> 217 class_weight=class_weight)\n 218 \n 219 outs = to_list(outs)\n\nc:\\users\\tom_9\\appdata\\local\\continuum\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\keras\\engine\\training.py in train_on_batch(self, x, y, sample_weight, class_weight)\n 1215 ins = x + y + sample_weights\n 1216 self._make_train_function()\n-> 1217 outputs = self.train_function(ins)\n 1218 return unpack_singleton(outputs)\n 1219 \n\nc:\\users\\tom_9\\appdata\\local\\continuum\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py in __call__(self, inputs)\n 2713 return self._legacy_call(inputs)\n 2714 \n-> 2715 return self._call(inputs)\n 2716 else:\n 2717 if py_any(is_tensor(x) for x in inputs):\n\nc:\\users\\tom_9\\appdata\\local\\continuum\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py in _call(self, inputs)\n 2673 fetched = self._callable_fn(*array_vals, run_metadata=self.run_metadata)\n 2674 else:\n-> 2675 fetched = self._callable_fn(*array_vals)\n 2676 return fetched[:len(self.outputs)]\n 2677 \n\nc:\\users\\tom_9\\appdata\\local\\continuum\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\tensorflow\\python\\client\\session.py in __call__(self, *args, **kwargs)\n 1397 ret = tf_session.TF_SessionRunCallable(\n 1398 self._session._session, self._handle, args, status,\n-> 1399 run_metadata_ptr)\n 1400 if run_metadata:\n 1401 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)\n\nc:\\users\\tom_9\\appdata\\local\\continuum\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\tensorflow\\python\\framework\\errors_impl.py in __exit__(self, type_arg, value_arg, traceback_arg)\n 524 None, None,\n 525 compat.as_text(c_api.TF_Message(self.status.status)),\n--> 526 c_api.TF_GetCode(self.status.status))\n 527 # Delete the underlying status object from memory otherwise it stays alive\n 528 # as there is a reference to status from this from the traceback due to\n\nResourceExhaustedError: OOM when allocating tensor with shape[8,64,64,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc\n [[{{node fpn_p3upsampled_2/ResizeNearestNeighbor}} = ResizeNearestNeighbor[T=DT_FLOAT, _class=[\"loc:@train...ighborGrad\"], align_corners=false, _device=\"/job:localhost/replica:0/task:0/device:GPU:0\"](fpn_p3add_2/add-3-0-TransposeNCHWToNHWC-LayoutOptimizer, fpn_p3upsampled_2/mul)]]\nHint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.\n\n [[{{node ROI_2/strided_slice_172/_4643}} = _Recv[client_terminated=false, recv_device=\"/job:localhost/replica:0/task:0/device:CPU:0\", send_device=\"/job:localhost/replica:0/task:0/device:GPU:0\", send_device_incarnation=1, tensor_name=\"edge_4919_ROI_2/strided_slice_172\", tensor_type=DT_FLOAT, _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"]()]]\nHint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.\n@deepfailure It looks like you are running out of GPU memory. (I don't think the install warning is a problem.) You may need to set a smaller batch size. Or if other programs might have been using the GPU, they may need to be terminated. And if you are running it in a notebook, you may need to restart the kernel so it releases its GPU memory.
", "rawMarkdown": "@deepfailure It looks like you are running out of GPU memory. (I don't think the install warning is a problem.) You may need to set a smaller batch size. Or if other programs might have been using the GPU, they may need to be terminated. And if you are running it in a notebook, you may need to restart the kernel so it releases its GPU memory." } ] }, { "id": 408651, "postDate": "2018-10-23T08:24:59.710Z", "content": "Hello, sorry for answering so late but in case it helps someone I solved it by using GPU instead of CPU and using Keras version 2.2.3.
", "rawMarkdown": "Hello, sorry for answering so late but in case it helps someone I solved it by using GPU instead of CPU and using Keras version 2.2.3." }, { "id": 401365, "postDate": "2018-10-10T00:22:58.547Z", "content": "I get another error on Google Cloud Platform. GPU Tesla k80, anaconda3, python 3.5 \nValueError(\"Function has keyword-only arguments or annotations\"\nValueError: Function has keyword-only arguments or annotations, use getfullargspec() API which can support them when i try to run \nmodel.train(...) \nOn google colab similar lesson 3 works. Google gives this solution \nLink\nBut I still do not understand where to fix it. I am very new to IT, maybe someone here knows what to do?
Hi deepfailure (lol nice username ;)
\n\nWhere did you get your code from? (lesson3?)\nI try getting an updated version of keras and cloning the Mask_RCNN from github like in https://www.kaggle.com/hmendonca/mask-rcnn-and-coco-transfer-learning-lb-0-155/notebook
", "votes": 2, "replies": [ { "id": 399412, "author_name": "Andy Harless", "author_url": "", "post_date": "2018-10-05T18:23:06.630000", "content": "I am now getting a similar error now on code that used to work, even in an environment where I just cloned Mask_RCNN this morning.
", "votes": 0, "replies": [] }, { "id": 399432, "author_name": "Andy Harless", "author_url": "", "post_date": "2018-10-05T19:14:58.503000", "content": "Apparently my old code still works on the old GCP instance that used Tensorflow 1.10.1. and Keras 2.2.3. On the new instance it is TF 1.11.0 and Keras 2.2.4, and it doesn't work. Maybe the new version breaks Mask_RCNN. It also seems to work on a Colab instance using TF 1.11.0 and Keras 2.1.6, so maybe I'll try to downgrade Keras on the new GCP instance.
", "votes": 0, "replies": [] }, { "id": 399440, "author_name": "Andy Harless", "author_url": "", "post_date": "2018-10-05T19:30:50.383000", "content": "No, downgrading Keras doesn't solve the problem.
", "votes": 0, "replies": [] }, { "id": 399465, "author_name": "Andy Harless", "author_url": "", "post_date": "2018-10-05T20:51:07.057000", "content": "OK, I think I've finally solved my problem, but I don't know if my solution fixes the OP's problem. There seem to be two separate issues here. The first is that MRCNN doesn't work properly with the CPU version of Tensorflow, for reasons that are unclear to me. The second is that when MRCNN installs on a GCP Tensorflow deep learning instance, it upgrades tensorflow-gpu to a version that doesn't work with Google's deep learning image, so after installing MRCNN, when I imported tensorflow, I got the CPU version (about which see the previous sentence). So I had to uninstall tensorflow-gpu and reinstall the version that my instance likes. (The requirement for an image-specific version of Tensorflow is described in the text that appears when you log in to the instance.) In my case the reinstall command was:
\n\n!pip3 install /opt/deeplearning/binaries/tensorflow/tensorflow_gpu-1.10.1-cp35-cp35m-linux_x86_64.whl\nbut this will of course differ depending on the details of the image you're using.
", "votes": 2, "replies": [] }, { "id": 399483, "author_name": "", "author_url": "", "post_date": "2018-10-05T21:39:08.780000", "content": "", "votes": 0, "replies": [] }, { "id": 399501, "author_name": "Henrique Mendonça", "author_url": "", "post_date": "2018-10-05T23:09:39.087000", "content": "Yeah, I'm seeing the same error when running a kaggle kernel :/
\n\nhttps://www.kaggle.com/hmendonca/mask-rcnn-and-coco-transfer-learning-lb-0-155/notebook
", "votes": 0, "replies": [] }, { "id": 399752, "author_name": "Andy Harless", "author_url": "", "post_date": "2018-10-06T17:14:18.960000", "content": "I think your kaggle kernel error is a different one.
", "votes": 0, "replies": [] }, { "id": 399797, "author_name": "Henrique Mendonça", "author_url": "", "post_date": "2018-10-06T19:25:17.027000", "content": "Yeah, you're right Andy. My error seems to be related to the dataset selection itself. Lame me!
", "votes": 0, "replies": [] }, { "id": 400561, "author_name": "deepfailure", "author_url": "", "post_date": "2018-10-08T14:48:34.537000", "content": "Hello Henrique, the first code I used was from lesson-3 and I installed most of the stuff following an online tutorial: https://www.youtube.com/watch?v=2TikTv6PWDw&t=468s (in the tutorial they installed tensorflow gpu but I just install the requirements seen inhttps://github.com/matterport/Mask_RCNN also I clone matterpot not the other github in the tutorial.)
\n\nNow I tried to follow your code with a different computer but I still get an error on the same place but it is a different one. (Error below)
\n\nCould it be that there is an error when installing MaskRCNN? When I install it this warning appears:
\n\nWARNING:root:Fail load requirements file, so using default ones.\nzip_safe flag not set; analyzing archive contents...
\n\nThats all I get when installing MaskRCNN so I am not sure if I installed it well.
\n\nThe thing is I have checked and now pip does not have pip.req but pip._internal.req but I can't make it work with pip_internal.req
\n\nIn any case I have all the requirements installed so If the function parse_requirements only checks for requirements it should not be a problem
\n\nBy the way I am using Windows 10, installed tensorflow with pip.
\n\nThanks for everything guys!
\n\nError I get now:
\n\nStarting at epoch 0. LR=0.01\n\nCheckpoint Path: C:/Users/tom_9/OneDrive/Escritorio/ml-lessons-master/kaggle/working\\pneumonia20181008T1620\\mask_rcnn_pneumonia_{epoch:04d}.h5\nSelecting layers to train\nfpn_c5p5 (Conv2D)\nfpn_c4p4 (Conv2D)\nfpn_c3p3 (Conv2D)\nfpn_c2p2 (Conv2D)\nfpn_p5 (Conv2D)\nfpn_p2 (Conv2D)\nfpn_p3 (Conv2D)\nfpn_p4 (Conv2D)\nIn model: rpn_model\n rpn_conv_shared (Conv2D)\n rpn_class_raw (Conv2D)\n rpn_bbox_pred (Conv2D)\nmrcnn_mask_conv1 (TimeDistributed)\nmrcnn_mask_bn1 (TimeDistributed)\nmrcnn_mask_conv2 (TimeDistributed)\nmrcnn_mask_bn2 (TimeDistributed)\nmrcnn_class_conv1 (TimeDistributed)\nmrcnn_class_bn1 (TimeDistributed)\nmrcnn_mask_conv3 (TimeDistributed)\nmrcnn_mask_bn3 (TimeDistributed)\nmrcnn_class_conv2 (TimeDistributed)\nmrcnn_class_bn2 (TimeDistributed)\nmrcnn_mask_conv4 (TimeDistributed)\nmrcnn_mask_bn4 (TimeDistributed)\nmrcnn_bbox_fc (TimeDistributed)\nmrcnn_mask_deconv (TimeDistributed)\nmrcnn_class_logits (TimeDistributed)\nmrcnn_mask (TimeDistributed)\nEpoch 1/1\n---------------------------------------------------------------------------\nResourceExhaustedError Traceback (most recent call last)\n<timed exec> in <module>\n\n~\\OneDrive\\Escritorio\\ml-lessons-master\\kaggle\\Mask_RCNN\\mrcnn\\model.py in train(self, train_dataset, val_dataset, learning_rate, epochs, layers, augmentation, custom_callbacks, no_augmentation_sources)\n 2372 max_queue_size=100,\n 2373 workers=workers,\n-> 2374 use_multiprocessing=True,\n 2375 )\n 2376 self.epoch = max(self.epoch, epochs)\n\nc:\\users\\tom_9\\appdata\\local\\continuum\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\keras\\legacy\\interfaces.py in wrapper(*args, **kwargs)\n 89 warnings.warn('Update your `' + object_name + '` call to the ' +\n 90 'Keras 2 API: ' + signature, stacklevel=2)\n---> 91 return func(*args, **kwargs)\n 92 wrapper._original_function = func\n 93 return wrapper\n\nc:\\users\\tom_9\\appdata\\local\\continuum\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\keras\\engine\\training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)\n 1416 use_multiprocessing=use_multiprocessing,\n 1417 shuffle=shuffle,\n-> 1418 initial_epoch=initial_epoch)\n 1419 \n 1420 @interfaces.legacy_generator_methods_support\n\nc:\\users\\tom_9\\appdata\\local\\continuum\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\keras\\engine\\training_generator.py in fit_generator(model, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)\n 215 outs = model.train_on_batch(x, y,\n 216 sample_weight=sample_weight,\n--> 217 class_weight=class_weight)\n 218 \n 219 outs = to_list(outs)\n\nc:\\users\\tom_9\\appdata\\local\\continuum\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\keras\\engine\\training.py in train_on_batch(self, x, y, sample_weight, class_weight)\n 1215 ins = x + y + sample_weights\n 1216 self._make_train_function()\n-> 1217 outputs = self.train_function(ins)\n 1218 return unpack_singleton(outputs)\n 1219 \n\nc:\\users\\tom_9\\appdata\\local\\continuum\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py in __call__(self, inputs)\n 2713 return self._legacy_call(inputs)\n 2714 \n-> 2715 return self._call(inputs)\n 2716 else:\n 2717 if py_any(is_tensor(x) for x in inputs):\n\nc:\\users\\tom_9\\appdata\\local\\continuum\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py in _call(self, inputs)\n 2673 fetched = self._callable_fn(*array_vals, run_metadata=self.run_metadata)\n 2674 else:\n-> 2675 fetched = self._callable_fn(*array_vals)\n 2676 return fetched[:len(self.outputs)]\n 2677 \n\nc:\\users\\tom_9\\appdata\\local\\continuum\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\tensorflow\\python\\client\\session.py in __call__(self, *args, **kwargs)\n 1397 ret = tf_session.TF_SessionRunCallable(\n 1398 self._session._session, self._handle, args, status,\n-> 1399 run_metadata_ptr)\n 1400 if run_metadata:\n 1401 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)\n\nc:\\users\\tom_9\\appdata\\local\\continuum\\anaconda3\\envs\\maskrcnn\\lib\\site-packages\\tensorflow\\python\\framework\\errors_impl.py in __exit__(self, type_arg, value_arg, traceback_arg)\n 524 None, None,\n 525 compat.as_text(c_api.TF_Message(self.status.status)),\n--> 526 c_api.TF_GetCode(self.status.status))\n 527 # Delete the underlying status object from memory otherwise it stays alive\n 528 # as there is a reference to status from this from the traceback due to\n\nResourceExhaustedError: OOM when allocating tensor with shape[8,64,64,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc\n [[{{node fpn_p3upsampled_2/ResizeNearestNeighbor}} = ResizeNearestNeighbor[T=DT_FLOAT, _class=[\"loc:@train...ighborGrad\"], align_corners=false, _device=\"/job:localhost/replica:0/task:0/device:GPU:0\"](fpn_p3add_2/add-3-0-TransposeNCHWToNHWC-LayoutOptimizer, fpn_p3upsampled_2/mul)]]\nHint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.\n\n [[{{node ROI_2/strided_slice_172/_4643}} = _Recv[client_terminated=false, recv_device=\"/job:localhost/replica:0/task:0/device:CPU:0\", send_device=\"/job:localhost/replica:0/task:0/device:GPU:0\", send_device_incarnation=1, tensor_name=\"edge_4919_ROI_2/strided_slice_172\", tensor_type=DT_FLOAT, _device=\"/job:localhost/replica:0/task:0/device:CPU:0\"]()]]\nHint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.\n@deepfailure It looks like you are running out of GPU memory. (I don't think the install warning is a problem.) You may need to set a smaller batch size. Or if other programs might have been using the GPU, they may need to be terminated. And if you are running it in a notebook, you may need to restart the kernel so it releases its GPU memory.
", "votes": 0, "replies": [] } ] }, { "id": 408651, "author_name": "deepfailure", "author_url": "", "post_date": "2018-10-23T08:24:59.710000", "content": "Hello, sorry for answering so late but in case it helps someone I solved it by using GPU instead of CPU and using Keras version 2.2.3.
", "votes": 0, "replies": [] }, { "id": 401365, "author_name": "Blonde", "author_url": "", "post_date": "2018-10-10T00:22:58.547000", "content": "I get another error on Google Cloud Platform. GPU Tesla k80, anaconda3, python 3.5 \nValueError(\"Function has keyword-only arguments or annotations\"\nValueError: Function has keyword-only arguments or annotations, use getfullargspec() API which can support them when i try to run \nmodel.train(...) \nOn google colab similar lesson 3 works. Google gives this solution \nLink\nBut I still do not understand where to fix it. I am very new to IT, maybe someone here knows what to do?