Hi,
\n\nWith the Notebook docker image, my existing notebook stops working. I got a exception when running CuDNNLSTM (OpKernel is not available) even the GPU option was On. K.batch_dot starts complaining about mismatch shape or tensor conversion issue, which worked well before (I have not changed anything).
\n\nDoes anyone experience the same issues?
\n\nRegards,\nDuong nhu
", "messages": [ { "id": "438120", "postDate": "12/13/2018 06:10:30", "content": "Hi,
\n\nWith the Notebook docker image, my existing notebook stops working. I got a exception when running CuDNNLSTM (OpKernel is not available) even the GPU option was On. K.batch_dot starts complaining about mismatch shape or tensor conversion issue, which worked well before (I have not changed anything).
\n\nDoes anyone experience the same issues?
\n\nRegards,\nDuong nhu
", "rawMarkdown": "Hi,\n\nWith the Notebook docker image, my existing notebook stops working. I got a exception when running CuDNNLSTM (OpKernel is not available) even the GPU option was On. K.batch_dot starts complaining about mismatch shape or tensor conversion issue, which worked well before (I have not changed anything).\n\nDoes anyone experience the same issues?\n\nRegards,\nDuong nhu", "votes": null }, { "id": "438123", "postDate": "12/13/2018 06:16:14", "content": "@Duong - I have also experienced the same thing except with capsule layers. I have not changed anything from kernel implementations.
", "rawMarkdown": "Duong - I have also experienced the same thing except with capsule layers. I have not changed anything from kernel implementations.", "votes": null }, { "id": "438139", "postDate": "12/13/2018 06:55:40", "content": "Yeah!there are errors in the capsule layers now!
", "rawMarkdown": "Yeah!there are errors in the capsule layers now!", "votes": null }, { "id": "438154", "postDate": "12/13/2018 07:34:00", "content": "I cant even run any cuda layer now
", "rawMarkdown": "I cant even run any cuda layer now", "votes": null }, { "id": "438235", "postDate": "12/13/2018 10:12:36", "content": "Cuda layers work for me, capsule doesn't.
", "rawMarkdown": "Cuda layers work for me, capsule doesn't.", "votes": null }, { "id": "438267", "postDate": "12/13/2018 11:47:18", "content": "still unable to run Net with capsule layer using kaggle notebook, locally everythings run perfectly
", "rawMarkdown": "still unable to run Net with capsule layer using kaggle notebook, locally everythings run perfectly", "votes": null }, { "id": "438287", "postDate": "12/13/2018 12:20:05", "content": "(Temporary) solution is to replace K.batch_dot with tf.keras.backend.batch_dot
Tested and run smoothly, thanks for your help
", "rawMarkdown": "Tested and run smoothly, thanks for your help", "votes": null }, { "id": "438493", "postDate": "12/13/2018 19:43:47", "content": "I am also getting the shape mismatch from CuDNN (using PyTorch)
\n\nLooking at the docker image it looks like PyTorch was updated, but that wouldn't explain the issues people are seeing in Keras. I suspect CUDA has been updated also, but I don't see it in the latest commit to the docker image repository at https://github.com/Kaggle/docker-python
\n\n[Update] - latest notebook image has CUDA 9.0.176 and CuDNN version 7.4.01, which is later than the versions a few weeks ago (but I forget exactly what they were)
", "rawMarkdown": "I am also getting the shape mismatch from CuDNN (using PyTorch)\n\nLooking at the docker image it looks like PyTorch was updated, but that wouldn't explain the issues people are seeing in Keras. I suspect CUDA has been updated also, but I don't see it in the latest commit to the docker image repository at https://github.com/Kaggle/docker-python\n\n[Update] - latest notebook image has CUDA 9.0.176 and CuDNN version 7.4.01, which is later than the versions a few weeks ago (but I forget exactly what they were)", "votes": null }, { "id": "438544", "postDate": "12/13/2018 21:35:27", "content": "Old images seem not to work with Cuda. Latest image works. K.batch_dot still complains about mismatched shape.
\n\nTemp solution:
\n\n\n", "rawMarkdown": "Old images seem not to work with Cuda. Latest image works. K.batch_dot still complains about mismatched shape. \n\nTemp solution: \n> **Philipp wrote**\n> \n> > (Temporary) solution is to replace `K.batch_dot` with `tf.keras.backend.batch_dot`", "votes": null }, { "id": "438991", "postDate": "12/14/2018 14:25:31", "content": "Philipp wrote
\n \n\n\n(Temporary) solution is to replace
\nK.batch_dotwithtf.keras.backend.batch_dot
Further update - it's more specific than LSTMs generally. It's actually an incompatibility between PyTorch 1.0 and some code for Drop-connect that comes from originally from a Merity paper (Salesforce) at https://github.com/salesforce/awd-lstm-lm\nI'm trying to find a work-around currently and will report back if I am able to find one.
", "rawMarkdown": "Further update - it's more specific than LSTMs generally. It's actually an incompatibility between PyTorch 1.0 and some code for Drop-connect that comes from originally from a Merity paper (Salesforce) at https://github.com/salesforce/awd-lstm-lm\nI'm trying to find a work-around currently and will report back if I am able to find one.", "votes": null }, { "id": "440514", "postDate": "12/17/2018 16:41:56", "content": "I hacked WeightDrop around in such a way as to make it work (I think correctly) with PyTorch 1.0. See my last post on https://github.com/salesforce/awd-lstm-lm/issues/86
", "rawMarkdown": "I hacked WeightDrop around in such a way as to make it work (I **think** correctly) with PyTorch 1.0. See my last post on https://github.com/salesforce/awd-lstm-lm/issues/86", "votes": null } ], "comments": [ { "id": 438123, "author_name": "rdizzl3", "author_url": "", "post_date": "12/13/2018 06:16:14", "content": "@Duong - I have also experienced the same thing except with capsule layers. I have not changed anything from kernel implementations.
", "votes": null, "replies": [ { "id": 438139, "author_name": "gmhost", "author_url": "", "post_date": "12/13/2018 06:55:40", "content": "Yeah!there are errors in the capsule layers now!
", "votes": null, "replies": [] }, { "id": 438154, "author_name": "", "author_url": "", "post_date": "12/13/2018 07:34:00", "content": "I cant even run any cuda layer now
", "votes": null, "replies": [] }, { "id": 438235, "author_name": "philippsinger", "author_url": "", "post_date": "12/13/2018 10:12:36", "content": "Cuda layers work for me, capsule doesn't.
", "votes": null, "replies": [] } ] }, { "id": 438267, "author_name": "malekbadreddine", "author_url": "", "post_date": "12/13/2018 11:47:18", "content": "still unable to run Net with capsule layer using kaggle notebook, locally everythings run perfectly
", "votes": null, "replies": [ { "id": 438287, "author_name": "philippsinger", "author_url": "", "post_date": "12/13/2018 12:20:05", "content": "(Temporary) solution is to replace K.batch_dot with tf.keras.backend.batch_dot
Tested and run smoothly, thanks for your help
", "votes": null, "replies": [] } ] }, { "id": 438493, "author_name": "stevedraper", "author_url": "", "post_date": "12/13/2018 19:43:47", "content": "I am also getting the shape mismatch from CuDNN (using PyTorch)
\n\nLooking at the docker image it looks like PyTorch was updated, but that wouldn't explain the issues people are seeing in Keras. I suspect CUDA has been updated also, but I don't see it in the latest commit to the docker image repository at https://github.com/Kaggle/docker-python
\n\n[Update] - latest notebook image has CUDA 9.0.176 and CuDNN version 7.4.01, which is later than the versions a few weeks ago (but I forget exactly what they were)
", "votes": null, "replies": [ { "id": 438991, "author_name": "stevedraper", "author_url": "", "post_date": "12/14/2018 14:25:31", "content": "Further update - it's more specific than LSTMs generally. It's actually an incompatibility between PyTorch 1.0 and some code for Drop-connect that comes from originally from a Merity paper (Salesforce) at https://github.com/salesforce/awd-lstm-lm\nI'm trying to find a work-around currently and will report back if I am able to find one.
", "votes": null, "replies": [] }, { "id": 440514, "author_name": "stevedraper", "author_url": "", "post_date": "12/17/2018 16:41:56", "content": "I hacked WeightDrop around in such a way as to make it work (I think correctly) with PyTorch 1.0. See my last post on https://github.com/salesforce/awd-lstm-lm/issues/86
", "votes": null, "replies": [] } ] }, { "id": 438544, "author_name": "", "author_url": "", "post_date": "12/13/2018 21:35:27", "content": "Old images seem not to work with Cuda. Latest image works. K.batch_dot still complains about mismatched shape.
\n\nTemp solution:
\n\n\n", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "438120": "Hi,\n\nWith the Notebook docker image, my existing notebook stops working. I got a exception when running CuDNNLSTM (OpKernel is not available) even the GPU option was On. K.batch_dot starts complaining about mismatch shape or tensor conversion issue, which worked well before (I have not changed anything).\n\nDoes anyone experience the same issues?\n\nRegards,\nDuong nhu", "438123": "Duong - I have also experienced the same thing except with capsule layers. I have not changed anything from kernel implementations.", "438139": "Yeah!there are errors in the capsule layers now!", "438154": "I cant even run any cuda layer now", "438235": "Cuda layers work for me, capsule doesn't.", "438267": "still unable to run Net with capsule layer using kaggle notebook, locally everythings run perfectly", "438287": "(Temporary) solution is to replace `K.batch_dot` with `tf.keras.backend.batch_dot`", "438300": "Tested and run smoothly, thanks for your help", "438493": "I am also getting the shape mismatch from CuDNN (using PyTorch)\n\nLooking at the docker image it looks like PyTorch was updated, but that wouldn't explain the issues people are seeing in Keras. I suspect CUDA has been updated also, but I don't see it in the latest commit to the docker image repository at https://github.com/Kaggle/docker-python\n\n[Update] - latest notebook image has CUDA 9.0.176 and CuDNN version 7.4.01, which is later than the versions a few weeks ago (but I forget exactly what they were)", "438544": "Old images seem not to work with Cuda. Latest image works. K.batch_dot still complains about mismatched shape. \n\nTemp solution: \n> **Philipp wrote**\n> \n> > (Temporary) solution is to replace `K.batch_dot` with `tf.keras.backend.batch_dot`", "438991": "Further update - it's more specific than LSTMs generally. It's actually an incompatibility between PyTorch 1.0 and some code for Drop-connect that comes from originally from a Merity paper (Salesforce) at https://github.com/salesforce/awd-lstm-lm\nI'm trying to find a work-around currently and will report back if I am able to find one.", "440514": "I hacked WeightDrop around in such a way as to make it work (I **think** correctly) with PyTorch 1.0. See my last post on https://github.com/salesforce/awd-lstm-lm/issues/86" }, "source": "meta" }Philipp wrote
\n \n\n\n(Temporary) solution is to replace
\nK.batch_dotwithtf.keras.backend.batch_dot