I can commit and the run goes fine but whenever I submit to the competition. It's always Notebook Threw Exception. Anyone else running into this issue? I'm using a custom tf.keras.model
", "messages": [ { "id": "712302", "postDate": "01/07/2020 03:54:03", "content": "I can commit and the run goes fine but whenever I submit to the competition. It's always Notebook Threw Exception. Anyone else running into this issue? I'm using a custom tf.keras.model
", "rawMarkdown": "I can commit and the run goes fine but whenever I submit to the competition. It's always Notebook Threw Exception. Anyone else running into this issue? I'm using a custom tf.keras.model", "votes": null }, { "id": "712367", "postDate": "01/07/2020 06:28:11", "content": "I don't think the error is due to tf.keras.model. Could you post your kernel(if possible) so that we can take a look and help you?
See the following discusions:
\n\nAs you will read in the posts, the best you can do is to make an inference on the train set images and check if agrees with the time/memory limits.
", "rawMarkdown": "See the following discusions:\n\n- https://www.kaggle.com/c/bengaliai-cv19/discussion/123632\n\n- https://www.kaggle.com/c/bengaliai-cv19/discussion/123212\n\n- https://www.kaggle.com/c/bengaliai-cv19/discussion/123659\n\nAs you will read in the posts, the best you can do is to make an inference on the train set images and check if agrees with the time/memory limits.", "votes": null }, { "id": "712769", "postDate": "01/07/2020 15:09:46", "content": "@bibek777 Here is my kernel. https://www.kaggle.com/shujun717/rebuilt-no-bn. Much appreciated!
", "rawMarkdown": "bibek777 Here is my kernel. https://www.kaggle.com/shujun717/rebuilt-no-bn. Much appreciated!", "votes": null }, { "id": "712819", "postDate": "01/07/2020 16:00:50", "content": "@bibek777 Here is my kernel. https://www.kaggle.com/shujun717/rebuilt-no-bn. Much appreciated!
", "rawMarkdown": "bibek777 Here is my kernel. https://www.kaggle.com/shujun717/rebuilt-no-bn. Much appreciated!", "votes": null }, { "id": "712905", "postDate": "01/07/2020 17:55:13", "content": "I looked at your kernel; looks fine to me. Don't know why itz giving the error. I will let you know if I find something else.
", "rawMarkdown": "I looked at your kernel; looks fine to me. Don't know why itz giving the error. I will let you know if I find something else.", "votes": null }, { "id": "712954", "postDate": "01/07/2020 18:55:19", "content": "Yeah it runs fine when commmited in the kernel but when submitted, something goes wrong. Thx!
", "rawMarkdown": "Yeah it runs fine when commmited in the kernel but when submitted, something goes wrong. Thx!", "votes": null }, { "id": "712959", "postDate": "01/07/2020 19:02:48", "content": "Are you sure about the submission file layout format? grapheme - vowel - consonant or consonant - grapheme - vowel?
", "rawMarkdown": "Are you sure about the submission file layout format? grapheme - vowel - consonant or consonant - grapheme - vowel?", "votes": null }, { "id": "712992", "postDate": "01/07/2020 19:54:21", "content": "actually i think the order in my code is wrong; however, one of my previous tensorflow models worked with the wrong order and i did get a reasonable score of ~ 0.93. Moreover, the notebook only runs for 4 minutes before it crashes when submitted so it did not even get to the submission part yet
", "rawMarkdown": "actually i think the order in my code is wrong; however, one of my previous tensorflow models worked with the wrong order and i did get a reasonable score of ~ 0.93. Moreover, the notebook only runs for 4 minutes before it crashes when submitted so it did not even get to the submission part yet", "votes": null }, { "id": "712994", "postDate": "01/07/2020 19:55:06", "content": "I tried doing inference with the training set. It worked in the kernels when committed
", "rawMarkdown": "I tried doing inference with the training set. It worked in the kernels when committed", "votes": null }, { "id": "713216", "postDate": "01/08/2020 03:05:18", "content": "Hi shujun,
\n\nAre you loading model weights also? I dint see that in your code. After loading the model you also have to load it's weights may be that's the issue for you..
", "rawMarkdown": "Hi shujun,\n\nAre you loading model weights also? I dint see that in your code. After loading the model you also have to load it's weights may be that's the issue for you..", "votes": null }, { "id": "714132", "postDate": "01/09/2020 05:13:38", "content": "The dummy test data that is replaced with actual test data is quite large . keeping that in mind are you not loading 10s of thousands of images in a large array ?
\n\nimages=np.asarray(images,dtype='float16')
If you want to debug this , do one thing , run the same code on the train set (which is quite large) and not on the dummy test set (like how you are doing now) . You will find the exact memory error during commit itself .
", "rawMarkdown": "The dummy test data that is replaced with actual test data is quite large . keeping that in mind are you not loading 10s of thousands of images in a large array ?\n\n`images=np.asarray(images,dtype='float16')`\n\nIf you want to debug this , do one thing , run the same code on the train set (which is quite large) and not on the dummy test set (like how you are doing now) . You will find the exact memory error during commit itself .", "votes": null }, { "id": "714784", "postDate": "01/09/2020 18:31:31", "content": "I didn't think that was the issue since that worked with one of my tf models with the training set. But I think you are right, since I tried getting batches of images at a time and it worked. Maybe the submission gets a little less memory than the notebook, because images=np.asarray(images,dtype='float16')\nworked with the training set in the notebook?
", "rawMarkdown": "I didn't think that was the issue since that worked with one of my tf models with the training set. But I think you are right, since I tried getting batches of images at a time and it worked. Maybe the submission gets a little less memory than the notebook, because images=np.asarray(images,dtype='float16')\nworked with the training set in the notebook?", "votes": null } ], "comments": [ { "id": 712367, "author_name": "bibek777", "author_url": "", "post_date": "01/07/2020 06:28:11", "content": "I don't think the error is due to tf.keras.model. Could you post your kernel(if possible) so that we can take a look and help you?
@bibek777 Here is my kernel. https://www.kaggle.com/shujun717/rebuilt-no-bn. Much appreciated!
", "votes": null, "replies": [] }, { "id": 712905, "author_name": "bibek777", "author_url": "", "post_date": "01/07/2020 17:55:13", "content": "I looked at your kernel; looks fine to me. Don't know why itz giving the error. I will let you know if I find something else.
", "votes": null, "replies": [] }, { "id": 712954, "author_name": "shujun717", "author_url": "", "post_date": "01/07/2020 18:55:19", "content": "Yeah it runs fine when commmited in the kernel but when submitted, something goes wrong. Thx!
", "votes": null, "replies": [] } ] }, { "id": 712541, "author_name": "imeintanis", "author_url": "", "post_date": "01/07/2020 11:02:02", "content": "See the following discusions:
\n\nAs you will read in the posts, the best you can do is to make an inference on the train set images and check if agrees with the time/memory limits.
", "votes": null, "replies": [ { "id": 712994, "author_name": "shujun717", "author_url": "", "post_date": "01/07/2020 19:55:06", "content": "I tried doing inference with the training set. It worked in the kernels when committed
", "votes": null, "replies": [] } ] }, { "id": 712769, "author_name": "shujun717", "author_url": "", "post_date": "01/07/2020 15:09:46", "content": "@bibek777 Here is my kernel. https://www.kaggle.com/shujun717/rebuilt-no-bn. Much appreciated!
", "votes": null, "replies": [ { "id": 712959, "author_name": "ipythonx", "author_url": "", "post_date": "01/07/2020 19:02:48", "content": "Are you sure about the submission file layout format? grapheme - vowel - consonant or consonant - grapheme - vowel?
", "votes": null, "replies": [] }, { "id": 712992, "author_name": "shujun717", "author_url": "", "post_date": "01/07/2020 19:54:21", "content": "actually i think the order in my code is wrong; however, one of my previous tensorflow models worked with the wrong order and i did get a reasonable score of ~ 0.93. Moreover, the notebook only runs for 4 minutes before it crashes when submitted so it did not even get to the submission part yet
", "votes": null, "replies": [] } ] }, { "id": 713216, "author_name": "amit9484", "author_url": "", "post_date": "01/08/2020 03:05:18", "content": "Hi shujun,
\n\nAre you loading model weights also? I dint see that in your code. After loading the model you also have to load it's weights may be that's the issue for you..
", "votes": null, "replies": [] }, { "id": 714132, "author_name": "phoenix9032", "author_url": "", "post_date": "01/09/2020 05:13:38", "content": "The dummy test data that is replaced with actual test data is quite large . keeping that in mind are you not loading 10s of thousands of images in a large array ?
\n\nimages=np.asarray(images,dtype='float16')
If you want to debug this , do one thing , run the same code on the train set (which is quite large) and not on the dummy test set (like how you are doing now) . You will find the exact memory error during commit itself .
", "votes": null, "replies": [ { "id": 714784, "author_name": "shujun717", "author_url": "", "post_date": "01/09/2020 18:31:31", "content": "I didn't think that was the issue since that worked with one of my tf models with the training set. But I think you are right, since I tried getting batches of images at a time and it worked. Maybe the submission gets a little less memory than the notebook, because images=np.asarray(images,dtype='float16')\nworked with the training set in the notebook?
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "712302": "I can commit and the run goes fine but whenever I submit to the competition. It's always Notebook Threw Exception. Anyone else running into this issue? I'm using a custom tf.keras.model", "712367": "I don't think the error is due to `tf.keras.model`. Could you post your kernel(if possible) so that we can take a look and help you?", "712541": "See the following discusions:\n\n- https://www.kaggle.com/c/bengaliai-cv19/discussion/123632\n\n- https://www.kaggle.com/c/bengaliai-cv19/discussion/123212\n\n- https://www.kaggle.com/c/bengaliai-cv19/discussion/123659\n\nAs you will read in the posts, the best you can do is to make an inference on the train set images and check if agrees with the time/memory limits.", "712769": "bibek777 Here is my kernel. https://www.kaggle.com/shujun717/rebuilt-no-bn. Much appreciated!", "712819": "bibek777 Here is my kernel. https://www.kaggle.com/shujun717/rebuilt-no-bn. Much appreciated!", "712905": "I looked at your kernel; looks fine to me. Don't know why itz giving the error. I will let you know if I find something else.", "712954": "Yeah it runs fine when commmited in the kernel but when submitted, something goes wrong. Thx!", "712959": "Are you sure about the submission file layout format? grapheme - vowel - consonant or consonant - grapheme - vowel?", "712992": "actually i think the order in my code is wrong; however, one of my previous tensorflow models worked with the wrong order and i did get a reasonable score of ~ 0.93. Moreover, the notebook only runs for 4 minutes before it crashes when submitted so it did not even get to the submission part yet", "712994": "I tried doing inference with the training set. It worked in the kernels when committed", "713216": "Hi shujun,\n\nAre you loading model weights also? I dint see that in your code. After loading the model you also have to load it's weights may be that's the issue for you..", "714132": "The dummy test data that is replaced with actual test data is quite large . keeping that in mind are you not loading 10s of thousands of images in a large array ?\n\n`images=np.asarray(images,dtype='float16')`\n\nIf you want to debug this , do one thing , run the same code on the train set (which is quite large) and not on the dummy test set (like how you are doing now) . You will find the exact memory error during commit itself .", "714784": "I didn't think that was the issue since that worked with one of my tf models with the training set. But I think you are right, since I tried getting batches of images at a time and it worked. Maybe the submission gets a little less memory than the notebook, because images=np.asarray(images,dtype='float16')\nworked with the training set in the notebook?" }, "source": "meta" }