I trained a CNN with Keras in my kernel. After 10 epochs, it usually reaches an accuracy of 90% and AUC around 0.95 on the validation data. However, when I commit my code, the model almost always gets stuck from the very beginning. The loss never decreases and the training accuracy stays at 50%. But it performs normally when I run it within the kernel instead of committing it. What might be wrong? Thanks a lot!
", "messages": [ { "id": "462969", "postDate": "01/29/2019 06:53:54", "content": "I trained a CNN with Keras in my kernel. After 10 epochs, it usually reaches an accuracy of 90% and AUC around 0.95 on the validation data. However, when I commit my code, the model almost always gets stuck from the very beginning. The loss never decreases and the training accuracy stays at 50%. But it performs normally when I run it within the kernel instead of committing it. What might be wrong? Thanks a lot!
", "rawMarkdown": "I trained a CNN with Keras in my kernel. After 10 epochs, it usually reaches an accuracy of 90% and AUC around 0.95 on the validation data. However, when I commit my code, the model almost always gets stuck from the very beginning. The loss never decreases and the training accuracy stays at 50%. But it performs normally when I run it within the kernel instead of committing it. What might be wrong? Thanks a lot!", "votes": null }, { "id": "463006", "postDate": "01/29/2019 08:23:15", "content": "What means \"commit your code\"?
", "rawMarkdown": "What means \"commit your code\"?", "votes": null }, { "id": "464972", "postDate": "02/01/2019 23:01:50", "content": "He means clicking commit on the kaggle kernel. I have the same problem, my commits never finish, they just abort in random place.
", "rawMarkdown": "He means clicking commit on the kaggle kernel. I have the same problem, my commits never finish, they just abort in random place.", "votes": null }, { "id": "466712", "postDate": "02/05/2019 21:01:50", "content": "My guess: use index=False when saving the csv
", "rawMarkdown": "My guess: use index=False when saving the csv", "votes": null }, { "id": "475644", "postDate": "02/21/2019 03:06:36", "content": "Mine have been hitting the time limit recently. They were working great a few weeks ago and then, without me changing anything, stopped working. I thought it had to do with a corrupted image file. In my process of trying to isolate any potential problem file I realized its the number of images I train on. Doesn't really help me figure out what's going on though. One day my kernel runs no problems, the next it loses a worker while training and hangs until it times out unless I train a smaller set of data.
", "rawMarkdown": "Mine have been hitting the time limit recently. They were working great a few weeks ago and then, without me changing anything, stopped working. I thought it had to do with a corrupted image file. In my process of trying to isolate any potential problem file I realized its the number of images I train on. Doesn't really help me figure out what's going on though. One day my kernel runs no problems, the next it loses a worker while training and hangs until it times out unless I train a smaller set of data.", "votes": null }, { "id": "581601", "postDate": "07/22/2019 06:38:22", "content": "Hey @kaimingk , Did you find any solution to this problem? I too am facing the very same problem now, when I train my model in the edit option, the model converges, while in case of commiting, my val_acc gets stuck to only one value while loss becomes 'nan'. Please do tell me if there is any solution for this problem. Thanks in advance.
", "rawMarkdown": "Hey @kaimingk , Did you find any solution to this problem? I too am facing the very same problem now, when I train my model in the edit option, the model converges, while in case of commiting, my val_acc gets stuck to only one value while loss becomes 'nan'. Please do tell me if there is any solution for this problem. Thanks in advance.", "votes": null }, { "id": "585143", "postDate": "07/27/2019 02:49:22", "content": "Hi Gajendra. Sorry it's been such a long time and I can't remember exactly how I fixed it. I recall that I changed the model architecture though. I think maybe you could commit the code with only a few iterations and output the validation result and check it against the true label. If you wanna check my code, here is the link: Tumor Detection with Keras CNN
", "rawMarkdown": "Hi Gajendra. Sorry it's been such a long time and I can't remember exactly how I fixed it. I recall that I changed the model architecture though. I think maybe you could commit the code with only a few iterations and output the validation result and check it against the true label. If you wanna check my code, here is the link: [Tumor Detection with Keras CNN](https://www.kaggle.com/kaimingk/tumor-detection-with-keras-cnn)", "votes": null } ], "comments": [ { "id": 463006, "author_name": "cqcqxq", "author_url": "", "post_date": "01/29/2019 08:23:15", "content": "What means \"commit your code\"?
", "votes": null, "replies": [] }, { "id": 464972, "author_name": "sq5rix", "author_url": "", "post_date": "02/01/2019 23:01:50", "content": "He means clicking commit on the kaggle kernel. I have the same problem, my commits never finish, they just abort in random place.
", "votes": null, "replies": [ { "id": 475644, "author_name": "reidtc", "author_url": "", "post_date": "02/21/2019 03:06:36", "content": "Mine have been hitting the time limit recently. They were working great a few weeks ago and then, without me changing anything, stopped working. I thought it had to do with a corrupted image file. In my process of trying to isolate any potential problem file I realized its the number of images I train on. Doesn't really help me figure out what's going on though. One day my kernel runs no problems, the next it loses a worker while training and hangs until it times out unless I train a smaller set of data.
", "votes": null, "replies": [] } ] }, { "id": 466712, "author_name": "wappyotoole", "author_url": "", "post_date": "02/05/2019 21:01:50", "content": "My guess: use index=False when saving the csv
", "votes": null, "replies": [] }, { "id": 581601, "author_name": "sarques", "author_url": "", "post_date": "07/22/2019 06:38:22", "content": "Hey @kaimingk , Did you find any solution to this problem? I too am facing the very same problem now, when I train my model in the edit option, the model converges, while in case of commiting, my val_acc gets stuck to only one value while loss becomes 'nan'. Please do tell me if there is any solution for this problem. Thanks in advance.
", "votes": null, "replies": [ { "id": 585143, "author_name": "kaimingk", "author_url": "", "post_date": "07/27/2019 02:49:22", "content": "Hi Gajendra. Sorry it's been such a long time and I can't remember exactly how I fixed it. I recall that I changed the model architecture though. I think maybe you could commit the code with only a few iterations and output the validation result and check it against the true label. If you wanna check my code, here is the link: Tumor Detection with Keras CNN
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "462969": "I trained a CNN with Keras in my kernel. After 10 epochs, it usually reaches an accuracy of 90% and AUC around 0.95 on the validation data. However, when I commit my code, the model almost always gets stuck from the very beginning. The loss never decreases and the training accuracy stays at 50%. But it performs normally when I run it within the kernel instead of committing it. What might be wrong? Thanks a lot!", "463006": "What means \"commit your code\"?", "464972": "He means clicking commit on the kaggle kernel. I have the same problem, my commits never finish, they just abort in random place.", "466712": "My guess: use index=False when saving the csv", "475644": "Mine have been hitting the time limit recently. They were working great a few weeks ago and then, without me changing anything, stopped working. I thought it had to do with a corrupted image file. In my process of trying to isolate any potential problem file I realized its the number of images I train on. Doesn't really help me figure out what's going on though. One day my kernel runs no problems, the next it loses a worker while training and hangs until it times out unless I train a smaller set of data.", "581601": "Hey @kaimingk , Did you find any solution to this problem? I too am facing the very same problem now, when I train my model in the edit option, the model converges, while in case of commiting, my val_acc gets stuck to only one value while loss becomes 'nan'. Please do tell me if there is any solution for this problem. Thanks in advance.", "585143": "Hi Gajendra. Sorry it's been such a long time and I can't remember exactly how I fixed it. I recall that I changed the model architecture though. I think maybe you could commit the code with only a few iterations and output the validation result and check it against the true label. If you wanna check my code, here is the link: [Tumor Detection with Keras CNN](https://www.kaggle.com/kaimingk/tumor-detection-with-keras-cnn)" }, "source": "meta" }