I'm running into problems trying to commit kernels in order to submit my results. When trying to commit my kernel, the console always dies (I have to refresh the page at some later time), and for some reason the training cell of my notebook is always terminated half-way.
\n\nMore details:
\n\nfit() call always ends halfway through the second epoch (see below)[NbConvertApp] WARNING | Timeout waiting for IOPub output).Here is a typical output of my fit cell when run in the commit:
\nTrain on 1175509 samples, validate on 130613 samples\nEpoch 1/10\n1175509/1175509 [==============================] - 355s 302us/step - loss: 0.1170 - acc: 0.9543 - val_loss: 0.1049 - val_acc: 0.9574\nEpoch 2/10\n 351744/1175509 [=======>......................] - ETA: 3:57 - loss: 0.1040 - acc: 0.9590\n
Does anyone else have this problem? Is there a time limit for single notebook cells?
", "messages": [ { "id": "418244", "postDate": "11/09/2018 14:30:57", "content": "I'm running into problems trying to commit kernels in order to submit my results. When trying to commit my kernel, the console always dies (I have to refresh the page at some later time), and for some reason the training cell of my notebook is always terminated half-way.
\n\nMore details:
\n\nfit() call always ends halfway through the second epoch (see below)[NbConvertApp] WARNING | Timeout waiting for IOPub output).Here is a typical output of my fit cell when run in the commit:
\nTrain on 1175509 samples, validate on 130613 samples\nEpoch 1/10\n1175509/1175509 [==============================] - 355s 302us/step - loss: 0.1170 - acc: 0.9543 - val_loss: 0.1049 - val_acc: 0.9574\nEpoch 2/10\n 351744/1175509 [=======>......................] - ETA: 3:57 - loss: 0.1040 - acc: 0.9590\n
Does anyone else have this problem? Is there a time limit for single notebook cells?
", "rawMarkdown": "I'm running into problems trying to commit kernels in order to submit my results. When trying to commit my kernel, the console always dies (I have to refresh the page at some later time), and for some reason the training cell of my notebook is always terminated half-way.\n\n**More details:**\n\n 1. In the editor, I can run the complete notebook without running out of memory / time or anything. It takes around 30 minutes.\n 2. Previously, I committed a smaller version of the model which ran fine and the results were accepted by Kaggle, so I'm sure my code is correct\n 3. When trying to commit, the UI is terrible. The console always stops outputting after a while and all I can do is refresh the page. I've tried waiting for hours but the UI never recovers from this by itself.\n 4. I can check the cell outputs in the final kernel preview. For some reason, my Keras `fit()` call always ends halfway through the second epoch (see below)\n 5. After canceling the training, the notebook then continues it's course and produces a final submission file.\n 6. Retrying did not help. The total run time seems to hover around 2400 seconds\n 7. The run is always marked as successful, but in the log there is a warning (`[NbConvertApp] WARNING | Timeout waiting for IOPub output`).\n\nHere is a typical output of my `fit` cell when run in the commit:\n\n```\nTrain on 1175509 samples, validate on 130613 samples\nEpoch 1/10\n1175509/1175509 [==============================] - 355s 302us/step - loss: 0.1170 - acc: 0.9543 - val_loss: 0.1049 - val_acc: 0.9574\nEpoch 2/10\n 351744/1175509 [=======>......................] - ETA: 3:57 - loss: 0.1040 - acc: 0.9590\n```\n\nDoes anyone else have this problem? Is there a time limit for single notebook cells?", "votes": null }, { "id": "418440", "postDate": "11/09/2018 22:40:23", "content": "Yes I found the same problem... stops half way through.
", "rawMarkdown": "Yes I found the same problem... stops half way through.", "votes": null }, { "id": "418911", "postDate": "11/10/2018 21:10:00", "content": "same problem here. A easy work around is to put each epoch of training in its own block.
", "rawMarkdown": "same problem here. A easy work around is to put each epoch of training in its own block.", "votes": null }, { "id": "418984", "postDate": "11/11/2018 03:01:32", "content": "I have similar problems, I committed a kernel that should take less than 1 hour, 20 hours later it is still running with no results.
", "rawMarkdown": "I have similar problems, I committed a kernel that should take less than 1 hour, 20 hours later it is still running with no results.", "votes": null }, { "id": "419639", "postDate": "11/12/2018 10:50:30", "content": "I just started a new Kernel and did not run into this problem. Then I added a ModelCheckpoint callback to be able to get the best model of the training afterwards. Suddenly, I ran into the same issue again. This may be related to the problem. It saves the model after each epoch (or well, whenever it improves). Maybe this causes the kernel to cancel it? I will try disabling it and report back.
Never mind, that was not the problem. Even when not saving the best model, some cells are canceled, some are not. Here are two more examples (in the same kernel in the same commit!)
\n\n```\nTrain on 1240815 samples, validate on 65307 samples\nEpoch 1/4\n1240815/1240815 [==============================] - 208s 167us/step - loss: 0.1391 - acc: 0.9459 - val_loss: 0.1071 - val_acc: 0.9585
\n\nF1 Score - epoch: 1 - score: 0.649526
\n\nEpoch 2/4\n1240815/1240815 [==============================] - 201s 162us/step - loss: 0.1045 - acc: 0.9588 - val_loss: 0.1055 - val_acc: 0.9587
\n\nF1 Score - epoch: 2 - score: 0.646368
\n\nEpoch 3/4\n 576512/1240815 [============>.................] - ETA: 1:46 - loss: 0.0964 - acc: 0.9616\n```
\n\n... Some other stuff ...
\n\n```\nTrain on 1240815 samples, validate on 65307 samples\nEpoch 1/4\n1240815/1240815 [==============================] - 157s 127us/step - loss: 0.1620 - acc: 0.9372 - val_loss: 0.1080 - val_acc: 0.9583
\n\nF1 Score - epoch: 1 - score: 0.650368
\n\nEpoch 2/4\n1240815/1240815 [==============================] - 152s 122us/step - loss: 0.1074 - acc: 0.9577 - val_loss: 0.1049 - val_acc: 0.9590
\n\nF1 Score - epoch: 2 - score: 0.653796
\n\nEpoch 3/4\n1240815/1240815 [==============================] - 151s 122us/step - loss: 0.0994 - acc: 0.9607 - val_loss: 0.1051 - val_acc: 0.9587
\n\nF1 Score - epoch: 3 - score: 0.649134
\n\nEpoch 4/4\n1240815/1240815 [==============================] - 151s 122us/step - loss: 0.0933 - acc: 0.9630 - val_loss: 0.1037 - val_acc: 0.9598
\n\nF1 Score - epoch: 4 - score: 0.665111 \n```
\n\nSo the first cell stops after ~500 seconds, the second one finishes even though it needs 600 seconds to complete!
", "rawMarkdown": "Never mind, that was not the problem. Even when not saving the best model, some cells are canceled, some are not. Here are two more examples (in the same kernel in the same commit!)\n\n```\nTrain on 1240815 samples, validate on 65307 samples\nEpoch 1/4\n1240815/1240815 [==============================] - 208s 167us/step - loss: 0.1391 - acc: 0.9459 - val_loss: 0.1071 - val_acc: 0.9585\n\n F1 Score - epoch: 1 - score: 0.649526 \n\nEpoch 2/4\n1240815/1240815 [==============================] - 201s 162us/step - loss: 0.1045 - acc: 0.9588 - val_loss: 0.1055 - val_acc: 0.9587\n\n F1 Score - epoch: 2 - score: 0.646368 \n\nEpoch 3/4\n 576512/1240815 [============>.................] - ETA: 1:46 - loss: 0.0964 - acc: 0.9616\n```\n\n... Some other stuff ...\n\n```\nTrain on 1240815 samples, validate on 65307 samples\nEpoch 1/4\n1240815/1240815 [==============================] - 157s 127us/step - loss: 0.1620 - acc: 0.9372 - val_loss: 0.1080 - val_acc: 0.9583\n\n F1 Score - epoch: 1 - score: 0.650368 \n\nEpoch 2/4\n1240815/1240815 [==============================] - 152s 122us/step - loss: 0.1074 - acc: 0.9577 - val_loss: 0.1049 - val_acc: 0.9590\n\n F1 Score - epoch: 2 - score: 0.653796 \n\nEpoch 3/4\n1240815/1240815 [==============================] - 151s 122us/step - loss: 0.0994 - acc: 0.9607 - val_loss: 0.1051 - val_acc: 0.9587\n\n F1 Score - epoch: 3 - score: 0.649134 \n\nEpoch 4/4\n1240815/1240815 [==============================] - 151s 122us/step - loss: 0.0933 - acc: 0.9630 - val_loss: 0.1037 - val_acc: 0.9598\n\n F1 Score - epoch: 4 - score: 0.665111 \n```\n\nSo the first cell stops after ~500 seconds, the second one finishes even though it needs 600 seconds to complete!", "votes": null }, { "id": "419837", "postDate": "11/12/2018 16:08:43", "content": "This seems to be more common than I thought. Check out this public kernel: https://www.kaggle.com/danofer/different-embeddings-with-attention-fork\nCell 13 is also canceled before completion. Cell 18 should have the same run time but runs fine.
\n\nSo it may be that this only happens for the first long-running cell and is fine for all later ones. A work-around may be to run the training twice in two separate cells and hope the second time it finishes.
", "rawMarkdown": "This seems to be more common than I thought. Check out this public kernel: https://www.kaggle.com/danofer/different-embeddings-with-attention-fork\nCell 13 is also canceled before completion. Cell 18 should have the same run time but runs fine.\n\nSo it may be that this only happens for the first long-running cell and is fine for all later ones. A work-around may be to run the training twice in two separate cells and hope the second time it finishes.", "votes": null }, { "id": "421959", "postDate": "11/15/2018 15:55:53", "content": "Hi Max,\nI ran into the same problem of commit stopping halfway. I tried your work-around, but in vain. Did you get any other alternative for this?
", "rawMarkdown": "Hi Max,\nI ran into the same problem of commit stopping halfway. I tried your work-around, but in vain. Did you get any other alternative for this?", "votes": null }, { "id": "422692", "postDate": "11/16/2018 16:32:21", "content": "Hi Aafrin, I have not found a work-around, except to split the training into several cells. I have also contacted one of the organizers, who said they will look into it, and then didn't follow up... Maybe if we all contact them and make some ruckus we will be heard :)
", "rawMarkdown": "Hi Aafrin, I have not found a work-around, except to split the training into several cells. I have also contacted one of the organizers, who said they will look into it, and then didn't follow up... Maybe if we all contact them and make some ruckus we will be heard :)", "votes": null }, { "id": "422968", "postDate": "11/17/2018 06:37:34", "content": "just convert into a script. Works perfectly.
", "rawMarkdown": "just convert into a script. Works perfectly.", "votes": null }, { "id": "428503", "postDate": "11/27/2018 11:47:39", "content": "I believe I found a reason / workaround for this: When using Keras, simply pass verbose=2 to your model.fit() call. The progress bars that get logged by default cause quite a lot of traffic, and it seems this is what causes the cell to be canceled. By setting verbose to 2, you only get one line of log per epoch of training.
after I set verbose = 0 in the model.fit(), the problem was solved.
", "rawMarkdown": "after I set verbose = 0 in the model.fit(), the problem was solved.", "votes": null } ], "comments": [ { "id": 418440, "author_name": "venkykrishna", "author_url": "", "post_date": "11/09/2018 22:40:23", "content": "Yes I found the same problem... stops half way through.
", "votes": null, "replies": [] }, { "id": 418911, "author_name": "sijunhe9248", "author_url": "", "post_date": "11/10/2018 21:10:00", "content": "same problem here. A easy work around is to put each epoch of training in its own block.
", "votes": null, "replies": [] }, { "id": 418984, "author_name": "agentili", "author_url": "", "post_date": "11/11/2018 03:01:32", "content": "I have similar problems, I committed a kernel that should take less than 1 hour, 20 hours later it is still running with no results.
", "votes": null, "replies": [] }, { "id": 419639, "author_name": "mschumacher", "author_url": "", "post_date": "11/12/2018 10:50:30", "content": "I just started a new Kernel and did not run into this problem. Then I added a ModelCheckpoint callback to be able to get the best model of the training afterwards. Suddenly, I ran into the same issue again. This may be related to the problem. It saves the model after each epoch (or well, whenever it improves). Maybe this causes the kernel to cancel it? I will try disabling it and report back.
Never mind, that was not the problem. Even when not saving the best model, some cells are canceled, some are not. Here are two more examples (in the same kernel in the same commit!)
\n\n```\nTrain on 1240815 samples, validate on 65307 samples\nEpoch 1/4\n1240815/1240815 [==============================] - 208s 167us/step - loss: 0.1391 - acc: 0.9459 - val_loss: 0.1071 - val_acc: 0.9585
\n\nF1 Score - epoch: 1 - score: 0.649526
\n\nEpoch 2/4\n1240815/1240815 [==============================] - 201s 162us/step - loss: 0.1045 - acc: 0.9588 - val_loss: 0.1055 - val_acc: 0.9587
\n\nF1 Score - epoch: 2 - score: 0.646368
\n\nEpoch 3/4\n 576512/1240815 [============>.................] - ETA: 1:46 - loss: 0.0964 - acc: 0.9616\n```
\n\n... Some other stuff ...
\n\n```\nTrain on 1240815 samples, validate on 65307 samples\nEpoch 1/4\n1240815/1240815 [==============================] - 157s 127us/step - loss: 0.1620 - acc: 0.9372 - val_loss: 0.1080 - val_acc: 0.9583
\n\nF1 Score - epoch: 1 - score: 0.650368
\n\nEpoch 2/4\n1240815/1240815 [==============================] - 152s 122us/step - loss: 0.1074 - acc: 0.9577 - val_loss: 0.1049 - val_acc: 0.9590
\n\nF1 Score - epoch: 2 - score: 0.653796
\n\nEpoch 3/4\n1240815/1240815 [==============================] - 151s 122us/step - loss: 0.0994 - acc: 0.9607 - val_loss: 0.1051 - val_acc: 0.9587
\n\nF1 Score - epoch: 3 - score: 0.649134
\n\nEpoch 4/4\n1240815/1240815 [==============================] - 151s 122us/step - loss: 0.0933 - acc: 0.9630 - val_loss: 0.1037 - val_acc: 0.9598
\n\nF1 Score - epoch: 4 - score: 0.665111 \n```
\n\nSo the first cell stops after ~500 seconds, the second one finishes even though it needs 600 seconds to complete!
", "votes": null, "replies": [] }, { "id": 419837, "author_name": "mschumacher", "author_url": "", "post_date": "11/12/2018 16:08:43", "content": "This seems to be more common than I thought. Check out this public kernel: https://www.kaggle.com/danofer/different-embeddings-with-attention-fork\nCell 13 is also canceled before completion. Cell 18 should have the same run time but runs fine.
\n\nSo it may be that this only happens for the first long-running cell and is fine for all later ones. A work-around may be to run the training twice in two separate cells and hope the second time it finishes.
", "votes": null, "replies": [] }, { "id": 421959, "author_name": "aafrin", "author_url": "", "post_date": "11/15/2018 15:55:53", "content": "Hi Max,\nI ran into the same problem of commit stopping halfway. I tried your work-around, but in vain. Did you get any other alternative for this?
", "votes": null, "replies": [] }, { "id": 422692, "author_name": "mschumacher", "author_url": "", "post_date": "11/16/2018 16:32:21", "content": "Hi Aafrin, I have not found a work-around, except to split the training into several cells. I have also contacted one of the organizers, who said they will look into it, and then didn't follow up... Maybe if we all contact them and make some ruckus we will be heard :)
", "votes": null, "replies": [] }, { "id": 422968, "author_name": "sijunhe9248", "author_url": "", "post_date": "11/17/2018 06:37:34", "content": "just convert into a script. Works perfectly.
", "votes": null, "replies": [] } ] }, { "id": 428503, "author_name": "mschumacher", "author_url": "", "post_date": "11/27/2018 11:47:39", "content": "I believe I found a reason / workaround for this: When using Keras, simply pass verbose=2 to your model.fit() call. The progress bars that get logged by default cause quite a lot of traffic, and it seems this is what causes the cell to be canceled. By setting verbose to 2, you only get one line of log per epoch of training.
after I set verbose = 0 in the model.fit(), the problem was solved.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "418244": "I'm running into problems trying to commit kernels in order to submit my results. When trying to commit my kernel, the console always dies (I have to refresh the page at some later time), and for some reason the training cell of my notebook is always terminated half-way.\n\n**More details:**\n\n 1. In the editor, I can run the complete notebook without running out of memory / time or anything. It takes around 30 minutes.\n 2. Previously, I committed a smaller version of the model which ran fine and the results were accepted by Kaggle, so I'm sure my code is correct\n 3. When trying to commit, the UI is terrible. The console always stops outputting after a while and all I can do is refresh the page. I've tried waiting for hours but the UI never recovers from this by itself.\n 4. I can check the cell outputs in the final kernel preview. For some reason, my Keras `fit()` call always ends halfway through the second epoch (see below)\n 5. After canceling the training, the notebook then continues it's course and produces a final submission file.\n 6. Retrying did not help. The total run time seems to hover around 2400 seconds\n 7. The run is always marked as successful, but in the log there is a warning (`[NbConvertApp] WARNING | Timeout waiting for IOPub output`).\n\nHere is a typical output of my `fit` cell when run in the commit:\n\n```\nTrain on 1175509 samples, validate on 130613 samples\nEpoch 1/10\n1175509/1175509 [==============================] - 355s 302us/step - loss: 0.1170 - acc: 0.9543 - val_loss: 0.1049 - val_acc: 0.9574\nEpoch 2/10\n 351744/1175509 [=======>......................] - ETA: 3:57 - loss: 0.1040 - acc: 0.9590\n```\n\nDoes anyone else have this problem? Is there a time limit for single notebook cells?", "418440": "Yes I found the same problem... stops half way through.", "418911": "same problem here. A easy work around is to put each epoch of training in its own block.", "418984": "I have similar problems, I committed a kernel that should take less than 1 hour, 20 hours later it is still running with no results.", "419639": "I just started a new Kernel and did not run into this problem. Then I added a `ModelCheckpoint` callback to be able to get the best model of the training afterwards. Suddenly, I ran into the same issue again. This may be related to the problem. It saves the model after each epoch (or well, whenever it improves). Maybe this causes the kernel to cancel it? I will try disabling it and report back.", "419743": "Never mind, that was not the problem. Even when not saving the best model, some cells are canceled, some are not. Here are two more examples (in the same kernel in the same commit!)\n\n```\nTrain on 1240815 samples, validate on 65307 samples\nEpoch 1/4\n1240815/1240815 [==============================] - 208s 167us/step - loss: 0.1391 - acc: 0.9459 - val_loss: 0.1071 - val_acc: 0.9585\n\n F1 Score - epoch: 1 - score: 0.649526 \n\nEpoch 2/4\n1240815/1240815 [==============================] - 201s 162us/step - loss: 0.1045 - acc: 0.9588 - val_loss: 0.1055 - val_acc: 0.9587\n\n F1 Score - epoch: 2 - score: 0.646368 \n\nEpoch 3/4\n 576512/1240815 [============>.................] - ETA: 1:46 - loss: 0.0964 - acc: 0.9616\n```\n\n... Some other stuff ...\n\n```\nTrain on 1240815 samples, validate on 65307 samples\nEpoch 1/4\n1240815/1240815 [==============================] - 157s 127us/step - loss: 0.1620 - acc: 0.9372 - val_loss: 0.1080 - val_acc: 0.9583\n\n F1 Score - epoch: 1 - score: 0.650368 \n\nEpoch 2/4\n1240815/1240815 [==============================] - 152s 122us/step - loss: 0.1074 - acc: 0.9577 - val_loss: 0.1049 - val_acc: 0.9590\n\n F1 Score - epoch: 2 - score: 0.653796 \n\nEpoch 3/4\n1240815/1240815 [==============================] - 151s 122us/step - loss: 0.0994 - acc: 0.9607 - val_loss: 0.1051 - val_acc: 0.9587\n\n F1 Score - epoch: 3 - score: 0.649134 \n\nEpoch 4/4\n1240815/1240815 [==============================] - 151s 122us/step - loss: 0.0933 - acc: 0.9630 - val_loss: 0.1037 - val_acc: 0.9598\n\n F1 Score - epoch: 4 - score: 0.665111 \n```\n\nSo the first cell stops after ~500 seconds, the second one finishes even though it needs 600 seconds to complete!", "419837": "This seems to be more common than I thought. Check out this public kernel: https://www.kaggle.com/danofer/different-embeddings-with-attention-fork\nCell 13 is also canceled before completion. Cell 18 should have the same run time but runs fine.\n\nSo it may be that this only happens for the first long-running cell and is fine for all later ones. A work-around may be to run the training twice in two separate cells and hope the second time it finishes.", "421959": "Hi Max,\nI ran into the same problem of commit stopping halfway. I tried your work-around, but in vain. Did you get any other alternative for this?", "422692": "Hi Aafrin, I have not found a work-around, except to split the training into several cells. I have also contacted one of the organizers, who said they will look into it, and then didn't follow up... Maybe if we all contact them and make some ruckus we will be heard :)", "422968": "just convert into a script. Works perfectly.", "428503": "I believe I found a reason / workaround for this: When using Keras, simply pass `verbose=2` to your `model.fit()` call. The progress bars that get logged by default cause quite a lot of traffic, and it seems this is what causes the cell to be canceled. By setting verbose to 2, you only get one line of log per epoch of training.", "454130": "after I set verbose = 0 in the model.fit(), the problem was solved." }, "source": "meta" }