{ "id": 218870, "title": "Beginner having problems completing training", "url": "/competitions/cassava-leaf-disease-classification/discussion/218870", "author_name": "", "post_date": "2021-02-12T11:48:54.143544700Z", "votes": 6, "comment_count": 16, "views": 0, "content": "

Hello everyone, this is my first competition and I can't train for more than 4 epochs because, after that, my kernel asks me if I am still here and if I want to keep editing then resets. That causes all my saved weights to disappear. I don't see how I can get good results in 4 epochs. Thank you for your help.

\n

https://www.kaggle.com/charlesrongione/fork-of-cassava

\n

btw feedbacks about the content of my work are also welcome :)

", "messages": [ { "id": "1197757", "postDate": "02/12/2021 11:48:54", "content": "

Hello everyone, this is my first competition and I can't train for more than 4 epochs because, after that, my kernel asks me if I am still here and if I want to keep editing then resets. That causes all my saved weights to disappear. I don't see how I can get good results in 4 epochs. Thank you for your help.

\n

https://www.kaggle.com/charlesrongione/fork-of-cassava

\n

btw feedbacks about the content of my work are also welcome :)

", "rawMarkdown": "Hello everyone, this is my first competition and I can't train for more than 4 epochs because, after that, my kernel asks me if I am still here and if I want to keep editing then resets. That causes all my saved weights to disappear. I don't see how I can get good results in 4 epochs. Thank you for your help. \n\nhttps://www.kaggle.com/charlesrongione/fork-of-cassava\n\nbtw feedbacks about the content of my work are also welcome :)", "votes": null }, { "id": "1197820", "postDate": "02/12/2021 13:14:44", "content": "

The interactive session is active for just 40 minutes, if you are not active, your notebook is turned off. You need to save and run all your notebook for full training
\n\"\"

", "rawMarkdown": "The interactive session is active for just 40 minutes, if you are not active, your notebook is turned off. You need to `save and run all` your notebook for full training\n![](https://i.ibb.co/ZdyRPfR/Screen-Shot-2021-02-12-at-14-15-40.png)", "votes": null }, { "id": "1197824", "postDate": "02/12/2021 13:18:45", "content": "

@amiiiney Thank you for your answer. I am active. I never left my computer and I check the training every 5 minutes

\n

Plus, maybe I am wrong, but this method does not allow me to carefully watch my training and stop it to readjust some parameters then resume it. Neither to watch my progress bar, which I find very frustrating because I have no idea which batch is my network processing

", "rawMarkdown": "amiiiney Thank you for your answer. I am active. I never left my computer and I check the training every 5 minutes\n\nPlus, maybe I am wrong, but this method does not allow me to carefully watch my training and stop it to readjust some parameters then resume it. Neither to watch my progress bar, which I find very frustrating because I have no idea which batch is my network processing", "votes": null }, { "id": "1197833", "postDate": "02/12/2021 13:25:24", "content": "

I mean it's a better practice to just click on Save & Run All because then your notebook is running in the cloud and it won't stop training unless it exceeds 9 hours. You can just close the tab and check the notebook when the training is done.
\nThe way you are doing it requires you to be active by modifying/executing some cell every 40 minutes. Just being active without any activity in the notebook is not considered active :)

", "rawMarkdown": "I mean it's a better practice to just click on `Save & Run All` because then your notebook is running in the cloud and it won't stop training unless it exceeds 9 hours. You can just close the tab and check the notebook when the training is done.\nThe way you are doing it requires you to be active by modifying/executing some cell every 40 minutes. Just being active without any activity in the notebook is not considered active :)", "votes": null }, { "id": "1197855", "postDate": "02/12/2021 13:40:57", "content": "

@amiiiney I understand, thank you. From what I understand I have two options, please correct me if I am wrong :

\n\n

Does that make sense?

", "rawMarkdown": "amiiiney I understand, thank you. From what I understand I have two options, please correct me if I am wrong : \n\n - Either I run it on the cloud for maximum nine hours like you said but I am not able to watch how my model is doing, I have no idea what it is doing at time t\n\n - Or I just set each training process for 3 epoch (each one is taking 12 minutes) and rerun it from the previous process. But that way I can see my tqdm progress bar going and have more flexibility on my training progress.\n\nDoes that make sense?", "votes": null }, { "id": "1197864", "postDate": "02/12/2021 13:46:48", "content": "

Exactly! Those are your options! Maybe there is another option using tensorboard to visualize your training metrics while committing (I haven't used it before).
\nEDIT: The second option with setting an alarm every 35 minutes sounded more logical, training just 3 epochs will require you to load the weights every time you retrain to resume training.

", "rawMarkdown": "Exactly! Those are your options! Maybe there is another option using tensorboard to visualize your training metrics while committing (I haven't used it before).\nEDIT: The second option with setting an alarm every 35 minutes sounded more logical, training just 3 epochs will require you to load the weights every time you retrain to resume training.", "votes": null }, { "id": "1197990", "postDate": "02/12/2021 15:42:19", "content": "

So when I do Save & Run All can I close the tab? It will get the training done in the cloud? Oh, I never knew it thanks for sharing.

", "rawMarkdown": "So when I do Save & Run All can I close the tab? It will get the training done in the cloud? Oh, I never knew it thanks for sharing.", "votes": null }, { "id": "1198009", "postDate": "02/12/2021 16:09:37", "content": "

Yes! When you click on Save & Run All you can just turn off the interactive session and close everything!

", "rawMarkdown": "Yes! When you click on `Save & Run All` you can just turn off the interactive session and close everything!", "votes": null }, { "id": "1198021", "postDate": "02/12/2021 16:21:05", "content": "

Ok thanks

", "rawMarkdown": "Ok thanks", "votes": null }, { "id": "1198208", "postDate": "02/12/2021 19:45:36", "content": "

I created a link for this very purpose here

", "rawMarkdown": "I created a link for this very purpose [here](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/216461)", "votes": null }, { "id": "1198227", "postDate": "02/12/2021 19:54:51", "content": "

thank you :)

", "rawMarkdown": "thank you :)", "votes": null }, { "id": "1198254", "postDate": "02/12/2021 20:36:24", "content": "

you can start by folding your data on smaller data just so you can see how it trains

", "rawMarkdown": "you can start by folding your data on smaller data just so you can see how it trains", "votes": null }, { "id": "1198256", "postDate": "02/12/2021 20:44:36", "content": "

Hello @alincijov, Tank you for your concern. Do you mean to do a full training on a small sample of the dataset? That's what I did first but as expected I got nothing more than overfitting

", "rawMarkdown": "Hello @alincijov, Tank you for your concern. Do you mean to do a full training on a small sample of the dataset? That's what I did first but as expected I got nothing more than overfitting", "votes": null }, { "id": "1199132", "postDate": "02/13/2021 14:39:14", "content": "

Well @charlesrongione you can save the model after 3 epochs every time and train the new model using the weights of the save model. Its hard to do it every time but that's the solution I found other than using 'Save Version' option on Kaggle.
\nTo save the model you could just use model.save('file_name.h5') and use the weights of the model using
\nnew_model = tf.keras.load_model('path to h5 file') and run the new model again and yes it does works and the advantage of this on using 'Save Version' is that you can keep track of accuracy of model and stop or change the parameters in between.
\nHope it helps: )

", "rawMarkdown": "Well @charlesrongione you can save the model after 3 epochs every time and train the new model using the weights of the save model. Its hard to do it every time but that's the solution I found other than using **'Save Version'** option on Kaggle.\nTo save the model you could just use **model.save('file_name.h5')** and use the weights of the model using \n**new_model = tf.keras.load_model('path to h5 file')** and run the new model again and yes it does works and the advantage of this on using **'Save Version'** is that you can keep track of accuracy of model and stop or change the parameters in between.\nHope it helps: )", "votes": null }, { "id": "1199257", "postDate": "02/13/2021 16:54:20", "content": "

Thank you, I am on PyTorch so I did something slightly different but I get your point. Thank you for your help.

", "rawMarkdown": "Thank you, I am on PyTorch so I did something slightly different but I get your point. Thank you for your help.", "votes": null }, { "id": "1199883", "postDate": "02/14/2021 07:48:56", "content": "

Have you tried to use Dataloaders ?

", "rawMarkdown": "Have you tried to use Dataloaders ?", "votes": null }, { "id": "1200705", "postDate": "02/14/2021 22:23:54", "content": "

@alincijov I am sorry I don't understand the question? Yes of course I use DataLoaders. Am I missing something non-trivial here?

", "rawMarkdown": "alincijov I am sorry I don't understand the question? Yes of course I use DataLoaders. Am I missing something non-trivial here?", "votes": null } ], "comments": [ { "id": 1197820, "author_name": "amiiiney", "author_url": "", "post_date": "02/12/2021 13:14:44", "content": "

The interactive session is active for just 40 minutes, if you are not active, your notebook is turned off. You need to save and run all your notebook for full training
\n\"\"

", "votes": null, "replies": [ { "id": 1197824, "author_name": "charlesrongione", "author_url": "", "post_date": "02/12/2021 13:18:45", "content": "

@amiiiney Thank you for your answer. I am active. I never left my computer and I check the training every 5 minutes

\n

Plus, maybe I am wrong, but this method does not allow me to carefully watch my training and stop it to readjust some parameters then resume it. Neither to watch my progress bar, which I find very frustrating because I have no idea which batch is my network processing

", "votes": null, "replies": [] }, { "id": 1197833, "author_name": "amiiiney", "author_url": "", "post_date": "02/12/2021 13:25:24", "content": "

I mean it's a better practice to just click on Save & Run All because then your notebook is running in the cloud and it won't stop training unless it exceeds 9 hours. You can just close the tab and check the notebook when the training is done.
\nThe way you are doing it requires you to be active by modifying/executing some cell every 40 minutes. Just being active without any activity in the notebook is not considered active :)

", "votes": null, "replies": [] }, { "id": 1197855, "author_name": "charlesrongione", "author_url": "", "post_date": "02/12/2021 13:40:57", "content": "

@amiiiney I understand, thank you. From what I understand I have two options, please correct me if I am wrong :

\n\n

Does that make sense?

", "votes": null, "replies": [] }, { "id": 1197864, "author_name": "amiiiney", "author_url": "", "post_date": "02/12/2021 13:46:48", "content": "

Exactly! Those are your options! Maybe there is another option using tensorboard to visualize your training metrics while committing (I haven't used it before).
\nEDIT: The second option with setting an alarm every 35 minutes sounded more logical, training just 3 epochs will require you to load the weights every time you retrain to resume training.

", "votes": null, "replies": [] }, { "id": 1197990, "author_name": "sajidhussain3", "author_url": "", "post_date": "02/12/2021 15:42:19", "content": "

So when I do Save & Run All can I close the tab? It will get the training done in the cloud? Oh, I never knew it thanks for sharing.

", "votes": null, "replies": [] }, { "id": 1198009, "author_name": "amiiiney", "author_url": "", "post_date": "02/12/2021 16:09:37", "content": "

Yes! When you click on Save & Run All you can just turn off the interactive session and close everything!

", "votes": null, "replies": [] }, { "id": 1198021, "author_name": "sajidhussain3", "author_url": "", "post_date": "02/12/2021 16:21:05", "content": "

Ok thanks

", "votes": null, "replies": [] } ] }, { "id": 1198208, "author_name": "vickygoyal", "author_url": "", "post_date": "02/12/2021 19:45:36", "content": "

I created a link for this very purpose here

", "votes": null, "replies": [ { "id": 1198227, "author_name": "charlesrongione", "author_url": "", "post_date": "02/12/2021 19:54:51", "content": "

thank you :)

", "votes": null, "replies": [] } ] }, { "id": 1198254, "author_name": "alincijov", "author_url": "", "post_date": "02/12/2021 20:36:24", "content": "

you can start by folding your data on smaller data just so you can see how it trains

", "votes": null, "replies": [ { "id": 1198256, "author_name": "charlesrongione", "author_url": "", "post_date": "02/12/2021 20:44:36", "content": "

Hello @alincijov, Tank you for your concern. Do you mean to do a full training on a small sample of the dataset? That's what I did first but as expected I got nothing more than overfitting

", "votes": null, "replies": [] }, { "id": 1199883, "author_name": "alincijov", "author_url": "", "post_date": "02/14/2021 07:48:56", "content": "

Have you tried to use Dataloaders ?

", "votes": null, "replies": [] }, { "id": 1200705, "author_name": "charlesrongione", "author_url": "", "post_date": "02/14/2021 22:23:54", "content": "

@alincijov I am sorry I don't understand the question? Yes of course I use DataLoaders. Am I missing something non-trivial here?

", "votes": null, "replies": [] } ] }, { "id": 1199132, "author_name": "harshalgadhe", "author_url": "", "post_date": "02/13/2021 14:39:14", "content": "

Well @charlesrongione you can save the model after 3 epochs every time and train the new model using the weights of the save model. Its hard to do it every time but that's the solution I found other than using 'Save Version' option on Kaggle.
\nTo save the model you could just use model.save('file_name.h5') and use the weights of the model using
\nnew_model = tf.keras.load_model('path to h5 file') and run the new model again and yes it does works and the advantage of this on using 'Save Version' is that you can keep track of accuracy of model and stop or change the parameters in between.
\nHope it helps: )

", "votes": null, "replies": [ { "id": 1199257, "author_name": "charlesrongione", "author_url": "", "post_date": "02/13/2021 16:54:20", "content": "

Thank you, I am on PyTorch so I did something slightly different but I get your point. Thank you for your help.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1197757": "Hello everyone, this is my first competition and I can't train for more than 4 epochs because, after that, my kernel asks me if I am still here and if I want to keep editing then resets. That causes all my saved weights to disappear. I don't see how I can get good results in 4 epochs. Thank you for your help. \n\nhttps://www.kaggle.com/charlesrongione/fork-of-cassava\n\nbtw feedbacks about the content of my work are also welcome :)", "1197820": "The interactive session is active for just 40 minutes, if you are not active, your notebook is turned off. You need to `save and run all` your notebook for full training\n![](https://i.ibb.co/ZdyRPfR/Screen-Shot-2021-02-12-at-14-15-40.png)", "1197824": "amiiiney Thank you for your answer. I am active. I never left my computer and I check the training every 5 minutes\n\nPlus, maybe I am wrong, but this method does not allow me to carefully watch my training and stop it to readjust some parameters then resume it. Neither to watch my progress bar, which I find very frustrating because I have no idea which batch is my network processing", "1197833": "I mean it's a better practice to just click on `Save & Run All` because then your notebook is running in the cloud and it won't stop training unless it exceeds 9 hours. You can just close the tab and check the notebook when the training is done.\nThe way you are doing it requires you to be active by modifying/executing some cell every 40 minutes. Just being active without any activity in the notebook is not considered active :)", "1197855": "amiiiney I understand, thank you. From what I understand I have two options, please correct me if I am wrong : \n\n - Either I run it on the cloud for maximum nine hours like you said but I am not able to watch how my model is doing, I have no idea what it is doing at time t\n\n - Or I just set each training process for 3 epoch (each one is taking 12 minutes) and rerun it from the previous process. But that way I can see my tqdm progress bar going and have more flexibility on my training progress.\n\nDoes that make sense?", "1197864": "Exactly! Those are your options! Maybe there is another option using tensorboard to visualize your training metrics while committing (I haven't used it before).\nEDIT: The second option with setting an alarm every 35 minutes sounded more logical, training just 3 epochs will require you to load the weights every time you retrain to resume training.", "1197990": "So when I do Save & Run All can I close the tab? It will get the training done in the cloud? Oh, I never knew it thanks for sharing.", "1198009": "Yes! When you click on `Save & Run All` you can just turn off the interactive session and close everything!", "1198021": "Ok thanks", "1198208": "I created a link for this very purpose [here](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/216461)", "1198227": "thank you :)", "1198254": "you can start by folding your data on smaller data just so you can see how it trains", "1198256": "Hello @alincijov, Tank you for your concern. Do you mean to do a full training on a small sample of the dataset? That's what I did first but as expected I got nothing more than overfitting", "1199132": "Well @charlesrongione you can save the model after 3 epochs every time and train the new model using the weights of the save model. Its hard to do it every time but that's the solution I found other than using **'Save Version'** option on Kaggle.\nTo save the model you could just use **model.save('file_name.h5')** and use the weights of the model using \n**new_model = tf.keras.load_model('path to h5 file')** and run the new model again and yes it does works and the advantage of this on using **'Save Version'** is that you can keep track of accuracy of model and stop or change the parameters in between.\nHope it helps: )", "1199257": "Thank you, I am on PyTorch so I did something slightly different but I get your point. Thank you for your help.", "1199883": "Have you tried to use Dataloaders ?", "1200705": "alincijov I am sorry I don't understand the question? Yes of course I use DataLoaders. Am I missing something non-trivial here?" }, "source": "meta" }