Hi, I am new to Data Science, and very new to Image processing, actually this will be my first attempt. Hence my question may be naive. (Also kindly do not blast me for no submissions. I am a genuine person and trying to learn from Kaggle. I am trying, but it is taking humongous amount of time to train these models with my limited resources). I want to train a Keras model with a small epoch size( Say 5). Then I want to put this training in a loop. The loop will run for 10 iterations. I will state the reason as to why I am doing this. I don't have access to powerful hardware. A simple epoch on my machine takes 3 Hrs to run.I was trying to run for 50 epochs. 3 days into the training, Windows crashed and I lost every thing, time and whatever progress the model made. My sample code would be like this: ==============================================
\n\nMy idea is to run the model for small epochs, save the model, re load, recompile and continue with training for the next round with the same data. I believe this is equivalent to running the model for 50 epochs. From what I read about CNN and Keras, I understand this is possible as these models can be trained incrementally. Just wanted the advice of the experts as to whether this will work fine, or my understanding is wrong and I am making a fundamental mistake. Apologies for the long post. Thanks and regards Sayantan Raha
", "messages": [ { "id": "121639", "postDate": "05/28/2016 08:04:46", "content": "Hi, I am new to Data Science, and very new to Image processing, actually this will be my first attempt. Hence my question may be naive. (Also kindly do not blast me for no submissions. I am a genuine person and trying to learn from Kaggle. I am trying, but it is taking humongous amount of time to train these models with my limited resources). I want to train a Keras model with a small epoch size( Say 5). Then I want to put this training in a loop. The loop will run for 10 iterations. I will state the reason as to why I am doing this. I don't have access to powerful hardware. A simple epoch on my machine takes 3 Hrs to run.I was trying to run for 50 epochs. 3 days into the training, Windows crashed and I lost every thing, time and whatever progress the model made. My sample code would be like this: ==============================================
\n\nMy idea is to run the model for small epochs, save the model, re load, recompile and continue with training for the next round with the same data. I believe this is equivalent to running the model for 50 epochs. From what I read about CNN and Keras, I understand this is possible as these models can be trained incrementally. Just wanted the advice of the experts as to whether this will work fine, or my understanding is wrong and I am making a fundamental mistake. Apologies for the long post. Thanks and regards Sayantan Raha
", "rawMarkdown": "Hi, I am new to Data Science, and very new to Image processing, actually this will be my first attempt. Hence my question may be naive. (Also kindly do not blast me for no submissions. I am a genuine person and trying to learn from Kaggle. I am trying, but it is taking humongous amount of time to train these models with my limited resources). I want to train a Keras model with a small epoch size( Say 5). Then I want to put this training in a loop. The loop will run for 10 iterations. I will state the reason as to why I am doing this. I don't have access to powerful hardware. A simple epoch on my machine takes 3 Hrs to run.I was trying to run for 50 epochs. 3 days into the training, Windows crashed and I lost every thing, time and whatever progress the model made. My sample code would be like this: ============================================== \r\nfor i in range(10): print(i) model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=5, shuffle=True, verbose=1, validation_data=(X_valid, Y_valid), callbacks=callbacks) print(\"Saving model....\") save_model(model) \r\n=========================================================== \r\nMy idea is to run the model for small epochs, save the model, re load, recompile and continue with training for the next round with the same data. I believe this is equivalent to running the model for 50 epochs. From what I read about CNN and Keras, I understand this is possible as these models can be trained incrementally. Just wanted the advice of the experts as to whether this will work fine, or my understanding is wrong and I am making a fundamental mistake. Apologies for the long post. Thanks and regards Sayantan Raha", "votes": null }, { "id": "121645", "postDate": "05/28/2016 09:58:12", "content": "1, The g2.2xlarge instance on AWS is sufficient for the computational needs of this competition. In most cases, it costs less than $0.2 per hour. Compared with the GPU on laptops, the g2.2xlarge should be faster.
\n\n2, You could save the model to disk, reload the model from disk and continue the training phase afterwards.
\n\n3, As you used "fit" function, did you load all the image into memory? The resized images might be too small and even a human could not identify correctly. You may want to check the "fit_generator" function instead.
", "rawMarkdown": "1, The g2.2xlarge instance on AWS is sufficient for the computational needs of this competition. In most cases, it costs less than $0.2 per hour. Compared with the GPU on laptops, the g2.2xlarge should be faster.\r\n\r\n2, You could save the model to disk, reload the model from disk and continue the training phase afterwards.\r\n\r\n3, As you used \"fit\" function, did you load all the image into memory? The resized images might be too small and even a human could not identify correctly. You may want to check the \"fit_generator\" function instead.", "votes": null }, { "id": "121651", "postDate": "05/28/2016 12:01:38", "content": "for keras you can also use the callbacks and checkpoint to save your weights to disk periodically like below,
\n\nmodel = VGG_16()
\n\ncheckpointer = ModelCheckpoint(filepath="kerasweightsvgg16.pkl", verbose=1, save_best_only=True)
\n\nmodel.fit_generator(......., callbacks=[checkpointer])
", "rawMarkdown": "for keras you can also use the callbacks and checkpoint to save your weights to disk periodically like below,\r\n\r\nmodel = VGG_16()\r\n \r\ncheckpointer = ModelCheckpoint(filepath=\"kerasweightsvgg16.pkl\", verbose=1, save_best_only=True)\r\n\r\nmodel.fit_generator(......., callbacks=[checkpointer])", "votes": null }, { "id": "121661", "postDate": "05/28/2016 14:50:10", "content": "Thanks ChaseWind and DavidGbodiOdaibo .
\n\nI did resize the picture to 64x64, which as you said is really difficult for a human being to see. I tried with 224 and 128. But they were even slower.
\n\nI am loading the entire train set in memory, but training in batches. I will check out the fit_generator and see how I can optimize the entire cycle. Thanks once more.
\n\nRegards\nSayantan Raha
", "rawMarkdown": "Thanks ChaseWind and DavidGbodiOdaibo .\r\n\r\nI did resize the picture to 64x64, which as you said is really difficult for a human being to see. I tried with 224 and 128. But they were even slower. \r\n\r\nI am loading the entire train set in memory, but training in batches. I will check out the fit_generator and see how I can optimize the entire cycle. Thanks once more. \r\n\r\n\r\nRegards\r\nSayantan Raha", "votes": null }, { "id": "121671", "postDate": "05/28/2016 15:49:15", "content": "[quote=ChaseWind;121645]\n1, The g2.2xlarge instance on AWS is sufficient for the computational needs of this competition. In most cases, it costs less than $0.2 per hour. Compared with the GPU on laptops, the g2.2xlarge should be faster.\n[/quote]
\n\nThe on-demand price is $0.65/hour. I found it hard to get spot pricing that was much cheaper and reliable.
\n\nI ended up buying a 980 Ti for my own machine and it turned out to be 2.5x faster than AWS.
\n\nThe upcoming 1080 looks like a very cost effective option.
", "rawMarkdown": "[quote=ChaseWind;121645]\r\n1, The g2.2xlarge instance on AWS is sufficient for the computational needs of this competition. In most cases, it costs less than $0.2 per hour. Compared with the GPU on laptops, the g2.2xlarge should be faster.\r\n[/quote]\r\n\r\nThe on-demand price is $0.65/hour. I found it hard to get spot pricing that was much cheaper and reliable.\r\n\r\nI ended up buying a 980 Ti for my own machine and it turned out to be 2.5x faster than AWS.\r\n\r\nThe upcoming 1080 looks like a very cost effective option.", "votes": null }, { "id": "126854", "postDate": "07/12/2016 17:35:50", "content": "[quote=gauss256;121671]
\n\nI ended up buying a 980 Ti for my own machine and it turned out to be 2.5x faster than AWS.
\n\nThe upcoming 1080 looks like a very cost effective option.
\n\n[/quote]
\n\nThat sounds good. What's the configuration of your machine?
", "rawMarkdown": "[quote=gauss256;121671]\r\n\r\nI ended up buying a 980 Ti for my own machine and it turned out to be 2.5x faster than AWS.\r\n\r\nThe upcoming 1080 looks like a very cost effective option.\r\n\r\n[/quote]\r\n\r\nThat sounds good. What's the configuration of your machine?", "votes": null }, { "id": "127355", "postDate": "07/14/2016 00:53:26", "content": "Ubuntu 14.04, 32GB RAM, 512GB SSD, Nvidia 980 Ti, Intel i7
\n\nI'm very happy with it. :)
", "rawMarkdown": "Ubuntu 14.04, 32GB RAM, 512GB SSD, Nvidia 980 Ti, Intel i7\r\n\r\nI'm very happy with it. :)", "votes": null } ], "comments": [ { "id": 121645, "author_name": "xingyang", "author_url": "", "post_date": "05/28/2016 09:58:12", "content": "1, The g2.2xlarge instance on AWS is sufficient for the computational needs of this competition. In most cases, it costs less than $0.2 per hour. Compared with the GPU on laptops, the g2.2xlarge should be faster.
\n\n2, You could save the model to disk, reload the model from disk and continue the training phase afterwards.
\n\n3, As you used "fit" function, did you load all the image into memory? The resized images might be too small and even a human could not identify correctly. You may want to check the "fit_generator" function instead.
", "votes": null, "replies": [] }, { "id": 121651, "author_name": "godaibo", "author_url": "", "post_date": "05/28/2016 12:01:38", "content": "for keras you can also use the callbacks and checkpoint to save your weights to disk periodically like below,
\n\nmodel = VGG_16()
\n\ncheckpointer = ModelCheckpoint(filepath="kerasweightsvgg16.pkl", verbose=1, save_best_only=True)
\n\nmodel.fit_generator(......., callbacks=[checkpointer])
", "votes": null, "replies": [] }, { "id": 121661, "author_name": "sayantan", "author_url": "", "post_date": "05/28/2016 14:50:10", "content": "Thanks ChaseWind and DavidGbodiOdaibo .
\n\nI did resize the picture to 64x64, which as you said is really difficult for a human being to see. I tried with 224 and 128. But they were even slower.
\n\nI am loading the entire train set in memory, but training in batches. I will check out the fit_generator and see how I can optimize the entire cycle. Thanks once more.
\n\nRegards\nSayantan Raha
", "votes": null, "replies": [] }, { "id": 121671, "author_name": "gauss256", "author_url": "", "post_date": "05/28/2016 15:49:15", "content": "[quote=ChaseWind;121645]\n1, The g2.2xlarge instance on AWS is sufficient for the computational needs of this competition. In most cases, it costs less than $0.2 per hour. Compared with the GPU on laptops, the g2.2xlarge should be faster.\n[/quote]
\n\nThe on-demand price is $0.65/hour. I found it hard to get spot pricing that was much cheaper and reliable.
\n\nI ended up buying a 980 Ti for my own machine and it turned out to be 2.5x faster than AWS.
\n\nThe upcoming 1080 looks like a very cost effective option.
", "votes": null, "replies": [] }, { "id": 126854, "author_name": "akashtandon", "author_url": "", "post_date": "07/12/2016 17:35:50", "content": "[quote=gauss256;121671]
\n\nI ended up buying a 980 Ti for my own machine and it turned out to be 2.5x faster than AWS.
\n\nThe upcoming 1080 looks like a very cost effective option.
\n\n[/quote]
\n\nThat sounds good. What's the configuration of your machine?
", "votes": null, "replies": [] }, { "id": 127355, "author_name": "gauss256", "author_url": "", "post_date": "07/14/2016 00:53:26", "content": "Ubuntu 14.04, 32GB RAM, 512GB SSD, Nvidia 980 Ti, Intel i7
\n\nI'm very happy with it. :)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "121639": "Hi, I am new to Data Science, and very new to Image processing, actually this will be my first attempt. Hence my question may be naive. (Also kindly do not blast me for no submissions. I am a genuine person and trying to learn from Kaggle. I am trying, but it is taking humongous amount of time to train these models with my limited resources). I want to train a Keras model with a small epoch size( Say 5). Then I want to put this training in a loop. The loop will run for 10 iterations. I will state the reason as to why I am doing this. I don't have access to powerful hardware. A simple epoch on my machine takes 3 Hrs to run.I was trying to run for 50 epochs. 3 days into the training, Windows crashed and I lost every thing, time and whatever progress the model made. My sample code would be like this: ============================================== \r\nfor i in range(10): print(i) model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=5, shuffle=True, verbose=1, validation_data=(X_valid, Y_valid), callbacks=callbacks) print(\"Saving model....\") save_model(model) \r\n=========================================================== \r\nMy idea is to run the model for small epochs, save the model, re load, recompile and continue with training for the next round with the same data. I believe this is equivalent to running the model for 50 epochs. From what I read about CNN and Keras, I understand this is possible as these models can be trained incrementally. Just wanted the advice of the experts as to whether this will work fine, or my understanding is wrong and I am making a fundamental mistake. Apologies for the long post. Thanks and regards Sayantan Raha", "121645": "1, The g2.2xlarge instance on AWS is sufficient for the computational needs of this competition. In most cases, it costs less than $0.2 per hour. Compared with the GPU on laptops, the g2.2xlarge should be faster.\r\n\r\n2, You could save the model to disk, reload the model from disk and continue the training phase afterwards.\r\n\r\n3, As you used \"fit\" function, did you load all the image into memory? The resized images might be too small and even a human could not identify correctly. You may want to check the \"fit_generator\" function instead.", "121651": "for keras you can also use the callbacks and checkpoint to save your weights to disk periodically like below,\r\n\r\nmodel = VGG_16()\r\n \r\ncheckpointer = ModelCheckpoint(filepath=\"kerasweightsvgg16.pkl\", verbose=1, save_best_only=True)\r\n\r\nmodel.fit_generator(......., callbacks=[checkpointer])", "121661": "Thanks ChaseWind and DavidGbodiOdaibo .\r\n\r\nI did resize the picture to 64x64, which as you said is really difficult for a human being to see. I tried with 224 and 128. But they were even slower. \r\n\r\nI am loading the entire train set in memory, but training in batches. I will check out the fit_generator and see how I can optimize the entire cycle. Thanks once more. \r\n\r\n\r\nRegards\r\nSayantan Raha", "121671": "[quote=ChaseWind;121645]\r\n1, The g2.2xlarge instance on AWS is sufficient for the computational needs of this competition. In most cases, it costs less than $0.2 per hour. Compared with the GPU on laptops, the g2.2xlarge should be faster.\r\n[/quote]\r\n\r\nThe on-demand price is $0.65/hour. I found it hard to get spot pricing that was much cheaper and reliable.\r\n\r\nI ended up buying a 980 Ti for my own machine and it turned out to be 2.5x faster than AWS.\r\n\r\nThe upcoming 1080 looks like a very cost effective option.", "126854": "[quote=gauss256;121671]\r\n\r\nI ended up buying a 980 Ti for my own machine and it turned out to be 2.5x faster than AWS.\r\n\r\nThe upcoming 1080 looks like a very cost effective option.\r\n\r\n[/quote]\r\n\r\nThat sounds good. What's the configuration of your machine?", "127355": "Ubuntu 14.04, 32GB RAM, 512GB SSD, Nvidia 980 Ti, Intel i7\r\n\r\nI'm very happy with it. :)" }, "source": "meta" }