Hi I am relatively new and have trained my model on 20K images because of the data set size and have reached 86 percent. I am trying to get my data to colab to train .. I have already split the data for PyTorch into appropriate folders .. I zipped the file but colab tells me it is a bad zip file .. At any rate .. how should I zip the file or tar etc .. and what is the best way to import the data .. with PyTorch I have to split the data (I am practicing with it) I don't want to waste colab time splitting the data set into train 1 and 0 folders and valid 1 and 0 folders .. please any direction would be greatly appreciated!!
", "messages": [ { "id": "453280", "postDate": "01/10/2019 00:23:06", "content": "Hi I am relatively new and have trained my model on 20K images because of the data set size and have reached 86 percent. I am trying to get my data to colab to train .. I have already split the data for PyTorch into appropriate folders .. I zipped the file but colab tells me it is a bad zip file .. At any rate .. how should I zip the file or tar etc .. and what is the best way to import the data .. with PyTorch I have to split the data (I am practicing with it) I don't want to waste colab time splitting the data set into train 1 and 0 folders and valid 1 and 0 folders .. please any direction would be greatly appreciated!!
", "rawMarkdown": "Hi I am relatively new and have trained my model on 20K images because of the data set size and have reached 86 percent. I am trying to get my data to colab to train .. I have already split the data for PyTorch into appropriate folders .. I zipped the file but colab tells me it is a bad zip file .. At any rate .. how should I zip the file or tar etc .. and what is the best way to import the data .. with PyTorch I have to split the data (I am practicing with it) I don't want to waste colab time splitting the data set into train 1 and 0 folders and valid 1 and 0 folders .. please any direction would be greatly appreciated!!", "votes": null }, { "id": "453415", "postDate": "01/10/2019 07:00:14", "content": "Using Google Cloud Platform would be better than Colab.
", "rawMarkdown": "Using Google Cloud Platform would be better than Colab.", "votes": null }, { "id": "453519", "postDate": "01/10/2019 09:57:31", "content": "I agree with Tony's comment. I would recommend spending an hour or two getting familiar with Google Cloud Platform (GCP) and using a GPU there. The Colab GPU doesn't cut the cake for any decent sized training job.
\n\nGCP's Getting Started Guide (link below) is a great resource to go through and when you sign up you get $300 worth of credit - which should be more than enough for you train your model. It will also be really useful to get experience with GCP, as this is a very common platform used by professionals, so another tick on the resume per se.
\n\nhttps://cloud.google.com/gcp/getting-started/
", "rawMarkdown": "I agree with Tony's comment. I would recommend spending an hour or two getting familiar with Google Cloud Platform (GCP) and using a GPU there. The Colab GPU doesn't cut the cake for any decent sized training job.\n\nGCP's Getting Started Guide (link below) is a great resource to go through and when you sign up you get $300 worth of credit - which should be more than enough for you train your model. It will also be really useful to get experience with GCP, as this is a very common platform used by professionals, so another tick on the resume per se. \n\nhttps://cloud.google.com/gcp/getting-started/", "votes": null }, { "id": "454049", "postDate": "01/11/2019 04:35:10", "content": "Thank you both for your reply I will pursue that option! I have a friend who is familiar with Google cloud and was recommending it as well. Have a good evening.\nRegards, \nGennaro
", "rawMarkdown": "Thank you both for your reply I will pursue that option! I have a friend who is familiar with Google cloud and was recommending it as well. Have a good evening.\nRegards, \nGennaro", "votes": null }, { "id": "457042", "postDate": "01/16/2019 21:43:47", "content": "What problems of training with Google Colab for this Kaggle?
", "rawMarkdown": "What problems of training with Google Colab for this Kaggle?", "votes": null }, { "id": "457102", "postDate": "01/17/2019 00:16:45", "content": "Colab crashes when I try to unzip 230K images.. It is also difficult to unzip them while on Google drive or to send them individually.. Google cloud has offered substantially more power and flexability .. A bit of a learning curve but otherwise very helpful! Thanks for the input everyone was a great idea.
", "rawMarkdown": "Colab crashes when I try to unzip 230K images.. It is also difficult to unzip them while on Google drive or to send them individually.. Google cloud has offered substantially more power and flexability .. A bit of a learning curve but otherwise very helpful! Thanks for the input everyone was a great idea.", "votes": null }, { "id": "493176", "postDate": "03/18/2019 11:29:31", "content": "Hi,\nDid you succeed ? How did you do ?
", "rawMarkdown": "Hi,\nDid you succeed ? How did you do ?", "votes": null } ], "comments": [ { "id": 453415, "author_name": "easter3163", "author_url": "", "post_date": "01/10/2019 07:00:14", "content": "Using Google Cloud Platform would be better than Colab.
", "votes": null, "replies": [] }, { "id": 453519, "author_name": "todddequincey", "author_url": "", "post_date": "01/10/2019 09:57:31", "content": "I agree with Tony's comment. I would recommend spending an hour or two getting familiar with Google Cloud Platform (GCP) and using a GPU there. The Colab GPU doesn't cut the cake for any decent sized training job.
\n\nGCP's Getting Started Guide (link below) is a great resource to go through and when you sign up you get $300 worth of credit - which should be more than enough for you train your model. It will also be really useful to get experience with GCP, as this is a very common platform used by professionals, so another tick on the resume per se.
\n\nhttps://cloud.google.com/gcp/getting-started/
", "votes": null, "replies": [ { "id": 454049, "author_name": "gtm1235", "author_url": "", "post_date": "01/11/2019 04:35:10", "content": "Thank you both for your reply I will pursue that option! I have a friend who is familiar with Google cloud and was recommending it as well. Have a good evening.\nRegards, \nGennaro
", "votes": null, "replies": [] }, { "id": 457042, "author_name": "anikishaev", "author_url": "", "post_date": "01/16/2019 21:43:47", "content": "What problems of training with Google Colab for this Kaggle?
", "votes": null, "replies": [] }, { "id": 457102, "author_name": "gtm1235", "author_url": "", "post_date": "01/17/2019 00:16:45", "content": "Colab crashes when I try to unzip 230K images.. It is also difficult to unzip them while on Google drive or to send them individually.. Google cloud has offered substantially more power and flexability .. A bit of a learning curve but otherwise very helpful! Thanks for the input everyone was a great idea.
", "votes": null, "replies": [] } ] }, { "id": 493176, "author_name": "johnnyhalliday", "author_url": "", "post_date": "03/18/2019 11:29:31", "content": "Hi,\nDid you succeed ? How did you do ?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "453280": "Hi I am relatively new and have trained my model on 20K images because of the data set size and have reached 86 percent. I am trying to get my data to colab to train .. I have already split the data for PyTorch into appropriate folders .. I zipped the file but colab tells me it is a bad zip file .. At any rate .. how should I zip the file or tar etc .. and what is the best way to import the data .. with PyTorch I have to split the data (I am practicing with it) I don't want to waste colab time splitting the data set into train 1 and 0 folders and valid 1 and 0 folders .. please any direction would be greatly appreciated!!", "453415": "Using Google Cloud Platform would be better than Colab.", "453519": "I agree with Tony's comment. I would recommend spending an hour or two getting familiar with Google Cloud Platform (GCP) and using a GPU there. The Colab GPU doesn't cut the cake for any decent sized training job.\n\nGCP's Getting Started Guide (link below) is a great resource to go through and when you sign up you get $300 worth of credit - which should be more than enough for you train your model. It will also be really useful to get experience with GCP, as this is a very common platform used by professionals, so another tick on the resume per se. \n\nhttps://cloud.google.com/gcp/getting-started/", "454049": "Thank you both for your reply I will pursue that option! I have a friend who is familiar with Google cloud and was recommending it as well. Have a good evening.\nRegards, \nGennaro", "457042": "What problems of training with Google Colab for this Kaggle?", "457102": "Colab crashes when I try to unzip 230K images.. It is also difficult to unzip them while on Google drive or to send them individually.. Google cloud has offered substantially more power and flexability .. A bit of a learning curve but otherwise very helpful! Thanks for the input everyone was a great idea.", "493176": "Hi,\nDid you succeed ? How did you do ?" }, "source": "meta" }