![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2234817%2Fa26ab7eb093c89f660dbc0d33643c617%2FContainers.PNG?generation=1572519792944408&alt=media\")
As the GPU for kaggle is holding me by the neck , I have no other choice than to first setup the Colab environment before starting the competition . I have not participated in this competition from the beginning and only last few days , had a look into this ,since sevestral is going to get finished .If this is already discussed in this thread then apologies .
\n\nAttached is the script I have used to set up my Kaggle API for this comp in colab . The datasets were transferred into my drive very fast and I was able to start the training quickly .
\n\nIf you follow this method and set up once , before starting your notebook\n please dont forget to add the below lines in the first few cells .\n1. This will mount your google drive \n`from google.colab import drive
\n\ndrive.mount('/content/drive')`
\n\n2.Make a subdir in your colab runtime\n!mkdir './sample_data/clouds/'
Copy the data from your google drive to local runtime\n``\nif 1 :\npath_to_zip_file = '/content/drive/My Drive/understanding-clouds-resized.zip'\ndirectory_to_extract_to ='./sample_data/clouds/'\nwith zipfile.ZipFile(path_to_zip_file, 'r') as zip_ref:\n zip_ref.extractall(directory_to_extract_to)\nif 1 : \npath_to_zip_file = '/content/drive/My Drive/test_images.zip'\ndirectory_to_extract_to ='./sample_data/clouds/test_images/'\nwith zipfile.ZipFile(path_to_zip_file, 'r') as zip_ref:\n zip_ref.extractall(directory_to_extract_to)
path_to_zip_file = '/content/drive/My Drive/train.csv.zip'\ndirectory_to_extract_to ='./sample_data/clouds/'\nwith zipfile.ZipFile(path_to_zip_file, 'r') as zip_ref:\n zip_ref.extractall(directory_to_extract_to)`\n```
!cp '/content/drive/My Drive/sample_submission.csv' './sample_data/clouds/'
One more tips . Google Colab tend to disconnect . Therefore its better to save the checkpoint of your model in persistent storage i.e your google drive , not in local runtime env .Like below \ntorch.save(model.state_dict(), f'/content/drive/My Drive/model_fold_{}.pt')
As the GPU for kaggle is holding me by the neck , I have no other choice than to first setup the Colab environment before starting the competition . I have not participated in this competition from the beginning and only last few days , had a look into this ,since sevestral is going to get finished .If this is already discussed in this thread then apologies .
\n\nAttached is the script I have used to set up my Kaggle API for this comp in colab . The datasets were transferred into my drive very fast and I was able to start the training quickly .
\n\nIf you follow this method and set up once , before starting your notebook\n please dont forget to add the below lines in the first few cells .\n1. This will mount your google drive \n`from google.colab import drive
\n\ndrive.mount('/content/drive')`
\n\n2.Make a subdir in your colab runtime\n!mkdir './sample_data/clouds/'
Copy the data from your google drive to local runtime\n``\nif 1 :\npath_to_zip_file = '/content/drive/My Drive/understanding-clouds-resized.zip'\ndirectory_to_extract_to ='./sample_data/clouds/'\nwith zipfile.ZipFile(path_to_zip_file, 'r') as zip_ref:\n zip_ref.extractall(directory_to_extract_to)\nif 1 : \npath_to_zip_file = '/content/drive/My Drive/test_images.zip'\ndirectory_to_extract_to ='./sample_data/clouds/test_images/'\nwith zipfile.ZipFile(path_to_zip_file, 'r') as zip_ref:\n zip_ref.extractall(directory_to_extract_to)
path_to_zip_file = '/content/drive/My Drive/train.csv.zip'\ndirectory_to_extract_to ='./sample_data/clouds/'\nwith zipfile.ZipFile(path_to_zip_file, 'r') as zip_ref:\n zip_ref.extractall(directory_to_extract_to)`\n```
!cp '/content/drive/My Drive/sample_submission.csv' './sample_data/clouds/'
One more tips . Google Colab tend to disconnect . Therefore its better to save the checkpoint of your model in persistent storage i.e your google drive , not in local runtime env .Like below \ntorch.save(model.state_dict(), f'/content/drive/My Drive/model_fold_{}.pt')
colab is terrible mate,i recommend paperspace for you
", "rawMarkdown": "colab is terrible mate,i recommend paperspace for you", "votes": null }, { "id": "655061", "postDate": "10/22/2019 16:43:43", "content": "@mobassir paperspace is free or chargable?
", "rawMarkdown": "mobassir paperspace is free or chargable?", "votes": null }, { "id": "655088", "postDate": "10/22/2019 17:15:41", "content": "there are free option @raghaw .. Looks like a better and professional choice . I just didnt get time to get my hands around .. probably worth playing around .. I think @mobassir earlier posted a topic on this .
", "rawMarkdown": "there are free option @raghaw .. Looks like a better and professional choice . I just didnt get time to get my hands around .. probably worth playing around .. I think @mobassir earlier posted a topic on this .", "votes": null }, { "id": "655245", "postDate": "10/22/2019 21:05:12", "content": "yeah free but 10 notebook per month i assume,at least way way better than colab,i tried it a bit and loved it more than colab
", "rawMarkdown": "yeah free but 10 notebook per month i assume,at least way way better than colab,i tried it a bit and loved it more than colab", "votes": null }, { "id": "655419", "postDate": "10/23/2019 03:08:22", "content": "For 8$ a month you get 10 notebook and 200 GB persistent storage. In my opinion ,thats the lowest cost GPU option in the market . There is another option as well which I guess is 24$ . I am in India , still I think working professio3nals may be able to afford even 24$ a month . I am currently getting 0$ to setup and do handson , then subscribe to the paid one .
", "rawMarkdown": "For 8$ a month you get 10 notebook and 200 GB persistent storage. In my opinion ,thats the lowest cost GPU option in the market . There is another option as well which I guess is 24$ . I am in India , still I think working professio3nals may be able to afford even 24$ a month . I am currently getting 0$ to setup and do handson , then subscribe to the paid one .", "votes": null }, { "id": "655933", "postDate": "10/23/2019 17:30:59", "content": "@mobassir I just paid 8$ and got the paperspace set up for this comp and Boy it's thousand times better than colab . In colab there is so much disconnection, Gpu not available , I spent sleepless nights reloading saved model and train 10 epochs by 10 epochs . First time in weeks I am able to run it for 6 hrs continuously in a fast GPU . I owe you one . \nঅনেক অনেক ধন্যবাদ তোমাকে ভাই ।
", "rawMarkdown": "mobassir I just paid 8$ and got the paperspace set up for this comp and Boy it's thousand times better than colab . In colab there is so much disconnection, Gpu not available , I spent sleepless nights reloading saved model and train 10 epochs by 10 epochs . First time in weeks I am able to run it for 6 hrs continuously in a fast GPU . I owe you one . \nঅনেক অনেক ধন্যবাদ তোমাকে ভাই ।", "votes": null }, { "id": "655955", "postDate": "10/23/2019 17:58:55", "content": "i recommend you to use free resources of paperspace and if you still decide to spend money for deep learning then i think instead of paying 8$ frequently to paperspace you should buy a desktop for deep learning mate,i am also going to buy a good laptop only for deep learning within next 1/2 months, धन्यवाद भाई :) @phoenix9032
", "rawMarkdown": "i recommend you to use free resources of paperspace and if you still decide to spend money for deep learning then i think instead of paying 8$ frequently to paperspace you should buy a desktop for deep learning mate,i am also going to buy a good laptop only for deep learning within next 1/2 months, धन्यवाद भाई :) @phoenix9032", "votes": null }, { "id": "656130", "postDate": "10/23/2019 23:58:07", "content": "I wonder why kaggle no longer gives free gcp coupons anymore? For such competition GPU resources are so important for both kagglers and the competition itself. Many image-related competitions have low participation rate due to this.
", "rawMarkdown": "I wonder why kaggle no longer gives free gcp coupons anymore? For such competition GPU resources are so important for both kagglers and the competition itself. Many image-related competitions have low participation rate due to this.", "votes": null }, { "id": "656718", "postDate": "10/24/2019 14:57:26", "content": "I have found that qubvel/segmentation_models in Keras is not compatible with TPU in Colab. Even I have tried tf.keras and re-install the earlier version of Tensorflow...
", "rawMarkdown": "I have found that qubvel/segmentation_models in Keras is not compatible with TPU in Colab. Even I have tried tf.keras and re-install the earlier version of Tensorflow...", "votes": null }, { "id": "656731", "postDate": "10/24/2019 15:08:30", "content": "Oh , I have never tried TPU . I am also using pytorch and qubvel segmentation models .
", "rawMarkdown": "Oh , I have never tried TPU . I am also using pytorch and qubvel segmentation models .", "votes": null }, { "id": "656893", "postDate": "10/24/2019 18:17:21", "content": "After debugging, I have noticed that some of the functions he is using in the model are not compatible with TPU. So if you want to use TPU, you have to write your own code and carefully use those TPU-friendly functions, which is really annoying...
", "rawMarkdown": "After debugging, I have noticed that some of the functions he is using in the model are not compatible with TPU. So if you want to use TPU, you have to write your own code and carefully use those TPU-friendly functions, which is really annoying...", "votes": null }, { "id": "657780", "postDate": "10/25/2019 12:20:11", "content": "I am trying TPU. \nI did not adjust the parameters, but I was able to get a score of 0.656 on efficentnetB4.\nAnd we can increase the batch size.\nBut in resnet I was not able to get a good score.\nMaybe the code is wrong....
", "rawMarkdown": "I am trying TPU. \nI did not adjust the parameters, but I was able to get a score of 0.656 on efficentnetB4.\nAnd we can increase the batch size.\nBut in resnet I was not able to get a good score.\nMaybe the code is wrong....", "votes": null }, { "id": "658631", "postDate": "10/26/2019 09:32:03", "content": "@phoenix9032 are you getting a V100 on colab or a K80, for some reason my notebook only gets a K80 alloted.
", "rawMarkdown": "phoenix9032 are you getting a V100 on colab or a K80, for some reason my notebook only gets a K80 alloted.", "votes": null }, { "id": "658832", "postDate": "10/26/2019 15:47:19", "content": "How did you set up your notebook for this competition. It looks like that you still only get 5GB storage if you run a free instance despite paying $8 a month.
", "rawMarkdown": "How did you set up your notebook for this competition. It looks like that you still only get 5GB storage if you run a free instance despite paying $8 a month.", "votes": null }, { "id": "661757", "postDate": "10/30/2019 17:09:51", "content": "@kani23 Do you use qubvel/segmentation_models?
", "rawMarkdown": "kani23 Do you use qubvel/segmentation_models?", "votes": null }, { "id": "662258", "postDate": "10/31/2019 10:27:10", "content": "Yes, I use qubvel/segmentation_models and Keras.
", "rawMarkdown": "Yes, I use qubvel/segmentation_models and Keras.", "votes": null }, { "id": "662272", "postDate": "10/31/2019 11:07:25", "content": "Same here . But I suggest paperspace , as told my my friend @mobassir . The setup instructions are same as colab.
\n\nHere are few teasers .
\n\nThe Container Options
\n\n
The Price Options
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2234817%2Fcc91d80a4a94f75f6e94598475a42db0%2FOptions.PNG?generation=1572519830693395&alt=media\")
The Overall look\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2234817%2F6443f1bc4352f2621b470225435bd7a8%2FPaperspace.PNG?generation=1572520014923316&alt=media\")
Disclaimer : These are my own screenshots and I am not associated with Paperspace in any other way :P
", "rawMarkdown": "Same here . But I suggest paperspace , as told my my friend @mobassir . The setup instructions are same as colab.\n\nHere are few teasers .\n\nThe Container Options\n\n\n\nThe Price Options\n\n\n\nThe Overall look\n\n\n\nDisclaimer : These are my own screenshots and I am not associated with Paperspace in any other way :P", "votes": null }, { "id": "662564", "postDate": "10/31/2019 17:41:36", "content": "@phoenix9032 did you try saving model weights from paperspace?\ni need to know how i can save my model weights or csv file in my google drive like colab
", "rawMarkdown": "phoenix9032 did you try saving model weights from paperspace?\ni need to know how i can save my model weights or csv file in my google drive like colab", "votes": null }, { "id": "662575", "postDate": "10/31/2019 17:53:59", "content": "It just gives you the notebook server . You need to open and and run notebook like how you do for local system .you can upload /download files seamlessly from the notebook console . I will try to share the screenshot once I am logging in via computer .
", "rawMarkdown": "It just gives you the notebook server . You need to open and and run notebook like how you do for local system .you can upload /download files seamlessly from the notebook console . I will try to share the screenshot once I am logging in via computer .", "votes": null }, { "id": "662585", "postDate": "10/31/2019 18:06:24", "content": "ok dear @phoenix9032
", "rawMarkdown": "ok dear @phoenix9032", "votes": null } ], "comments": [ { "id": 655012, "author_name": "mobassir", "author_url": "", "post_date": "10/22/2019 15:38:08", "content": "colab is terrible mate,i recommend paperspace for you
", "votes": null, "replies": [ { "id": 655061, "author_name": "raghaw", "author_url": "", "post_date": "10/22/2019 16:43:43", "content": "@mobassir paperspace is free or chargable?
", "votes": null, "replies": [] }, { "id": 655088, "author_name": "phoenix9032", "author_url": "", "post_date": "10/22/2019 17:15:41", "content": "there are free option @raghaw .. Looks like a better and professional choice . I just didnt get time to get my hands around .. probably worth playing around .. I think @mobassir earlier posted a topic on this .
", "votes": null, "replies": [] }, { "id": 655245, "author_name": "mobassir", "author_url": "", "post_date": "10/22/2019 21:05:12", "content": "yeah free but 10 notebook per month i assume,at least way way better than colab,i tried it a bit and loved it more than colab
", "votes": null, "replies": [] }, { "id": 655419, "author_name": "phoenix9032", "author_url": "", "post_date": "10/23/2019 03:08:22", "content": "For 8$ a month you get 10 notebook and 200 GB persistent storage. In my opinion ,thats the lowest cost GPU option in the market . There is another option as well which I guess is 24$ . I am in India , still I think working professio3nals may be able to afford even 24$ a month . I am currently getting 0$ to setup and do handson , then subscribe to the paid one .
", "votes": null, "replies": [] }, { "id": 655933, "author_name": "phoenix9032", "author_url": "", "post_date": "10/23/2019 17:30:59", "content": "@mobassir I just paid 8$ and got the paperspace set up for this comp and Boy it's thousand times better than colab . In colab there is so much disconnection, Gpu not available , I spent sleepless nights reloading saved model and train 10 epochs by 10 epochs . First time in weeks I am able to run it for 6 hrs continuously in a fast GPU . I owe you one . \nঅনেক অনেক ধন্যবাদ তোমাকে ভাই ।
", "votes": null, "replies": [] }, { "id": 655955, "author_name": "mobassir", "author_url": "", "post_date": "10/23/2019 17:58:55", "content": "i recommend you to use free resources of paperspace and if you still decide to spend money for deep learning then i think instead of paying 8$ frequently to paperspace you should buy a desktop for deep learning mate,i am also going to buy a good laptop only for deep learning within next 1/2 months, धन्यवाद भाई :) @phoenix9032
", "votes": null, "replies": [] }, { "id": 656130, "author_name": "ajinomoto132", "author_url": "", "post_date": "10/23/2019 23:58:07", "content": "I wonder why kaggle no longer gives free gcp coupons anymore? For such competition GPU resources are so important for both kagglers and the competition itself. Many image-related competitions have low participation rate due to this.
", "votes": null, "replies": [] }, { "id": 658832, "author_name": "fschilder", "author_url": "", "post_date": "10/26/2019 15:47:19", "content": "How did you set up your notebook for this competition. It looks like that you still only get 5GB storage if you run a free instance despite paying $8 a month.
", "votes": null, "replies": [] } ] }, { "id": 656718, "author_name": "gogo827jz", "author_url": "", "post_date": "10/24/2019 14:57:26", "content": "I have found that qubvel/segmentation_models in Keras is not compatible with TPU in Colab. Even I have tried tf.keras and re-install the earlier version of Tensorflow...
", "votes": null, "replies": [ { "id": 656731, "author_name": "phoenix9032", "author_url": "", "post_date": "10/24/2019 15:08:30", "content": "Oh , I have never tried TPU . I am also using pytorch and qubvel segmentation models .
", "votes": null, "replies": [] }, { "id": 656893, "author_name": "gogo827jz", "author_url": "", "post_date": "10/24/2019 18:17:21", "content": "After debugging, I have noticed that some of the functions he is using in the model are not compatible with TPU. So if you want to use TPU, you have to write your own code and carefully use those TPU-friendly functions, which is really annoying...
", "votes": null, "replies": [] }, { "id": 657780, "author_name": "kani23", "author_url": "", "post_date": "10/25/2019 12:20:11", "content": "I am trying TPU. \nI did not adjust the parameters, but I was able to get a score of 0.656 on efficentnetB4.\nAnd we can increase the batch size.\nBut in resnet I was not able to get a good score.\nMaybe the code is wrong....
", "votes": null, "replies": [] }, { "id": 661757, "author_name": "gogo827jz", "author_url": "", "post_date": "10/30/2019 17:09:51", "content": "@kani23 Do you use qubvel/segmentation_models?
", "votes": null, "replies": [] }, { "id": 662258, "author_name": "kani23", "author_url": "", "post_date": "10/31/2019 10:27:10", "content": "Yes, I use qubvel/segmentation_models and Keras.
", "votes": null, "replies": [] } ] }, { "id": 658631, "author_name": "axel81", "author_url": "", "post_date": "10/26/2019 09:32:03", "content": "@phoenix9032 are you getting a V100 on colab or a K80, for some reason my notebook only gets a K80 alloted.
", "votes": null, "replies": [ { "id": 662272, "author_name": "phoenix9032", "author_url": "", "post_date": "10/31/2019 11:07:25", "content": "Same here . But I suggest paperspace , as told my my friend @mobassir . The setup instructions are same as colab.
\n\nHere are few teasers .
\n\nThe Container Options
\n\n
The Price Options
\n\n
The Overall look\n
Disclaimer : These are my own screenshots and I am not associated with Paperspace in any other way :P
", "votes": null, "replies": [] }, { "id": 662564, "author_name": "mobassir", "author_url": "", "post_date": "10/31/2019 17:41:36", "content": "@phoenix9032 did you try saving model weights from paperspace?\ni need to know how i can save my model weights or csv file in my google drive like colab
", "votes": null, "replies": [] }, { "id": 662575, "author_name": "phoenix9032", "author_url": "", "post_date": "10/31/2019 17:53:59", "content": "It just gives you the notebook server . You need to open and and run notebook like how you do for local system .you can upload /download files seamlessly from the notebook console . I will try to share the screenshot once I am logging in via computer .
", "votes": null, "replies": [] }, { "id": 662585, "author_name": "mobassir", "author_url": "", "post_date": "10/31/2019 18:06:24", "content": "ok dear @phoenix9032
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "654977": "As the GPU for kaggle is holding me by the neck , I have no other choice than to first setup the Colab environment before starting the competition . I have not participated in this competition from the beginning and only last few days , had a look into this ,since sevestral is going to get finished .If this is already discussed in this thread then apologies . \n\nAttached is the script I have used to set up my Kaggle API for this comp in colab . The datasets were transferred into my drive very fast and I was able to start the training quickly . \n\nIf you follow this method and set up once , before starting your notebook\n please dont forget to add the below lines in the first few cells .\n1. This will mount your google drive \n`from google.colab import drive\n\ndrive.mount('/content/drive')`\n\n2.Make a subdir in your colab runtime\n`!mkdir './sample_data/clouds/'`\n\n3. Copy the data from your google drive to local runtime\n```\n`if 1 :\n path_to_zip_file = '/content/drive/My Drive/understanding-clouds-resized.zip'\n directory_to_extract_to ='./sample_data/clouds/'\n with zipfile.ZipFile(path_to_zip_file, 'r') as zip_ref:\n zip_ref.extractall(directory_to_extract_to)\nif 1 : \n path_to_zip_file = '/content/drive/My Drive/test_images.zip'\n directory_to_extract_to ='./sample_data/clouds/test_images/'\n with zipfile.ZipFile(path_to_zip_file, 'r') as zip_ref:\n zip_ref.extractall(directory_to_extract_to)\n \n path_to_zip_file = '/content/drive/My Drive/train.csv.zip'\n directory_to_extract_to ='./sample_data/clouds/'\n with zipfile.ZipFile(path_to_zip_file, 'r') as zip_ref:\n zip_ref.extractall(directory_to_extract_to)`\n```\n\n`!cp '/content/drive/My Drive/sample_submission.csv' './sample_data/clouds/' `\n\nOne more tips . Google Colab tend to disconnect . Therefore its better to save the checkpoint of your model in persistent storage i.e your google drive , not in local runtime env .Like below \n`torch.save(model.state_dict(), f'/content/drive/My Drive/model_fold_{}.pt')`", "655012": "colab is terrible mate,i recommend paperspace for you", "655061": "mobassir paperspace is free or chargable?", "655088": "there are free option @raghaw .. Looks like a better and professional choice . I just didnt get time to get my hands around .. probably worth playing around .. I think @mobassir earlier posted a topic on this .", "655245": "yeah free but 10 notebook per month i assume,at least way way better than colab,i tried it a bit and loved it more than colab", "655419": "For 8$ a month you get 10 notebook and 200 GB persistent storage. In my opinion ,thats the lowest cost GPU option in the market . There is another option as well which I guess is 24$ . I am in India , still I think working professio3nals may be able to afford even 24$ a month . I am currently getting 0$ to setup and do handson , then subscribe to the paid one .", "655933": "mobassir I just paid 8$ and got the paperspace set up for this comp and Boy it's thousand times better than colab . In colab there is so much disconnection, Gpu not available , I spent sleepless nights reloading saved model and train 10 epochs by 10 epochs . First time in weeks I am able to run it for 6 hrs continuously in a fast GPU . I owe you one . \nঅনেক অনেক ধন্যবাদ তোমাকে ভাই ।", "655955": "i recommend you to use free resources of paperspace and if you still decide to spend money for deep learning then i think instead of paying 8$ frequently to paperspace you should buy a desktop for deep learning mate,i am also going to buy a good laptop only for deep learning within next 1/2 months, धन्यवाद भाई :) @phoenix9032", "656130": "I wonder why kaggle no longer gives free gcp coupons anymore? For such competition GPU resources are so important for both kagglers and the competition itself. Many image-related competitions have low participation rate due to this.", "656718": "I have found that qubvel/segmentation_models in Keras is not compatible with TPU in Colab. Even I have tried tf.keras and re-install the earlier version of Tensorflow...", "656731": "Oh , I have never tried TPU . I am also using pytorch and qubvel segmentation models .", "656893": "After debugging, I have noticed that some of the functions he is using in the model are not compatible with TPU. So if you want to use TPU, you have to write your own code and carefully use those TPU-friendly functions, which is really annoying...", "657780": "I am trying TPU. \nI did not adjust the parameters, but I was able to get a score of 0.656 on efficentnetB4.\nAnd we can increase the batch size.\nBut in resnet I was not able to get a good score.\nMaybe the code is wrong....", "658631": "phoenix9032 are you getting a V100 on colab or a K80, for some reason my notebook only gets a K80 alloted.", "658832": "How did you set up your notebook for this competition. It looks like that you still only get 5GB storage if you run a free instance despite paying $8 a month.", "661757": "kani23 Do you use qubvel/segmentation_models?", "662258": "Yes, I use qubvel/segmentation_models and Keras.", "662272": "Same here . But I suggest paperspace , as told my my friend @mobassir . The setup instructions are same as colab.\n\nHere are few teasers .\n\nThe Container Options\n\n\n\nThe Price Options\n\n\n\nThe Overall look\n\n\n\nDisclaimer : These are my own screenshots and I am not associated with Paperspace in any other way :P", "662564": "phoenix9032 did you try saving model weights from paperspace?\ni need to know how i can save my model weights or csv file in my google drive like colab", "662575": "It just gives you the notebook server . You need to open and and run notebook like how you do for local system .you can upload /download files seamlessly from the notebook console . I will try to share the screenshot once I am logging in via computer .", "662585": "ok dear @phoenix9032" }, "source": "meta" }