I'm using a chromebook currently. I don't have storage space for the entire train dataset or the compute resources. I was curious if I can still compete effectively by only relying on Kaggle notebook and my internet connection?
", "messages": [ { "id": "715848", "postDate": "01/10/2020 22:13:11", "content": "I'm using a chromebook currently. I don't have storage space for the entire train dataset or the compute resources. I was curious if I can still compete effectively by only relying on Kaggle notebook and my internet connection?
", "rawMarkdown": "I'm using a chromebook currently. I don't have storage space for the entire train dataset or the compute resources. I was curious if I can still compete effectively by only relying on Kaggle notebook and my internet connection?", "votes": null }, { "id": "715900", "postDate": "01/11/2020 00:10:48", "content": "It's highly unlikely, unfortunately. You can play around with the sample training but you won't be able to get high score without the full training set (unless you use non deep learning methods). Try applying for amazon or Google credits (see posts)
", "rawMarkdown": "It's highly unlikely, unfortunately. You can play around with the sample training but you won't be able to get high score without the full training set (unless you use non deep learning methods). Try applying for amazon or Google credits (see posts)", "votes": null }, { "id": "715942", "postDate": "01/11/2020 02:26:47", "content": "I am in 300. It may not be impressive but I just want to tell you that I train on full training data using google colaboratory. I have wrote some basic functions to make life easier. When I finally leave it for training I just close my laptop and open it in mobile phone. So with these little tricks I can train on whole data. If you want the functions and how I use them then I can share it here.
", "rawMarkdown": "I am in 300. It may not be impressive but I just want to tell you that I train on full training data using google colaboratory. I have wrote some basic functions to make life easier. When I finally leave it for training I just close my laptop and open it in mobile phone. So with these little tricks I can train on whole data. If you want the functions and how I use them then I can share it here.", "votes": null }, { "id": "715986", "postDate": "01/11/2020 05:23:01", "content": "The full training set is 500gb. Afik colab does not provide this kind of storage.
", "rawMarkdown": "The full training set is 500gb. Afik colab does not provide this kind of storage.", "votes": null }, { "id": "715988", "postDate": "01/11/2020 05:28:30", "content": "That's where I use a function which will download and extract a training set and will return it's path so that way I can easily iterate over whole training set. In that way I always have two training sets downloaded at the same time.
", "rawMarkdown": "That's where I use a function which will download and extract a training set and will return it's path so that way I can easily iterate over whole training set. In that way I always have two training sets downloaded at the same time.", "votes": null }, { "id": "716001", "postDate": "01/11/2020 06:06:17", "content": "I'm using Google Colab + Google Drive nothing else
", "rawMarkdown": "I'm using Google Colab + Google Drive nothing else", "votes": null }, { "id": "716007", "postDate": "01/11/2020 06:17:57", "content": "Ahhh i see. Clever! But a bit slow.
", "rawMarkdown": "Ahhh i see. Clever! But a bit slow.", "votes": null }, { "id": "716042", "postDate": "01/11/2020 07:13:42", "content": "Yeah, but it's the only way for me. I've applied for aws and tpu's let's see what happens.
", "rawMarkdown": "Yeah, but it's the only way for me. I've applied for aws and tpu's let's see what happens.", "votes": null }, { "id": "717510", "postDate": "01/13/2020 08:12:41", "content": "Thank you for sharing! That's really interesting!
", "rawMarkdown": "Thank you for sharing! That's really interesting!", "votes": null }, { "id": "721714", "postDate": "01/17/2020 16:31:40", "content": "Thank you for sharing this. Are you using Google Colab as a way to prototype your model offline to not waste Kaggle's GPU quota and then eventually running your notebook on Kaggle once you feel like you have a good CV score? Or is there some way to submit predictions from Colab that I'm missing?
", "rawMarkdown": "Thank you for sharing this. Are you using Google Colab as a way to prototype your model offline to not waste Kaggle's GPU quota and then eventually running your notebook on Kaggle once you feel like you have a good CV score? Or is there some way to submit predictions from Colab that I'm missing?", "votes": null }, { "id": "721727", "postDate": "01/17/2020 16:44:24", "content": "Colab is my offline GPU, I don't think there is a way to submit from colab to kaggle
", "rawMarkdown": "Colab is my offline GPU, I don't think there is a way to submit from colab to kaggle", "votes": null }, { "id": "727874", "postDate": "01/24/2020 06:24:40", "content": "Great. In case it's helpful for you then you can check out my 'maruti' python package. available via pip(pip3 install maruti). in maruti.deepfake.VideoDataset class you can easily download and group real-fake videos. ( it contains the cookies file with itself so that you don't have to worry).\nAnd one more thing if you're not using is that you can upload the dataset to kaggle directly from google drive. Keep Rocking.
", "rawMarkdown": "Great. In case it's helpful for you then you can check out my 'maruti' python package. available via pip(pip3 install maruti). in maruti.deepfake.VideoDataset class you can easily download and group real-fake videos. ( it contains the cookies file with itself so that you don't have to worry).\nAnd one more thing if you're not using is that you can upload the dataset to kaggle directly from google drive. Keep Rocking.", "votes": null }, { "id": "732636", "postDate": "01/30/2020 03:09:23", "content": "I'm very interested. Can you share the details of this method,Thanks very much!
", "rawMarkdown": "I'm very interested. Can you share the details of this method,Thanks very much!", "votes": null }, { "id": "732997", "postDate": "01/30/2020 14:44:20", "content": "You can just \n!pip3 install maruti \nor\n !pip install maruti\nThen use \ndataset = maruti.VideoDataset('path/to/folder/of/one/training/set')\nto download a training part use\ndataset = maruti.VideoDataset.from_part('00') #specify the part number in strings.\nonce it is done you can use\ndataset.video_groups to get groups of real-fake videos\ndataset.video_paths to get all the paths of videos\ndataset.path to get path of dataset\nthere are many more methods in the package. Sorry for no documentation. But the functions are quite clear in themselves. Get to me for any help.
If you are in need of GPU computing at 10X less cost then checkout: https://www.qblocks.cloud/?ref=qb_activate
\n\nIn $100 you can get a high end GPU instance for the entire month with good memory and storage for AI dev.
", "rawMarkdown": "If you are in need of GPU computing at 10X less cost then checkout: https://www.qblocks.cloud/?ref=qb_activate\n\nIn $100 you can get a high end GPU instance for the entire month with good memory and storage for AI dev.", "votes": null } ], "comments": [ { "id": 715900, "author_name": "moshel", "author_url": "", "post_date": "01/11/2020 00:10:48", "content": "It's highly unlikely, unfortunately. You can play around with the sample training but you won't be able to get high score without the full training set (unless you use non deep learning methods). Try applying for amazon or Google credits (see posts)
", "votes": null, "replies": [] }, { "id": 715942, "author_name": "ankitsainiankit", "author_url": "", "post_date": "01/11/2020 02:26:47", "content": "I am in 300. It may not be impressive but I just want to tell you that I train on full training data using google colaboratory. I have wrote some basic functions to make life easier. When I finally leave it for training I just close my laptop and open it in mobile phone. So with these little tricks I can train on whole data. If you want the functions and how I use them then I can share it here.
", "votes": null, "replies": [ { "id": 715986, "author_name": "moshel", "author_url": "", "post_date": "01/11/2020 05:23:01", "content": "The full training set is 500gb. Afik colab does not provide this kind of storage.
", "votes": null, "replies": [] }, { "id": 715988, "author_name": "ankitsainiankit", "author_url": "", "post_date": "01/11/2020 05:28:30", "content": "That's where I use a function which will download and extract a training set and will return it's path so that way I can easily iterate over whole training set. In that way I always have two training sets downloaded at the same time.
", "votes": null, "replies": [] }, { "id": 716007, "author_name": "moshel", "author_url": "", "post_date": "01/11/2020 06:17:57", "content": "Ahhh i see. Clever! But a bit slow.
", "votes": null, "replies": [] }, { "id": 716042, "author_name": "ankitsainiankit", "author_url": "", "post_date": "01/11/2020 07:13:42", "content": "Yeah, but it's the only way for me. I've applied for aws and tpu's let's see what happens.
", "votes": null, "replies": [] }, { "id": 732636, "author_name": "beeaware", "author_url": "", "post_date": "01/30/2020 03:09:23", "content": "I'm very interested. Can you share the details of this method,Thanks very much!
", "votes": null, "replies": [] }, { "id": 732997, "author_name": "ankitsainiankit", "author_url": "", "post_date": "01/30/2020 14:44:20", "content": "You can just \n!pip3 install maruti \nor\n !pip install maruti\nThen use \ndataset = maruti.VideoDataset('path/to/folder/of/one/training/set')\nto download a training part use\ndataset = maruti.VideoDataset.from_part('00') #specify the part number in strings.\nonce it is done you can use\ndataset.video_groups to get groups of real-fake videos\ndataset.video_paths to get all the paths of videos\ndataset.path to get path of dataset\nthere are many more methods in the package. Sorry for no documentation. But the functions are quite clear in themselves. Get to me for any help.
I'm using Google Colab + Google Drive nothing else
", "votes": null, "replies": [ { "id": 717510, "author_name": "jpmary", "author_url": "", "post_date": "01/13/2020 08:12:41", "content": "Thank you for sharing! That's really interesting!
", "votes": null, "replies": [] }, { "id": 721714, "author_name": "pairwiseseq", "author_url": "", "post_date": "01/17/2020 16:31:40", "content": "Thank you for sharing this. Are you using Google Colab as a way to prototype your model offline to not waste Kaggle's GPU quota and then eventually running your notebook on Kaggle once you feel like you have a good CV score? Or is there some way to submit predictions from Colab that I'm missing?
", "votes": null, "replies": [] }, { "id": 721727, "author_name": "behrouzp", "author_url": "", "post_date": "01/17/2020 16:44:24", "content": "Colab is my offline GPU, I don't think there is a way to submit from colab to kaggle
", "votes": null, "replies": [] }, { "id": 727874, "author_name": "ankitsainiankit", "author_url": "", "post_date": "01/24/2020 06:24:40", "content": "Great. In case it's helpful for you then you can check out my 'maruti' python package. available via pip(pip3 install maruti). in maruti.deepfake.VideoDataset class you can easily download and group real-fake videos. ( it contains the cookies file with itself so that you don't have to worry).\nAnd one more thing if you're not using is that you can upload the dataset to kaggle directly from google drive. Keep Rocking.
", "votes": null, "replies": [] } ] }, { "id": 894628, "author_name": "genesis96839", "author_url": "", "post_date": "06/20/2020 15:51:01", "content": "If you are in need of GPU computing at 10X less cost then checkout: https://www.qblocks.cloud/?ref=qb_activate
\n\nIn $100 you can get a high end GPU instance for the entire month with good memory and storage for AI dev.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "715848": "I'm using a chromebook currently. I don't have storage space for the entire train dataset or the compute resources. I was curious if I can still compete effectively by only relying on Kaggle notebook and my internet connection?", "715900": "It's highly unlikely, unfortunately. You can play around with the sample training but you won't be able to get high score without the full training set (unless you use non deep learning methods). Try applying for amazon or Google credits (see posts)", "715942": "I am in 300. It may not be impressive but I just want to tell you that I train on full training data using google colaboratory. I have wrote some basic functions to make life easier. When I finally leave it for training I just close my laptop and open it in mobile phone. So with these little tricks I can train on whole data. If you want the functions and how I use them then I can share it here.", "715986": "The full training set is 500gb. Afik colab does not provide this kind of storage.", "715988": "That's where I use a function which will download and extract a training set and will return it's path so that way I can easily iterate over whole training set. In that way I always have two training sets downloaded at the same time.", "716001": "I'm using Google Colab + Google Drive nothing else", "716007": "Ahhh i see. Clever! But a bit slow.", "716042": "Yeah, but it's the only way for me. I've applied for aws and tpu's let's see what happens.", "717510": "Thank you for sharing! That's really interesting!", "721714": "Thank you for sharing this. Are you using Google Colab as a way to prototype your model offline to not waste Kaggle's GPU quota and then eventually running your notebook on Kaggle once you feel like you have a good CV score? Or is there some way to submit predictions from Colab that I'm missing?", "721727": "Colab is my offline GPU, I don't think there is a way to submit from colab to kaggle", "727874": "Great. In case it's helpful for you then you can check out my 'maruti' python package. available via pip(pip3 install maruti). in maruti.deepfake.VideoDataset class you can easily download and group real-fake videos. ( it contains the cookies file with itself so that you don't have to worry).\nAnd one more thing if you're not using is that you can upload the dataset to kaggle directly from google drive. Keep Rocking.", "732636": "I'm very interested. Can you share the details of this method,Thanks very much!", "732997": "You can just \n!pip3 install maruti \nor\n !pip install maruti\nThen use \n`dataset = maruti.VideoDataset('path/to/folder/of/one/training/set')`\nto download a training part use\n`dataset = maruti.VideoDataset.from_part('00') #specify the part number in strings.`\nonce it is done you can use\ndataset.video_groups to get groups of real-fake videos\ndataset.video_paths to get all the paths of videos\ndataset.path to get path of dataset\nthere are many more methods in the package. Sorry for no documentation. But the functions are quite clear in themselves. Get to me for any help.", "894628": "If you are in need of GPU computing at 10X less cost then checkout: https://www.qblocks.cloud/?ref=qb_activate\n\nIn $100 you can get a high end GPU instance for the entire month with good memory and storage for AI dev." }, "source": "meta" }