![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3353913%2Fc4e60a8a529fc5adf70a88055956f9a8%2Fslowload.png?generation=1598089772938500&alt=media\")
Hello I am here looking for help. As the GPU quota is easy to reach I decided to run my code on other platforms, however, I am severely bottlenecked by the loading of sound files.
\nthe data loaders are forked from https://www.kaggle.com/hidehisaarai1213/introduction-to-sound-event-detection
\nif I load data in a Kaggle notebook It takes me 18 seconds to load 1000 sound files.
\nif I run the EXACT same code, on a google cloud platform with the same specs as a Kaggle kernel (15 ram 4 cores) it takes me 48.2 seconds.
\nOn google colab this is even worse, where it takes me 1 minute 34 secs to load the same 1000 files (and i already copied files to my current runtime).
\nAll 3 platforms have sound file v 0.10.3.
\nThis makes training a pain. Do you guys know what causes this? How do you guys store/load your data?
\n
Hello I am here looking for help. As the GPU quota is easy to reach I decided to run my code on other platforms, however, I am severely bottlenecked by the loading of sound files.
\nthe data loaders are forked from https://www.kaggle.com/hidehisaarai1213/introduction-to-sound-event-detection
\nif I load data in a Kaggle notebook It takes me 18 seconds to load 1000 sound files.
\nif I run the EXACT same code, on a google cloud platform with the same specs as a Kaggle kernel (15 ram 4 cores) it takes me 48.2 seconds.
\nOn google colab this is even worse, where it takes me 1 minute 34 secs to load the same 1000 files (and i already copied files to my current runtime).
\nAll 3 platforms have sound file v 0.10.3.
\nThis makes training a pain. Do you guys know what causes this? How do you guys store/load your data?
\n
Are you using SSD?
", "rawMarkdown": "Are you using SSD?", "votes": 3, "replies": [ { "id": 981285, "postDate": "2020-08-22T10:46:58.467Z", "content": "ohh that is a good point.. i didn't even consider that, haha. I just looked it up, on my gcloud instance I just chose the standard HDD, I will try attaching a ssd and moving my data there now. I am not sure that's something you can do on colab.
\ntyvm for your help. i was hesitant to ask but I wouldn't have figured this out haha
", "rawMarkdown": "ohh that is a good point.. i didn't even consider that, haha. I just looked it up, on my gcloud instance I just chose the standard HDD, I will try attaching a ssd and moving my data there now. I am not sure that's something you can do on colab.\n\ntyvm for your help. i was hesitant to ask but I wouldn't have figured this out haha" }, { "id": 981303, "postDate": "2020-08-22T11:02:11.103Z", "content": "I have the same issue on colab, I wonder if the storage is a SSD or HDD, does anyone know ?
", "rawMarkdown": "I have the same issue on colab, I wonder if the storage is a SSD or HDD, does anyone know ?" }, { "id": 981314, "postDate": "2020-08-22T11:16:02.877Z", "content": "it seems like on colab it's HDD, my kernel on colab takes ages too
", "rawMarkdown": "it seems like on colab it's HDD, my kernel on colab takes ages too" }, { "id": 981371, "postDate": "2020-08-22T11:53:49.793Z", "content": "yep its an issue with the drive.. i just set up a new gcloud instance with an ssd and ran the same code again:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3353913%2Fe1dd1d2436cb5708f5d9cbda418debf0%2Fgcloud%20ssd.png?generation=1598097218099970&alt=media\")
this gives me the same speed as kaggle
\nalso, from what I can tell there is no way to do the same with colab
", "rawMarkdown": "yep its an issue with the drive.. i just set up a new gcloud instance with an ssd and ran the same code again:\n\n\n\n\n\n\nthis gives me the same speed as kaggle\n\n\n\nalso, from what I can tell there is no way to do the same with colab", "votes": 4 }, { "id": 981746, "postDate": "2020-08-22T16:58:44.210Z", "content": "Thanks for the information. At least I know why the code was so long in colab (5h instead of 20 min per epoch .. ). I thought I was doing something wrong.
", "rawMarkdown": "Thanks for the information. At least I know why the code was so long in colab (5h instead of 20 min per epoch .. ). I thought I was doing something wrong. " }, { "id": 981899, "postDate": "2020-08-22T20:15:35.047Z", "content": "@proletheus Can you please share how you moved the data onto Google Cloud. Did you have to upload all of your data into a bucket, or is there a way to point to multiple datasets from Kaggle and use it from there? For example, I want to use https://www.kaggle.com/rohanrao/xeno-canto-bird-recordings-extended-a-m as external data, do I need to upload it myself or is there a better way? Thank you
", "rawMarkdown": "@proletheus Can you please share how you moved the data onto Google Cloud. Did you have to upload all of your data into a bucket, or is there a way to point to multiple datasets from Kaggle and use it from there? For example, I want to use https://www.kaggle.com/rohanrao/xeno-canto-bird-recordings-extended-a-m as external data, do I need to upload it myself or is there a better way? Thank you", "votes": 1 }, { "id": 981911, "postDate": "2020-08-22T20:42:43.340Z", "content": "yes use the kaggle api
\nhttps://github.com/Kaggle/kaggle-api
\nbasically:
\n!pip install kaggle
\nget the kaggle.json file from your profile and upload it.
\nthen call something like:
\n!kaggle competitions list
\n(this will give you an error and tell you where you have to move the kaggle.json file haha)
\nmove kaggle.json there
then you can copy the API command of whatever you want to download (its the small dots on top of each other in the right corner)
\n!kaggle datasets download -d rohanrao/xeno-canto-bird-recordings-extended-a-m
\nit will download it to your cloud instance. you can then move it to a bucket for long term storage but I don't think that's necessary
\nif you have any issues setting up gcloud let me know
", "rawMarkdown": "yes use the kaggle api\n\nhttps://github.com/Kaggle/kaggle-api\n\n\nbasically:\n\n!pip install kaggle\n\nget the kaggle.json file from your profile and upload it.\n\nthen call something like:\n\n!kaggle competitions list\n\n(this will give you an error and tell you where you have to move the kaggle.json file haha)\nmove kaggle.json there\n\nthen you can copy the API command of whatever you want to download (its the small dots on top of each other in the right corner)\n\n!kaggle datasets download -d rohanrao/xeno-canto-bird-recordings-extended-a-m\n\n\nit will download it to your cloud instance. you can then move it to a bucket for long term storage but I don't think that's necessary\n\n\n\nif you have any issues setting up gcloud let me know", "votes": 2 }, { "id": 981914, "postDate": "2020-08-22T20:48:21.780Z", "content": "@proletheus Please can you tell me does this also work with Kernel Output file? For example, I have a kernel that resamples that external dataset and changes it from .mp3 into .wav … is there a way to get that kernel output file into GCloud? Or do I need a separate dataset for each?
", "rawMarkdown": "@proletheus Please can you tell me does this also work with Kernel Output file? For example, I have a kernel that resamples that external dataset and changes it from .mp3 into .wav ... is there a way to get that kernel output file into GCloud? Or do I need a separate dataset for each?" }, { "id": 981918, "postDate": "2020-08-22T20:51:16.170Z", "content": "you can go to any notebook and click the 3 dots in the top right corner -> copy API command:
\nkaggle kernels pull chanhu/training-bird-simple-baseline
\npulls the jupyter notebook.
\nyou want the kernel output however so change it to
kaggle kernels output chanhu/training-bird-simple-baseline
", "rawMarkdown": "you can go to any notebook and click the 3 dots in the top right corner -> copy API command:\n\nkaggle kernels pull chanhu/training-bird-simple-baseline\n\npulls the jupyter notebook.\nyou want the kernel output however so change it to \n\nkaggle kernels output chanhu/training-bird-simple-baseline\n\n\n", "votes": 1 }, { "id": 981919, "postDate": "2020-08-22T20:53:17.747Z", "content": "I love you bro. I will try this and get back to you. I assume I only need to do this once, since the SSD will persist even if I stop the instance? Maybe can you recommend me the settings you used when creating the instance?
", "rawMarkdown": "I love you bro. I will try this and get back to you. I assume I only need to do this once, since the SSD will persist even if I stop the instance? Maybe can you recommend me the settings you used when creating the instance?" }, { "id": 981920, "postDate": "2020-08-22T20:55:32.807Z", "content": "yes, the SSD will persist. right now I have everything on a bucket, then I just made a new VM and copied everything to that VM, that's how I was able to try so fast:P
\nI think you could also put everything on the SSD and then mount that disk to your instances. - I haven't tried that
", "rawMarkdown": "yes, the SSD will persist. right now I have everything on a bucket, then I just made a new VM and copied everything to that VM, that's how I was able to try so fast:P\n\nI think you could also put everything on the SSD and then mount that disk to your instances. - I haven't tried that", "votes": 1 }, { "id": 994950, "postDate": "2020-09-02T04:54:41.190Z", "rawMarkdown": "", "isDeleted": true }, { "id": 996301, "postDate": "2020-09-03T07:51:36.297Z", "content": "is the storage on colab really SSD? I ran this command !lsblk -d -o name,rota and got 0, which means SSD.
i have the same prob. Time in Colab > 3 times than Kaggle. If use GCS , speed will be the same. Maybe because harddisk in Colab is HDD, Kaggle in SSD.
", "rawMarkdown": "i have the same prob. Time in Colab > 3 times than Kaggle. If use GCS , speed will be the same. Maybe because harddisk in Colab is HDD, Kaggle in SSD. ", "votes": 1 }, { "id": 996416, "postDate": "2020-09-03T09:23:55.753Z", "content": "I may have found the issue, if you upgrade librosa on colab, it seems to work better. the librosa version on colab is 0.5.X, on kaggle 0.8.0. I haven't benchmark against Kaggle as I can't upload all my data on it, but it seems to help when upgrading librosa.
", "rawMarkdown": "I may have found the issue, if you upgrade librosa on colab, it seems to work better. the librosa version on colab is 0.5.X, on kaggle 0.8.0. I haven't benchmark against Kaggle as I can't upload all my data on it, but it seems to help when upgrading librosa." }, { "id": 995110, "postDate": "2020-09-02T07:07:32.253Z", "content": "If one use fuse-zip or similar to save space and read from the mounted drive, there you have another reason.
", "rawMarkdown": "If one use fuse-zip or similar to save space and read from the mounted drive, there you have another reason." }, { "id": 982850, "postDate": "2020-08-23T18:01:37.240Z", "content": "That's a huge bummer :/
\nOne epoch to train on Kaggle ~ 12 minutes
\nOne epoch to train on Colab ~ 3.5 hours
Is there any way to make training viable on Colab?
", "rawMarkdown": "That's a huge bummer :/\nOne epoch to train on Kaggle ~ 12 minutes\nOne epoch to train on Colab ~ 3.5 hours\n\nIs there any way to make training viable on Colab?", "replies": [ { "id": 982899, "postDate": "2020-08-23T19:25:23.967Z", "content": "i suppose there is an other issue going on then:p for me epochs go from 10ish minutes to 40, so roughly a 4-5x decrease in speed.
\nwhat gpu you have on colab?
", "rawMarkdown": "i suppose there is an other issue going on then:p for me epochs go from 10ish minutes to 40, so roughly a 4-5x decrease in speed.\n\nwhat gpu you have on colab? " }, { "id": 984264, "postDate": "2020-08-25T02:10:58.203Z", "content": "hmm… this is strange, considering we have literally the same code running on the kaggle kernels and the instance, GPU is Tesla P100
\nThere is a bottleneck in reading using librosa in the dataloader for me, trying to work my way around that for now.
Are you using SSD?
", "votes": 3, "replies": [ { "id": 981285, "author_name": "Proletheus", "author_url": "", "post_date": "2020-08-22T10:46:58.467000", "content": "ohh that is a good point.. i didn't even consider that, haha. I just looked it up, on my gcloud instance I just chose the standard HDD, I will try attaching a ssd and moving my data there now. I am not sure that's something you can do on colab.
\ntyvm for your help. i was hesitant to ask but I wouldn't have figured this out haha
", "votes": 0, "replies": [] }, { "id": 981303, "author_name": "Shiro", "author_url": "", "post_date": "2020-08-22T11:02:11.103000", "content": "I have the same issue on colab, I wonder if the storage is a SSD or HDD, does anyone know ?
", "votes": 0, "replies": [] }, { "id": 981314, "author_name": "Phi", "author_url": "", "post_date": "2020-08-22T11:16:02.877000", "content": "it seems like on colab it's HDD, my kernel on colab takes ages too
", "votes": 0, "replies": [] }, { "id": 981371, "author_name": "Proletheus", "author_url": "", "post_date": "2020-08-22T11:53:49.793000", "content": "yep its an issue with the drive.. i just set up a new gcloud instance with an ssd and ran the same code again:
\n
this gives me the same speed as kaggle
\nalso, from what I can tell there is no way to do the same with colab
", "votes": 4, "replies": [] }, { "id": 981746, "author_name": "Shiro", "author_url": "", "post_date": "2020-08-22T16:58:44.210000", "content": "Thanks for the information. At least I know why the code was so long in colab (5h instead of 20 min per epoch .. ). I thought I was doing something wrong.
", "votes": 0, "replies": [] }, { "id": 981899, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2020-08-22T20:15:35.047000", "content": "@proletheus Can you please share how you moved the data onto Google Cloud. Did you have to upload all of your data into a bucket, or is there a way to point to multiple datasets from Kaggle and use it from there? For example, I want to use https://www.kaggle.com/rohanrao/xeno-canto-bird-recordings-extended-a-m as external data, do I need to upload it myself or is there a better way? Thank you
", "votes": 1, "replies": [] }, { "id": 981911, "author_name": "Proletheus", "author_url": "", "post_date": "2020-08-22T20:42:43.340000", "content": "yes use the kaggle api
\nhttps://github.com/Kaggle/kaggle-api
\nbasically:
\n!pip install kaggle
\nget the kaggle.json file from your profile and upload it.
\nthen call something like:
\n!kaggle competitions list
\n(this will give you an error and tell you where you have to move the kaggle.json file haha)
\nmove kaggle.json there
then you can copy the API command of whatever you want to download (its the small dots on top of each other in the right corner)
\n!kaggle datasets download -d rohanrao/xeno-canto-bird-recordings-extended-a-m
\nit will download it to your cloud instance. you can then move it to a bucket for long term storage but I don't think that's necessary
\nif you have any issues setting up gcloud let me know
", "votes": 2, "replies": [] }, { "id": 981914, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2020-08-22T20:48:21.780000", "content": "@proletheus Please can you tell me does this also work with Kernel Output file? For example, I have a kernel that resamples that external dataset and changes it from .mp3 into .wav … is there a way to get that kernel output file into GCloud? Or do I need a separate dataset for each?
", "votes": 0, "replies": [] }, { "id": 981918, "author_name": "Proletheus", "author_url": "", "post_date": "2020-08-22T20:51:16.170000", "content": "you can go to any notebook and click the 3 dots in the top right corner -> copy API command:
\nkaggle kernels pull chanhu/training-bird-simple-baseline
\npulls the jupyter notebook.
\nyou want the kernel output however so change it to
kaggle kernels output chanhu/training-bird-simple-baseline
", "votes": 1, "replies": [] }, { "id": 981919, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2020-08-22T20:53:17.747000", "content": "I love you bro. I will try this and get back to you. I assume I only need to do this once, since the SSD will persist even if I stop the instance? Maybe can you recommend me the settings you used when creating the instance?
", "votes": 0, "replies": [] }, { "id": 981920, "author_name": "Proletheus", "author_url": "", "post_date": "2020-08-22T20:55:32.807000", "content": "yes, the SSD will persist. right now I have everything on a bucket, then I just made a new VM and copied everything to that VM, that's how I was able to try so fast:P
\nI think you could also put everything on the SSD and then mount that disk to your instances. - I haven't tried that
", "votes": 1, "replies": [] }, { "id": 994950, "author_name": "", "author_url": "", "post_date": "2020-09-02T04:54:41.190000", "content": "", "votes": 0, "replies": [] }, { "id": 996301, "author_name": "Camaro", "author_url": "", "post_date": "2020-09-03T07:51:36.297000", "content": "is the storage on colab really SSD? I ran this command !lsblk -d -o name,rota and got 0, which means SSD.
i have the same prob. Time in Colab > 3 times than Kaggle. If use GCS , speed will be the same. Maybe because harddisk in Colab is HDD, Kaggle in SSD.
", "votes": 1, "replies": [] }, { "id": 996416, "author_name": "Shiro", "author_url": "", "post_date": "2020-09-03T09:23:55.753000", "content": "I may have found the issue, if you upgrade librosa on colab, it seems to work better. the librosa version on colab is 0.5.X, on kaggle 0.8.0. I haven't benchmark against Kaggle as I can't upload all my data on it, but it seems to help when upgrading librosa.
", "votes": 0, "replies": [] }, { "id": 995110, "author_name": "Kirderf", "author_url": "", "post_date": "2020-09-02T07:07:32.253000", "content": "If one use fuse-zip or similar to save space and read from the mounted drive, there you have another reason.
", "votes": 0, "replies": [] }, { "id": 982850, "author_name": "Saransh Agarwal", "author_url": "", "post_date": "2020-08-23T18:01:37.240000", "content": "That's a huge bummer :/
\nOne epoch to train on Kaggle ~ 12 minutes
\nOne epoch to train on Colab ~ 3.5 hours
Is there any way to make training viable on Colab?
", "votes": 0, "replies": [ { "id": 982899, "author_name": "Proletheus", "author_url": "", "post_date": "2020-08-23T19:25:23.967000", "content": "i suppose there is an other issue going on then:p for me epochs go from 10ish minutes to 40, so roughly a 4-5x decrease in speed.
\nwhat gpu you have on colab?
", "votes": 0, "replies": [] }, { "id": 984264, "author_name": "Saransh Agarwal", "author_url": "", "post_date": "2020-08-25T02:10:58.203000", "content": "hmm… this is strange, considering we have literally the same code running on the kaggle kernels and the instance, GPU is Tesla P100
\nThere is a bottleneck in reading using librosa in the dataloader for me, trying to work my way around that for now.