Has anyone tried training a TPU that you paid for on your own GCP project on a Kaggle dataset? If this is possible, it would be great because it avoids having to pay for your own hosting of the data on GCP.
", "messages": [ { "id": "920573", "postDate": "07/08/2020 17:18:18", "content": "Has anyone tried training a TPU that you paid for on your own GCP project on a Kaggle dataset? If this is possible, it would be great because it avoids having to pay for your own hosting of the data on GCP.
", "rawMarkdown": "Has anyone tried training a TPU that you paid for on your own GCP project on a Kaggle dataset? If this is possible, it would be great because it avoids having to pay for your own hosting of the data on GCP.", "votes": null }, { "id": "920588", "postDate": "07/08/2020 17:37:22", "content": "Does that mean we can not use colab TPU to train model on Kaggle data? I couldn't find any way to upload data to GCP storage for free using colab. Probably, it's not possible to use TPUs without GCP account in colab.
", "rawMarkdown": "Does that mean we can not use colab TPU to train model on Kaggle data? I couldn't find any way to upload data to GCP storage for free using colab. Probably, it's not possible to use TPUs without GCP account in colab.", "votes": null }, { "id": "920591", "postDate": "07/08/2020 17:39:15", "content": "I was hoping that the colab TPU could connect to the kaggle dataset in a similar way as the Kaggle TPU.
\n\nIt does get a gs address. Maybe there's a hacky way to pass the storage address to the colab tpu?
", "rawMarkdown": "I was hoping that the colab TPU could connect to the kaggle dataset in a similar way as the Kaggle TPU.\n\nIt does get a gs address. Maybe there's a hacky way to pass the storage address to the colab tpu?", "votes": null }, { "id": "920610", "postDate": "07/08/2020 17:55:17", "content": "I am not sure of that. I tried that today but got error and couldn't find anything on how to connect it to GC storage.
\n\nIf you will find anything on this, then I would appreciate the help.
", "rawMarkdown": "I am not sure of that. I tried that today but got error and couldn't find anything on how to connect it to GC storage. \n\nIf you will find anything on this, then I would appreciate the help.", "votes": null }, { "id": "920639", "postDate": "07/08/2020 18:15:21", "content": "Yes, there is a hacky way to do exactly that.
\n\nRather than tediously spell it all out I'm simply going to give you a link to a working Alaska2 notebook I have on Colab. The actual notebook doesn't give a great leaderboard result. It's a work in progress that I may abandon. It's a fork from xhlulu. https://www.kaggle.com/xhlulu/alaska2-efficientnet-on-tpus\n\nHowever, it does work to use Colab to access the Kaggle data without downloading anything. You'll find the first Epoch takes quite a while but the rest are very much quicker.
\n\n", "rawMarkdown": "Yes, there is a hacky way to do exactly that.\n\nRather than tediously spell it all out I'm simply going to give you a link to a working Alaska2 notebook I have on Colab. The actual notebook doesn't give a great leaderboard result. It's a work in progress that I may abandon. It's a fork from xhlulu. [https://www.kaggle.com/xhlulu/alaska2-efficientnet-on-tpus\n](https://www.kaggle.com/xhlulu/alaska2-efficientnet-on-tpus)\nHowever, it does work to use Colab to access the Kaggle data without downloading anything. You'll find the first Epoch takes quite a while but the rest are very much quicker.\n\n[Alaska2 on Colab TPU.ipynb](https://colab.research.google.com/drive/1Dx9-2j1ZV9JHl0Cae-Uz03fFhauviYEH?usp=sharing)", "votes": null }, { "id": "920648", "postDate": "07/08/2020 18:20:53", "content": "very interesting approach to the problem. If I understand correctly, you mounted the GCS files onto the colab notebook, which I believe would be much faster than downloading 30gb.
\n\nI am guessing you can get the kaggle dataset address when you run a kaggle notebook?
", "rawMarkdown": "very interesting approach to the problem. If I understand correctly, you mounted the GCS files onto the colab notebook, which I believe would be much faster than downloading 30gb. \n\nI am guessing you can get the kaggle dataset address when you run a kaggle notebook?", "votes": null }, { "id": "920657", "postDate": "07/08/2020 18:26:45", "content": "Yes, and you'll have to keep fixing that address as it does expire.
\n\nI have done this for a number of recent projects using jpegs and/or TFRecords with success. The big difference here was that the list of file names had to be constructed by reading in filenames from a directory folder rather than from a csv file. That took a while to work out how to do that using the google filing system.
", "rawMarkdown": "Yes, and you'll have to keep fixing that address as it does expire. \n\nI have done this for a number of recent projects using jpegs and/or TFRecords with success. The big difference here was that the list of file names had to be constructed by reading in filenames from a directory folder rather than from a csv file. That took a while to work out how to do that using the google filing system.", "votes": null }, { "id": "920681", "postDate": "07/08/2020 18:46:37", "content": "So I found a way to transfer a kaggle dataset to my own cloud storage bucket.
\n\ngsutil -m cp -r gs://kds-d7606dbfd8ee47349bbbb9988a9e87859e3b3b51f73e9e96b81f109a gs://kaggledata1
\n\nThis is if you want to use the GCP TPU instead of the Colab or Kaggle TPU
", "rawMarkdown": "So I found a way to transfer a kaggle dataset to my own cloud storage bucket. \n\ngsutil -m cp -r gs://kds-d7606dbfd8ee47349bbbb9988a9e87859e3b3b51f73e9e96b81f109a gs://kaggledata1\n\nThis is if you want to use the GCP TPU instead of the Colab or Kaggle TPU", "votes": null } ], "comments": [ { "id": 920588, "author_name": "urvishp80", "author_url": "", "post_date": "07/08/2020 17:37:22", "content": "Does that mean we can not use colab TPU to train model on Kaggle data? I couldn't find any way to upload data to GCP storage for free using colab. Probably, it's not possible to use TPUs without GCP account in colab.
", "votes": null, "replies": [ { "id": 920591, "author_name": "hooong", "author_url": "", "post_date": "07/08/2020 17:39:15", "content": "I was hoping that the colab TPU could connect to the kaggle dataset in a similar way as the Kaggle TPU.
\n\nIt does get a gs address. Maybe there's a hacky way to pass the storage address to the colab tpu?
", "votes": null, "replies": [] }, { "id": 920610, "author_name": "urvishp80", "author_url": "", "post_date": "07/08/2020 17:55:17", "content": "I am not sure of that. I tried that today but got error and couldn't find anything on how to connect it to GC storage.
\n\nIf you will find anything on this, then I would appreciate the help.
", "votes": null, "replies": [] }, { "id": 920639, "author_name": "mutantspore", "author_url": "", "post_date": "07/08/2020 18:15:21", "content": "Yes, there is a hacky way to do exactly that.
\n\nRather than tediously spell it all out I'm simply going to give you a link to a working Alaska2 notebook I have on Colab. The actual notebook doesn't give a great leaderboard result. It's a work in progress that I may abandon. It's a fork from xhlulu. https://www.kaggle.com/xhlulu/alaska2-efficientnet-on-tpus\n\nHowever, it does work to use Colab to access the Kaggle data without downloading anything. You'll find the first Epoch takes quite a while but the rest are very much quicker.
\n\n", "votes": null, "replies": [] }, { "id": 920648, "author_name": "hooong", "author_url": "", "post_date": "07/08/2020 18:20:53", "content": "very interesting approach to the problem. If I understand correctly, you mounted the GCS files onto the colab notebook, which I believe would be much faster than downloading 30gb.
\n\nI am guessing you can get the kaggle dataset address when you run a kaggle notebook?
", "votes": null, "replies": [] }, { "id": 920657, "author_name": "mutantspore", "author_url": "", "post_date": "07/08/2020 18:26:45", "content": "Yes, and you'll have to keep fixing that address as it does expire.
\n\nI have done this for a number of recent projects using jpegs and/or TFRecords with success. The big difference here was that the list of file names had to be constructed by reading in filenames from a directory folder rather than from a csv file. That took a while to work out how to do that using the google filing system.
", "votes": null, "replies": [] }, { "id": 920681, "author_name": "hooong", "author_url": "", "post_date": "07/08/2020 18:46:37", "content": "So I found a way to transfer a kaggle dataset to my own cloud storage bucket.
\n\ngsutil -m cp -r gs://kds-d7606dbfd8ee47349bbbb9988a9e87859e3b3b51f73e9e96b81f109a gs://kaggledata1
\n\nThis is if you want to use the GCP TPU instead of the Colab or Kaggle TPU
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "920573": "Has anyone tried training a TPU that you paid for on your own GCP project on a Kaggle dataset? If this is possible, it would be great because it avoids having to pay for your own hosting of the data on GCP.", "920588": "Does that mean we can not use colab TPU to train model on Kaggle data? I couldn't find any way to upload data to GCP storage for free using colab. Probably, it's not possible to use TPUs without GCP account in colab.", "920591": "I was hoping that the colab TPU could connect to the kaggle dataset in a similar way as the Kaggle TPU.\n\nIt does get a gs address. Maybe there's a hacky way to pass the storage address to the colab tpu?", "920610": "I am not sure of that. I tried that today but got error and couldn't find anything on how to connect it to GC storage. \n\nIf you will find anything on this, then I would appreciate the help.", "920639": "Yes, there is a hacky way to do exactly that.\n\nRather than tediously spell it all out I'm simply going to give you a link to a working Alaska2 notebook I have on Colab. The actual notebook doesn't give a great leaderboard result. It's a work in progress that I may abandon. It's a fork from xhlulu. [https://www.kaggle.com/xhlulu/alaska2-efficientnet-on-tpus\n](https://www.kaggle.com/xhlulu/alaska2-efficientnet-on-tpus)\nHowever, it does work to use Colab to access the Kaggle data without downloading anything. You'll find the first Epoch takes quite a while but the rest are very much quicker.\n\n[Alaska2 on Colab TPU.ipynb](https://colab.research.google.com/drive/1Dx9-2j1ZV9JHl0Cae-Uz03fFhauviYEH?usp=sharing)", "920648": "very interesting approach to the problem. If I understand correctly, you mounted the GCS files onto the colab notebook, which I believe would be much faster than downloading 30gb. \n\nI am guessing you can get the kaggle dataset address when you run a kaggle notebook?", "920657": "Yes, and you'll have to keep fixing that address as it does expire. \n\nI have done this for a number of recent projects using jpegs and/or TFRecords with success. The big difference here was that the list of file names had to be constructed by reading in filenames from a directory folder rather than from a csv file. That took a while to work out how to do that using the google filing system.", "920681": "So I found a way to transfer a kaggle dataset to my own cloud storage bucket. \n\ngsutil -m cp -r gs://kds-d7606dbfd8ee47349bbbb9988a9e87859e3b3b51f73e9e96b81f109a gs://kaggledata1\n\nThis is if you want to use the GCP TPU instead of the Colab or Kaggle TPU" }, "source": "meta" }