1、Put the data into googledrive, and use it as a local, and do googlecolaboratory. ＝＞2、Do it via gcp.
\n2、Do it via gcp. upload it to git, connect it to gcp's cui, and create a virtual environment. ＝＝＞It's too much work, and it's hard to do with vim and tmux.
\n3, via kaggle. The communication behavior is too slow, and I fall asleep.
\nDo I have to give up on this?
\nI don't want to give up.
Is it possible to use googlecorabo?
\nSomeone told me to make it small, but what if I want to make it all? I thought. Do you guys use kaggle kernel?
\nSorry. I don't know.
\nIt takes me about 3 hours to put the first one of 161616*143 data, so it takes me 48 hours to put all of them. If I put in too much data, I get an error.
\nWhat should I do with the file name?
", "messages": [ { "id": 1459339, "postDate": "2021-08-08T10:27:38.580Z", "content": "1、Put the data into googledrive, and use it as a local, and do googlecolaboratory. ＝＞2、Do it via gcp.
\n2、Do it via gcp. upload it to git, connect it to gcp's cui, and create a virtual environment. ＝＝＞It's too much work, and it's hard to do with vim and tmux.
\n3, via kaggle. The communication behavior is too slow, and I fall asleep.
\nDo I have to give up on this?
\nI don't want to give up.
Is it possible to use googlecorabo?
\nSomeone told me to make it small, but what if I want to make it all? I thought. Do you guys use kaggle kernel?
\nSorry. I don't know.
\nIt takes me about 3 hours to put the first one of 161616*143 data, so it takes me 48 hours to put all of them. If I put in too much data, I get an error.
\nWhat should I do with the file name?
", "rawMarkdown": "1、Put the data into googledrive, and use it as a local, and do googlecolaboratory. ＝＞2、Do it via gcp.\n\n2、Do it via gcp. upload it to git, connect it to gcp's cui, and create a virtual environment. ＝＝＞It's too much work, and it's hard to do with vim and tmux.\n\n3, via kaggle. The communication behavior is too slow, and I fall asleep.\n\n\nDo I have to give up on this?\nI don't want to give up.\n\n\nIs it possible to use googlecorabo?\n\nSomeone told me to make it small, but what if I want to make it all? I thought. Do you guys use kaggle kernel?\n\nSorry. I don't know.\n\nIt takes me about 3 hours to put the first one of 16*16*16*143 data, so it takes me 48 hours to put all of them. If I put in too much data, I get an error.\n\n\n\n\nWhat should I do with the file name?", "votes": 4 }, { "id": 1460515, "postDate": "2021-08-08T23:34:04.007Z", "content": "Using TPUs and TFRecords I can train a single epoch in about 4m. See the discussion Things I've tried and how they worked out for more details.
", "rawMarkdown": "Using TPUs and TFRecords I can train a single epoch in about 4m. See the discussion [Things I've tried and how they worked out](https://www.kaggle.com/c/g2net-gravitational-wave-detection/discussion/261721) for more details. ", "votes": 2 }, { "id": 1459957, "postDate": "2021-08-08T16:12:25.593Z", "content": "You can use TF with TPU. First, prepare (or use existing) dataset with TFRecords. Second, use Kaggle or Colab to train you model.
\nUse KaggleDatasets() to get GCS_DS_PATH on Kaggle. If you want to use Colab, you can simply use GCS_DS_PATH from Kaggle (just print it). This way you won't need to upload your data anywhere. Please note that for this to work, dataset with TFRecords should be public (there's a way to use TPU with private datasets, but it's somewhat complicated, and I haven't tried that).
Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris
There's a similar discussion here How to kagglers handle big data
", "rawMarkdown": "There's a similar discussion here[ How to kagglers handle big data](https://www.kaggle.com/c/g2net-gravitational-wave-detection/discussion/261414)" }, { "id": 1462713, "postDate": "2021-08-10T01:26:10.833Z", "content": "Try tf_records.
\nThere is a method I have devised myself.
\nIt is to set the colab extension to 2TB, cd the ls, bring it to where I want it to happen, and put the data in all at once. This is really good, it's like attacking with cui.
I'll be referring to this.
\nThanks again. I'll take number one.
", "rawMarkdown": "Try tf_records.\n\nThere is a method I have devised myself.\nIt is to set the colab extension to 2TB, cd the ls, bring it to where I want it to happen, and put the data in all at once. This is really good, it's like attacking with cui.\n\nI'll be referring to this.\n\nThanks again. I'll take number one." }, { "id": 1461317, "postDate": "2021-08-09T10:44:09.573Z", "rawMarkdown": "", "votes": 4, "isDeleted": true }, { "id": 1463424, "postDate": "2021-08-10T07:27:13.193Z", "content": "Thank you so much to all three of you.
", "rawMarkdown": "Thank you so much to all three of you." } ], "comments": [ { "id": 1460515, "author_name": "Fractal Feelings", "author_url": "", "post_date": "2021-08-08T23:34:04.007000", "content": "Using TPUs and TFRecords I can train a single epoch in about 4m. See the discussion Things I've tried and how they worked out for more details.
", "votes": 2, "replies": [] }, { "id": 1459957, "author_name": "Araik Tamazian", "author_url": "", "post_date": "2021-08-08T16:12:25.593000", "content": "You can use TF with TPU. First, prepare (or use existing) dataset with TFRecords. Second, use Kaggle or Colab to train you model.
\nUse KaggleDatasets() to get GCS_DS_PATH on Kaggle. If you want to use Colab, you can simply use GCS_DS_PATH from Kaggle (just print it). This way you won't need to upload your data anywhere. Please note that for this to work, dataset with TFRecords should be public (there's a way to use TPU with private datasets, but it's somewhat complicated, and I haven't tried that).
Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris
There's a similar discussion here How to kagglers handle big data
", "votes": 0, "replies": [] }, { "id": 1462713, "author_name": "kou yamazaki at Hokkaido", "author_url": "", "post_date": "2021-08-10T01:26:10.833000", "content": "Try tf_records.
\nThere is a method I have devised myself.
\nIt is to set the colab extension to 2TB, cd the ls, bring it to where I want it to happen, and put the data in all at once. This is really good, it's like attacking with cui.
I'll be referring to this.
\nThanks again. I'll take number one.
", "votes": 0, "replies": [] }, { "id": 1461317, "author_name": "", "author_url": "", "post_date": "2021-08-09T10:44:09.573000", "content": "", "votes": 4, "replies": [] }, { "id": 1463424, "author_name": "kou yamazaki at Hokkaido", "author_url": "", "post_date": "2021-08-10T07:27:13.193000", "content": "Thank you so much to all three of you.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1459339": "1、Put the data into googledrive, and use it as a local, and do googlecolaboratory. ＝＞2、Do it via gcp.\n\n2、Do it via gcp. upload it to git, connect it to gcp's cui, and create a virtual environment. ＝＝＞It's too much work, and it's hard to do with vim and tmux.\n\n3, via kaggle. The communication behavior is too slow, and I fall asleep.\n\n\nDo I have to give up on this?\nI don't want to give up.\n\n\nIs it possible to use googlecorabo?\n\nSomeone told me to make it small, but what if I want to make it all? I thought. Do you guys use kaggle kernel?\n\nSorry. I don't know.\n\nIt takes me about 3 hours to put the first one of 16*16*16*143 data, so it takes me 48 hours to put all of them. If I put in too much data, I get an error.\n\n\n\n\nWhat should I do with the file name?", "1460515": "Using TPUs and TFRecords I can train a single epoch in about 4m. See the discussion [Things I've tried and how they worked out](https://www.kaggle.com/c/g2net-gravitational-wave-detection/discussion/261721) for more details. ", "1459957": "You can use TF with TPU. First, prepare (or use existing) dataset with TFRecords. Second, use Kaggle or Colab to train you model. \n\nUse `KaggleDatasets()` to get `GCS_DS_PATH ` on Kaggle. If you want to use Colab, you can simply use `GCS_DS_PATH ` from Kaggle (just print it). This way you won't need to upload your data anywhere. Please note that for this to work, dataset with TFRecords should be public (there's a way to use TPU with private datasets, but it's somewhat complicated, and I haven't tried that).", "1561204": "Hey All,\n\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.\n\nThank you all,\n\nRegards,\nChris", "1463537": "There's a similar discussion here[ How to kagglers handle big data](https://www.kaggle.com/c/g2net-gravitational-wave-detection/discussion/261414)", "1462713": "Try tf_records.\n\nThere is a method I have devised myself.\nIt is to set the colab extension to 2TB, cd the ls, bring it to where I want it to happen, and put the data in all at once. This is really good, it's like attacking with cui.\n\nI'll be referring to this.\n\nThanks again. I'll take number one.", "1461317": "", "1463424": "Thank you so much to all three of you." } }