This is a huge dataset and usually few GPU hours are going in training the model. If we are participating in multiple competition, we know how precious GPU hours are on Kaggle.
\nIt is better to train the model in Colab and use the submission file in Kaggle.
\nFew helpful links below
Another useful way is to use parquet dataset prepared by Saurabh (@saurabhshahane) below. Thanks Saurabh!
\nThis is a huge dataset and usually few GPU hours are going in training the model. If we are participating in multiple competition, we know how precious GPU hours are on Kaggle.
\nIt is better to train the model in Colab and use the submission file in Kaggle.
\nFew helpful links below
Another useful way is to use parquet dataset prepared by Saurabh (@saurabhshahane) below. Thanks Saurabh!
\nHow are guys going about Colab's small disk size? I see 30 GB available, not enough to train on the entire (compressed) dataset. For example, @luigisaetta's TFRecords take 41 GB.
\nI tried to use my google drive, but it does not handle larger number of files. The files unzipped to gdrive are somewhere buffered (os.path.exist() == True), but never arrive and are lost on a new VM.
Thank you very much. This is very helpful!
", "rawMarkdown": "Thank you very much. This is very helpful!", "votes": 1, "replies": [ { "id": 1265272, "postDate": "2021-04-06T18:40:54.057Z", "content": "Sure thing!
", "rawMarkdown": "Sure thing!" } ] }, { "id": 1263413, "postDate": "2021-04-05T11:54:47.463Z", "content": "You can directly submit a submission from colab itself using the API token as far as I know.
\nI personally prefer Colab over Kaggle Notebooks too.
Ok that is great, I knew that we can get the data and run python code to get final submission file but was not aware that we can make direct submission from colab to kaggle. Thanks for sharing!
", "rawMarkdown": "Ok that is great, I knew that we can get the data and run python code to get final submission file but was not aware that we can make direct submission from colab to kaggle. Thanks for sharing!" } ] }, { "id": 1256830, "postDate": "2021-03-30T09:29:30.590Z", "content": "This is helpful. Thank you @saurabhbagchi for sharing.
", "rawMarkdown": "This is helpful. Thank you @saurabhbagchi for sharing.", "votes": 1 }, { "id": 1255163, "postDate": "2021-03-28T14:11:14.907Z", "content": "I wonder how all you guys succeed using GPU. For me, with TPU I'm still not able to train on the entire dataset (exceeding TPU session time). In my team we have defined a strategy to split the training in several step, and should start working on the entire dataset by today evening.
\nIn the meantime, since it is not off topic, I remember that I have reduced the images to 256x256 and packed in TFRecords. The dataset is: Herb-256x256-TfRecords.
", "rawMarkdown": "I wonder how all you guys succeed using GPU. For me, with TPU I'm still not able to train on the entire dataset (exceeding TPU session time). In my team we have defined a strategy to split the training in several step, and should start working on the entire dataset by today evening.\n\nIn the meantime, since it is not off topic, I remember that I have reduced the images to 256x256 and packed in TFRecords. The dataset is: [Herb-256x256-TfRecords](https://www.kaggle.com/luigisaetta/herb2021-256).", "votes": 1, "replies": [ { "id": 1255164, "postDate": "2021-03-28T14:14:52.593Z", "content": "Thanks @luigisaetta, this is very helpful!
", "rawMarkdown": "Thanks @luigisaetta, this is very helpful!" } ] }, { "id": 1255060, "postDate": "2021-03-28T11:40:06.917Z", "content": "This is helpful. Thank you @saurabhbagchi for sharing. This is an important toolset.
", "rawMarkdown": "This is helpful. Thank you @saurabhbagchi for sharing. This is an important toolset.", "votes": 1, "replies": [ { "id": 1255162, "postDate": "2021-03-28T14:11:12.283Z", "content": "Most welcome @olusesiadebisi !
", "rawMarkdown": "Most welcome @olusesiadebisi !" } ] }, { "id": 1311377, "postDate": "2021-05-17T11:12:38.113Z", "content": "thanks for the info!
", "rawMarkdown": "thanks for the info!", "votes": 1 }, { "id": 1288017, "postDate": "2021-04-29T15:52:07.757Z", "content": "Thanks for giving alternative option
", "rawMarkdown": "Thanks for giving alternative option", "votes": 1 }, { "id": 1285238, "postDate": "2021-04-26T17:53:45.873Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!", "votes": 1 }, { "id": 1270124, "postDate": "2021-04-11T10:11:01.690Z", "content": "Thank you.
", "rawMarkdown": "Thank you.", "votes": 1 } ], "comments": [ { "id": 1284958, "author_name": "Marius Wanko", "author_url": "", "post_date": "2021-04-26T12:59:31.203000", "content": "How are guys going about Colab's small disk size? I see 30 GB available, not enough to train on the entire (compressed) dataset. For example, @luigisaetta's TFRecords take 41 GB.
\nI tried to use my google drive, but it does not handle larger number of files. The files unzipped to gdrive are somewhere buffered (os.path.exist() == True), but never arrive and are lost on a new VM.
Thank you very much. This is very helpful!
", "votes": 1, "replies": [ { "id": 1265272, "author_name": "Old Monk", "author_url": "", "post_date": "2021-04-06T18:40:54.057000", "content": "Sure thing!
", "votes": 0, "replies": [] } ] }, { "id": 1263413, "author_name": "Manas Vardhan", "author_url": "", "post_date": "2021-04-05T11:54:47.463000", "content": "You can directly submit a submission from colab itself using the API token as far as I know.
\nI personally prefer Colab over Kaggle Notebooks too.
Ok that is great, I knew that we can get the data and run python code to get final submission file but was not aware that we can make direct submission from colab to kaggle. Thanks for sharing!
", "votes": 0, "replies": [] } ] }, { "id": 1256830, "author_name": "Sumit Gahlot", "author_url": "", "post_date": "2021-03-30T09:29:30.590000", "content": "This is helpful. Thank you @saurabhbagchi for sharing.
", "votes": 1, "replies": [] }, { "id": 1255163, "author_name": "Luigi Saetta", "author_url": "", "post_date": "2021-03-28T14:11:14.907000", "content": "I wonder how all you guys succeed using GPU. For me, with TPU I'm still not able to train on the entire dataset (exceeding TPU session time). In my team we have defined a strategy to split the training in several step, and should start working on the entire dataset by today evening.
\nIn the meantime, since it is not off topic, I remember that I have reduced the images to 256x256 and packed in TFRecords. The dataset is: Herb-256x256-TfRecords.
", "votes": 1, "replies": [ { "id": 1255164, "author_name": "Old Monk", "author_url": "", "post_date": "2021-03-28T14:14:52.593000", "content": "Thanks @luigisaetta, this is very helpful!
", "votes": 0, "replies": [] } ] }, { "id": 1255060, "author_name": "Olusesi Adebisi", "author_url": "", "post_date": "2021-03-28T11:40:06.917000", "content": "This is helpful. Thank you @saurabhbagchi for sharing. This is an important toolset.
", "votes": 1, "replies": [ { "id": 1255162, "author_name": "Old Monk", "author_url": "", "post_date": "2021-03-28T14:11:12.283000", "content": "Most welcome @olusesiadebisi !
", "votes": 0, "replies": [] } ] }, { "id": 1311377, "author_name": "Kushagra Singh", "author_url": "", "post_date": "2021-05-17T11:12:38.113000", "content": "thanks for the info!
", "votes": 1, "replies": [] }, { "id": 1288017, "author_name": "Shubh Patel", "author_url": "", "post_date": "2021-04-29T15:52:07.757000", "content": "Thanks for giving alternative option
", "votes": 1, "replies": [] }, { "id": 1285238, "author_name": "Miguel Xavier Santos", "author_url": "", "post_date": "2021-04-26T17:53:45.873000", "content": "Thanks for sharing!
", "votes": 1, "replies": [] }, { "id": 1270124, "author_name": "Prince Bubezi", "author_url": "", "post_date": "2021-04-11T10:11:01.690000", "content": "Thank you.
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "1254924": "This is a huge dataset and usually few GPU hours are going in training the model. If we are participating in multiple competition, we know how precious GPU hours are on Kaggle.\nIt is better to train the model in Colab and use the submission file in Kaggle.\nFew helpful links below\n\n- https://www.kaggle.com/general/74235\n- https://www.kaggle.com/learn-forum/82587\n\nAnother useful way is to use parquet dataset prepared by Saurabh (@saurabhshahane) below. Thanks Saurabh!\n\n- https://www.kaggle.com/saurabhshahane/herbarium-2021-parquet-train-and-test?select=Train.parquet", "1284958": "How are guys going about Colab's small disk size? I see 30 GB available, not enough to train on the entire (compressed) dataset. For example, [@luigisaetta](https://www.kaggle.com/luigisaetta)'s TFRecords take 41 GB.\nI tried to use my google drive, but it does not handle larger number of files. The files unzipped to gdrive are somewhere buffered (os.path.exist() == True), but never arrive and are lost on a new VM.", "1265050": "Thank you very much. This is very helpful!", "1263413": "You can directly submit a submission from colab itself using the API token as far as I know. \nI personally prefer Colab over Kaggle Notebooks too.", "1256830": "This is helpful. Thank you @saurabhbagchi for sharing.", "1255163": "I wonder how all you guys succeed using GPU. For me, with TPU I'm still not able to train on the entire dataset (exceeding TPU session time). In my team we have defined a strategy to split the training in several step, and should start working on the entire dataset by today evening.\n\nIn the meantime, since it is not off topic, I remember that I have reduced the images to 256x256 and packed in TFRecords. The dataset is: [Herb-256x256-TfRecords](https://www.kaggle.com/luigisaetta/herb2021-256).", "1255060": "This is helpful. Thank you @saurabhbagchi for sharing. This is an important toolset.", "1311377": "thanks for the info!", "1288017": "Thanks for giving alternative option", "1285238": "Thanks for sharing!", "1270124": "Thank you." } }