I was wondering if anyone was able to perform this competition on Colab, considering:
\n1) 429 - Too Many Requests received after running
\n! kaggle competitions download -c landmark-recognition-2020
\n2) 70GB drive storage space per instance
I was wondering if anyone was able to perform this competition on Colab, considering:
\n1) 429 - Too Many Requests received after running
\n! kaggle competitions download -c landmark-recognition-2020
\n2) 70GB drive storage space per instance
Hi @shymammoth
\nTry for error on request
\n!pip uninstall -y kaggle
\n!pip install --upgrade pip
\n!pip install kaggle==1.5.6
Storage is definitely an issue because the competition data is somewhere around 100GB.
\nInstead of downloading the data use GCS path and access data files. The kernel shows an example of training with GCS path:
\nhttps://www.kaggle.com/ragnar123/efficientnetb3-data-pipeline-and-model
Hope this helps!
", "rawMarkdown": "Hi @shymammoth \n1. Try for error on request\n!pip uninstall -y kaggle\n!pip install --upgrade pip\n!pip install kaggle==1.5.6\n\n2. Storage is definitely an issue because the competition data is somewhere around 100GB.\nInstead of downloading the data use GCS path and access data files. The kernel shows an example of training with GCS path:\nhttps://www.kaggle.com/ragnar123/efficientnetb3-data-pipeline-and-model\n\n**Hope this helps!**", "votes": null }, { "id": "1027165", "postDate": "09/25/2020 22:49:17", "content": "Yeah, I'm doing this competition exclusively on Colab Pro's TPU, and use GCS to store data. You can use this dirty JS trick to prevent the Colab session from disconnecting.
\nAlso, Keetar, the guy in top 3 of this competition, is also using only Colab Pro + GCS (at least he said so in the 1st place writeup of the Retrieval Track of this competition).
\nUnfortunately, since there's only 4 days to go, you'll certainly won't be able to train a proper model for this competition. It take DAYS to train models for such a large dataset.
", "rawMarkdown": "Yeah, I'm doing this competition exclusively on Colab Pro's TPU, and use GCS to store data. You can use [this dirty JS trick](https://stackoverflow.com/questions/57113226/how-to-prevent-google-colab-from-disconnecting) to prevent the Colab session from disconnecting.\n\nAlso, [Keetar](https://www.kaggle.com/keetar), the guy in top 3 of this competition, is also using only Colab Pro + GCS (at least [he said so in the 1st place writeup of the Retrieval Track of this competition](https://www.kaggle.com/c/landmark-retrieval-2020/discussion/176037)).\n\nUnfortunately, since there's only 4 days to go, you'll certainly won't be able to train a proper model for this competition. It take DAYS to train models for such a large dataset.", "votes": null }, { "id": "1027780", "postDate": "09/26/2020 11:07:01", "content": "Thanks all for your valuable comments. Doing this competition as part as a school assignment so I guess position does not really matter for me. I do understand that GCS buckets are free for a certain threshold of request, but am quite afraid I will receive a bill shock uploading 90GB of data and querying from it.
\nHope I can do something decent within these next few days
", "rawMarkdown": "Thanks all for your valuable comments. Doing this competition as part as a school assignment so I guess position does not really matter for me. I do understand that GCS buckets are free for a certain threshold of request, but am quite afraid I will receive a bill shock uploading 90GB of data and querying from it.\n\nHope I can do something decent within these next few days", "votes": null }, { "id": "1027791", "postDate": "09/26/2020 11:16:48", "content": "Wow, that sounds like a really cool school assignment 😲😲 You can use the datasets that others has converted for this competition:
\nThey should be enough to beat the baseline :)
", "rawMarkdown": "Wow, that sounds like a really cool school assignment 😲😲 You can use the datasets that others has converted for this competition:\n\n- https://www.kaggle.com/c/landmark-recognition-2020/discussion/180056\n (this one is even with an example training notebook - very cool)\n- https://www.kaggle.com/c/landmark-recognition-2020/discussion/186487\n\nThey should be enough to beat the baseline :)", "votes": null }, { "id": "1028609", "postDate": "09/27/2020 03:17:41", "content": "Thanks a lot there! :)
", "rawMarkdown": "Thanks a lot there! :)", "votes": null }, { "id": "1029049", "postDate": "09/27/2020 12:29:27", "content": "@chankhavu Thanks for sharing my dataset link. 👍
", "rawMarkdown": "chankhavu Thanks for sharing my dataset link. 👍", "votes": null } ], "comments": [ { "id": 1026677, "author_name": "jagadish13", "author_url": "", "post_date": "09/25/2020 13:39:26", "content": "Hi @shymammoth
\nTry for error on request
\n!pip uninstall -y kaggle
\n!pip install --upgrade pip
\n!pip install kaggle==1.5.6
Storage is definitely an issue because the competition data is somewhere around 100GB.
\nInstead of downloading the data use GCS path and access data files. The kernel shows an example of training with GCS path:
\nhttps://www.kaggle.com/ragnar123/efficientnetb3-data-pipeline-and-model
Hope this helps!
", "votes": null, "replies": [] }, { "id": 1027165, "author_name": "chankhavu", "author_url": "", "post_date": "09/25/2020 22:49:17", "content": "Yeah, I'm doing this competition exclusively on Colab Pro's TPU, and use GCS to store data. You can use this dirty JS trick to prevent the Colab session from disconnecting.
\nAlso, Keetar, the guy in top 3 of this competition, is also using only Colab Pro + GCS (at least he said so in the 1st place writeup of the Retrieval Track of this competition).
\nUnfortunately, since there's only 4 days to go, you'll certainly won't be able to train a proper model for this competition. It take DAYS to train models for such a large dataset.
", "votes": null, "replies": [ { "id": 1027780, "author_name": "shymammoth", "author_url": "", "post_date": "09/26/2020 11:07:01", "content": "Thanks all for your valuable comments. Doing this competition as part as a school assignment so I guess position does not really matter for me. I do understand that GCS buckets are free for a certain threshold of request, but am quite afraid I will receive a bill shock uploading 90GB of data and querying from it.
\nHope I can do something decent within these next few days
", "votes": null, "replies": [] }, { "id": 1027791, "author_name": "chankhavu", "author_url": "", "post_date": "09/26/2020 11:16:48", "content": "Wow, that sounds like a really cool school assignment 😲😲 You can use the datasets that others has converted for this competition:
\nThey should be enough to beat the baseline :)
", "votes": null, "replies": [] }, { "id": 1028609, "author_name": "shymammoth", "author_url": "", "post_date": "09/27/2020 03:17:41", "content": "Thanks a lot there! :)
", "votes": null, "replies": [] }, { "id": 1029049, "author_name": "jkreddy", "author_url": "", "post_date": "09/27/2020 12:29:27", "content": "@chankhavu Thanks for sharing my dataset link. 👍
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1026541": "I was wondering if anyone was able to perform this competition on Colab, considering:\n\n1) 429 - Too Many Requests received after running \n`! kaggle competitions download -c landmark-recognition-2020`\n2) 70GB drive storage space per instance", "1026677": "Hi @shymammoth \n1. Try for error on request\n!pip uninstall -y kaggle\n!pip install --upgrade pip\n!pip install kaggle==1.5.6\n\n2. Storage is definitely an issue because the competition data is somewhere around 100GB.\nInstead of downloading the data use GCS path and access data files. The kernel shows an example of training with GCS path:\nhttps://www.kaggle.com/ragnar123/efficientnetb3-data-pipeline-and-model\n\n**Hope this helps!**", "1027165": "Yeah, I'm doing this competition exclusively on Colab Pro's TPU, and use GCS to store data. You can use [this dirty JS trick](https://stackoverflow.com/questions/57113226/how-to-prevent-google-colab-from-disconnecting) to prevent the Colab session from disconnecting.\n\nAlso, [Keetar](https://www.kaggle.com/keetar), the guy in top 3 of this competition, is also using only Colab Pro + GCS (at least [he said so in the 1st place writeup of the Retrieval Track of this competition](https://www.kaggle.com/c/landmark-retrieval-2020/discussion/176037)).\n\nUnfortunately, since there's only 4 days to go, you'll certainly won't be able to train a proper model for this competition. It take DAYS to train models for such a large dataset.", "1027780": "Thanks all for your valuable comments. Doing this competition as part as a school assignment so I guess position does not really matter for me. I do understand that GCS buckets are free for a certain threshold of request, but am quite afraid I will receive a bill shock uploading 90GB of data and querying from it.\n\nHope I can do something decent within these next few days", "1027791": "Wow, that sounds like a really cool school assignment 😲😲 You can use the datasets that others has converted for this competition:\n\n- https://www.kaggle.com/c/landmark-recognition-2020/discussion/180056\n (this one is even with an example training notebook - very cool)\n- https://www.kaggle.com/c/landmark-recognition-2020/discussion/186487\n\nThey should be enough to beat the baseline :)", "1028609": "Thanks a lot there! :)", "1029049": "chankhavu Thanks for sharing my dataset link. 👍" }, "source": "meta" }