I have noticed on several competitions now the data is provided in a few different formats, tfrecord + png/jpg/tif etc. It seems a bit wasteful to have all formats bundled together and give us no option to download specifically the files we want. Obviously, we could go to the menu and download what we want one by one but then we also have to organize directories the same way.
\nThis data is already fairly large but increasing it up to 160gb when I know I am going to delete the tfrecords after I spend the 6 hours (and I know in many parts of the world internet is even slower than mine) it is going to take to download them is a bit inconvenient. I think it also likely scares off competitors who see such a high GB and think they wont be able to handle the requirements for the competition either because of compute or storage constraints.
\nIf an option to pick format could be provided that would be a big quality of life upgrade.
", "messages": [ { "id": 1181881, "postDate": "2021-02-02T07:30:06.597Z", "content": "I have noticed on several competitions now the data is provided in a few different formats, tfrecord + png/jpg/tif etc. It seems a bit wasteful to have all formats bundled together and give us no option to download specifically the files we want. Obviously, we could go to the menu and download what we want one by one but then we also have to organize directories the same way.
\nThis data is already fairly large but increasing it up to 160gb when I know I am going to delete the tfrecords after I spend the 6 hours (and I know in many parts of the world internet is even slower than mine) it is going to take to download them is a bit inconvenient. I think it also likely scares off competitors who see such a high GB and think they wont be able to handle the requirements for the competition either because of compute or storage constraints.
\nIf an option to pick format could be provided that would be a big quality of life upgrade.
", "rawMarkdown": "I have noticed on several competitions now the data is provided in a few different formats, tfrecord + png/jpg/tif etc. It seems a bit wasteful to have all formats bundled together and give us no option to download specifically the files we want. Obviously, we could go to the menu and download what we want one by one but then we also have to organize directories the same way. \n\nThis data is already fairly large but increasing it up to 160gb when I know I am going to delete the tfrecords after I spend the 6 hours (and I know in many parts of the world internet is even slower than mine) it is going to take to download them is a bit inconvenient. I think it also likely scares off competitors who see such a high GB and think they wont be able to handle the requirements for the competition either because of compute or storage constraints. \n\nIf an option to pick format could be provided that would be a big quality of life upgrade. ", "votes": 11 }, { "id": 1182610, "postDate": "2021-02-02T14:09:59.630Z", "content": "Totally agree
\nAnd using Kaggle API will download the whole, I cannot just download a specific folder.
", "rawMarkdown": "Totally agree\n\nAnd using Kaggle API will download the whole, I cannot just download a specific folder. ", "votes": 1 }, { "id": 1183684, "postDate": "2021-02-03T06:38:26.260Z", "content": "Try this for downlaoding specific file.
\nkaggle competitions download -c human-protein-atlas-image-classification -f train.zip\nkaggle competitions download -c human-protein-atlas-image-classification -f test.zip\n\nmkdir -p dataset\n\nunzip train.zip -d dataset/train\nunzip test.zip -d dataset/test\nkaggle competitions download -f
\n-f FILE_NAME, --file FILE_NAME
\n {File name, all files downloaded if not provided}
\nsource Kaggle api
Unfortunately we don't have an easy way to fix this. We just asked the support team at Kaggle, and here is their response:
\n\n", "rawMarkdown": "Unfortunately we don't have an easy way to fix this. We just asked the support team at Kaggle, and here is their response: \n> The TFRecords are included in the competition bundle because we've included TFRecords in the hidden dataset, so the competition mechanics require it. They are relatively small (~20GB) and shouldn't add significantly to the overall download size.", "replies": [ { "id": 1185292, "postDate": "2021-02-04T05:20:14.930Z", "content": "The TFRecords are included in the competition bundle because we've included TFRecords in the hidden dataset, so the competition mechanics require it. They are relatively small (~20GB) and shouldn't add significantly to the overall download size.
\n
My cynical translation:
\n\n\nThe TFRecords are included in the competition bundle because we've included TFRecords in the hidden dataset, because we are now a subsidiary of Google, and Google TFRecords work really well on Google TPUs, and Google wants you to use TPUs.
\n
Added 5 Feb…
\nI think providing TFRecords (unbundled) to promote TPU use is a great idea. Just puzzled by the reply that someone gave to our competition host.
I wasn't going to say it… But that's how I interpreted it as well.
", "rawMarkdown": "I wasn't going to say it... But that's how I interpreted it as well. " } ] }, { "id": 1182077, "postDate": "2021-02-02T09:41:12.343Z", "content": "I was under the impression that clicking the \"train\" folder followed by download would only download the train data without the corresponding tfrecords, or have I misunderstood?
", "rawMarkdown": "I was under the impression that clicking the \"train\" folder followed by download would only download the train data without the corresponding tfrecords, or have I misunderstood?", "replies": [ { "id": 1182112, "postDate": "2021-02-02T10:01:45.130Z", "content": "Yes, that is the case but then we would also need to click and download the test folder as well, and the train.csv and the sample_submission etc. And then organize the directory structure ourselves.
\nIt is a minor kaggle suggestion, not really specific to this competition just to give us a little more flexibility.
", "rawMarkdown": "Yes, that is the case but then we would also need to click and download the test folder as well, and the train.csv and the sample_submission etc. And then organize the directory structure ourselves.\n\nIt is a minor kaggle suggestion, not really specific to this competition just to give us a little more flexibility. ", "votes": 1 }, { "id": 1182139, "postDate": "2021-02-02T10:19:23.927Z", "content": "I see what you mean. I can forward the feedback to Kaggle staff at our next meeting.
", "rawMarkdown": "I see what you mean. I can forward the feedback to Kaggle staff at our next meeting.", "votes": 2 } ] } ], "comments": [ { "id": 1182610, "author_name": "Serigne ", "author_url": "", "post_date": "2021-02-02T14:09:59.630000", "content": "Totally agree
\nAnd using Kaggle API will download the whole, I cannot just download a specific folder.
", "votes": 1, "replies": [] }, { "id": 1183684, "author_name": "Neo", "author_url": "", "post_date": "2021-02-03T06:38:26.260000", "content": "Try this for downlaoding specific file.
\nkaggle competitions download -c human-protein-atlas-image-classification -f train.zip\nkaggle competitions download -c human-protein-atlas-image-classification -f test.zip\n\nmkdir -p dataset\n\nunzip train.zip -d dataset/train\nunzip test.zip -d dataset/test\nkaggle competitions download -f
\n-f FILE_NAME, --file FILE_NAME
\n {File name, all files downloaded if not provided}
\nsource Kaggle api
Unfortunately we don't have an easy way to fix this. We just asked the support team at Kaggle, and here is their response:
\n\n", "votes": 0, "replies": [ { "id": 1185292, "author_name": "JohnM", "author_url": "", "post_date": "2021-02-04T05:20:14.930000", "content": "The TFRecords are included in the competition bundle because we've included TFRecords in the hidden dataset, so the competition mechanics require it. They are relatively small (~20GB) and shouldn't add significantly to the overall download size.
\n
My cynical translation:
\n\n\nThe TFRecords are included in the competition bundle because we've included TFRecords in the hidden dataset, because we are now a subsidiary of Google, and Google TFRecords work really well on Google TPUs, and Google wants you to use TPUs.
\n
Added 5 Feb…
\nI think providing TFRecords (unbundled) to promote TPU use is a great idea. Just puzzled by the reply that someone gave to our competition host.
I wasn't going to say it… But that's how I interpreted it as well.
", "votes": 0, "replies": [] } ] }, { "id": 1182077, "author_name": "Casper Winsnes", "author_url": "", "post_date": "2021-02-02T09:41:12.343000", "content": "I was under the impression that clicking the \"train\" folder followed by download would only download the train data without the corresponding tfrecords, or have I misunderstood?
", "votes": 0, "replies": [ { "id": 1182112, "author_name": "ryches", "author_url": "", "post_date": "2021-02-02T10:01:45.130000", "content": "Yes, that is the case but then we would also need to click and download the test folder as well, and the train.csv and the sample_submission etc. And then organize the directory structure ourselves.
\nIt is a minor kaggle suggestion, not really specific to this competition just to give us a little more flexibility.
", "votes": 1, "replies": [] }, { "id": 1182139, "author_name": "Casper Winsnes", "author_url": "", "post_date": "2021-02-02T10:19:23.927000", "content": "I see what you mean. I can forward the feedback to Kaggle staff at our next meeting.
", "votes": 2, "replies": [] } ] } ], "raw_markdown_by_id": { "1181881": "I have noticed on several competitions now the data is provided in a few different formats, tfrecord + png/jpg/tif etc. It seems a bit wasteful to have all formats bundled together and give us no option to download specifically the files we want. Obviously, we could go to the menu and download what we want one by one but then we also have to organize directories the same way. \n\nThis data is already fairly large but increasing it up to 160gb when I know I am going to delete the tfrecords after I spend the 6 hours (and I know in many parts of the world internet is even slower than mine) it is going to take to download them is a bit inconvenient. I think it also likely scares off competitors who see such a high GB and think they wont be able to handle the requirements for the competition either because of compute or storage constraints. \n\nIf an option to pick format could be provided that would be a big quality of life upgrade. ", "1182610": "Totally agree\n\nAnd using Kaggle API will download the whole, I cannot just download a specific folder. ", "1183684": "Try this for downlaoding specific file.\n\n```\nkaggle competitions download -c human-protein-atlas-image-classification -f train.zip\nkaggle competitions download -c human-protein-atlas-image-classification -f test.zip\n\nmkdir -p dataset\n\nunzip train.zip -d dataset/train\nunzip test.zip -d dataset/test\n```\n\n\nkaggle competitions download -f\n\n-f FILE_NAME, --file FILE_NAME\n {File name, all files downloaded if not provided}\nsource [Kaggle api](https://github.com/Kaggle/kaggle-api)", "1183310": "Unfortunately we don't have an easy way to fix this. We just asked the support team at Kaggle, and here is their response: \n> The TFRecords are included in the competition bundle because we've included TFRecords in the hidden dataset, so the competition mechanics require it. They are relatively small (~20GB) and shouldn't add significantly to the overall download size.", "1182077": "I was under the impression that clicking the \"train\" folder followed by download would only download the train data without the corresponding tfrecords, or have I misunderstood?" } }