{ "id": 240411, "title": "Missing studies from Kaggle API download", "url": "/competitions/siim-covid19-detection/discussion/240411", "author_name": "Ian Pan", "post_date": "2021-05-19T17:35:58.679000", "votes": 23, "comment_count": 19, "views": 0, "content": "

I downloaded the competition data via the Kaggle API. It seems there are some missing studies. I only have 5,819/6,054 train studies and 1,178/1,214 test studies. Are other people having the same issue?

", "messages": [ { "id": 1315297, "postDate": "2021-05-19T17:35:58.680Z", "content": "

I downloaded the competition data via the Kaggle API. It seems there are some missing studies. I only have 5,819/6,054 train studies and 1,178/1,214 test studies. Are other people having the same issue?

", "rawMarkdown": "I downloaded the competition data via the Kaggle API. It seems there are some missing studies. I only have 5,819/6,054 train studies and 1,178/1,214 test studies. Are other people having the same issue?", "votes": 23 }, { "id": 1318222, "postDate": "2021-05-22T05:58:34.287Z", "content": "

Final update: this has been fixed and verified. All bundles are now containing the correct full number of files.

", "rawMarkdown": "Final update: this has been fixed and verified. All bundles are now containing the correct full number of files.", "votes": 3, "replies": [ { "id": 1318275, "postDate": "2021-05-22T07:03:24.557Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1315546, "postDate": "2021-05-19T22:36:10.893Z", "content": "

Update: We've replicated the error and are rebuilding the bundle. It appears there was some corruption that occurred in its .zip packaging. We expect the issue to be resolved in the next 24 hours. Sorry for the inconvenience, and we appreciate your patience as we correct it.

\n

In the meantime, the bundle available in Kaggle notebooks with the dataset loaded is in tact and working, so that is an alternative.

", "rawMarkdown": "Update: We've replicated the error and are rebuilding the bundle. It appears there was some corruption that occurred in its .zip packaging. We expect the issue to be resolved in the next 24 hours. Sorry for the inconvenience, and we appreciate your patience as we correct it.\n\nIn the meantime, the bundle available in Kaggle notebooks with the dataset loaded is in tact and working, so that is an alternative. ", "votes": 2 }, { "id": 1315318, "postDate": "2021-05-19T17:58:54.093Z", "content": "

I've got the same problem. I got missing files through kaggle notebook.

", "rawMarkdown": "I've got the same problem. I got missing files through kaggle notebook.", "votes": 2, "replies": [ { "id": 1315483, "postDate": "2021-05-19T20:41:45.570Z", "content": "

Thanks @osciiart! Which files are you missing? What are the counts?

", "rawMarkdown": "Thanks @osciiart! Which files are you missing? What are the counts?" } ] }, { "id": 1325751, "postDate": "2021-05-28T01:38:27.177Z", "content": "

maybe try to uninstall and reinstall the kaggle api with the latest version. I had a similar issue in an another competition few weeks ago, it was related to an outdated kaggle api.

", "rawMarkdown": "maybe try to uninstall and reinstall the kaggle api with the latest version. I had a similar issue in an another competition few weeks ago, it was related to an outdated kaggle api.", "replies": [ { "id": 1326887, "postDate": "2021-05-28T17:58:42.203Z", "content": "

hello thank you this worked!

", "rawMarkdown": "hello thank you this worked!", "votes": 1 } ] }, { "id": 1325539, "postDate": "2021-05-27T20:18:12.703Z", "content": "

I am having the same exact issue I only got train_study_level.csv, train_image_level.csv, sample submission and in addition I only got like 30 images and im not sure if it is from train or test

", "rawMarkdown": "I am having the same exact issue I only got train_study_level.csv, train_image_level.csv, sample submission and in addition I only got like 30 images and im not sure if it is from train or test", "replies": [ { "id": 1325571, "postDate": "2021-05-27T20:47:48.620Z", "content": "

How large was the file that downloaded for you? Should have been a single zip.

", "rawMarkdown": "How large was the file that downloaded for you? Should have been a single zip." }, { "id": 1325577, "postDate": "2021-05-27T20:50:56.823Z", "content": "

Im not sure the size of the file, because it did not say when I downloaded it i got multiple zip files not a single one

", "rawMarkdown": "Im not sure the size of the file, because it did not say when I downloaded it i got multiple zip files not a single one" }, { "id": 1325588, "postDate": "2021-05-27T20:59:24.580Z", "content": "

Ahh, okay. Were you intending to download separate files, or would you like the full archive? That should be kaggle competitions download -c siim-covid19-detection

\n

I'll look into why the files-level download might not be downloading everything.

", "rawMarkdown": "Ahh, okay. Were you intending to download separate files, or would you like the full archive? That should be `kaggle competitions download -c siim-covid19-detection`\n\nI'll look into why the files-level download might not be downloading everything." }, { "id": 1325623, "postDate": "2021-05-27T21:24:09.340Z", "content": "

I put 'kaggle competitions download -c siim-covid19-detection' in colab and got seperate files but i do want the full archive

", "rawMarkdown": "I put 'kaggle competitions download -c siim-covid19-detection' in colab and got seperate files but i do want the full archive" }, { "id": 1325631, "postDate": "2021-05-27T21:43:19.320Z", "content": "

Colab might have a (very) old version of the Kaggle API. Can you try updating it? (I don't recall if pip works on Colab, but pip install kaggle --upgrade should do it.) The command you mention should download the full archive.

", "rawMarkdown": "Colab might have a (very) old version of the Kaggle API. Can you try updating it? (I don't recall if `pip` works on Colab, but `pip install kaggle --upgrade` should do it.) The command you mention *should* download the full archive." }, { "id": 1326892, "postDate": "2021-05-28T17:59:19.407Z", "content": "

I uninstalled and reinstalled kaggle and it worked but thank you for your help

", "rawMarkdown": "I uninstalled and reinstalled kaggle and it worked but thank you for your help" } ] }, { "id": 1323478, "postDate": "2021-05-26T09:00:56.043Z", "content": "

Does anyone know of a way to download only the missing studies instead of re-downloading the entire 84GB archive? Unfortunately, on my connection, it takes around 8 hours just to download the archive. I've tried:

\n

kaggle competitions download -f <study_path> -c siim-covid19-detection

\n

but I get 404 - Not Found errors. I do understand that -f is meant to work with files and not directories but I don't see any other options in the API documentation.

\n

Besides, kaggle competitions files -c siim-covid19-detection only returns a handful of files. So, I'm not sure I'll be able to download even if I specify the full file paths. Has anyone tried this?

\n

Thanks in advance!

", "rawMarkdown": "Does anyone know of a way to download only the missing studies instead of re-downloading the entire 84GB archive? Unfortunately, on my connection, it takes around 8 hours just to download the archive. I've tried:\n\n`kaggle competitions download -f -c siim-covid19-detection`\n\nbut I get 404 - Not Found errors. I do understand that -f is meant to work with files and not directories but I don't see any other options in the [API documentation](https://github.com/Kaggle/kaggle-api).\n\nBesides, `kaggle competitions files -c siim-covid19-detection` only returns a handful of files. So, I'm not sure I'll be able to download even if I specify the full file paths. Has anyone tried this?\n\nThanks in advance!", "replies": [ { "id": 1325548, "postDate": "2021-05-27T20:33:17.893Z", "content": "

I did 'kaggle competitions files -c siim-covid19-detection' aswell and I only got some of the items, have you found a way to get all of them?

", "rawMarkdown": "I did 'kaggle competitions files -c siim-covid19-detection' aswell and I only got some of the items, have you found a way to get all of them?" }, { "id": 1326021, "postDate": "2021-05-28T07:01:41.023Z", "content": "

I ended up re-downloading the archive overnight.

\n

In the new archive, 201 files had bad CRCs and a visual inspection revealed that the files were, indeed, corrupt. Since no one else has reported this issue, I assume, the corruption must have occurred at my end.

\n

Fortunately, even though the files were corrupt, at least now I had full file paths to use with kaggle competitions download -f <dcm_path> -c siim-covid19-detection. After running into a \"Too many Requests\" error after approx. 150 individual downloads yesterday, I've just finished downloading the remaining files.

\n

I now have 6334 & 1263 DCMs in the train and test directories, respectively.

", "rawMarkdown": "I ended up re-downloading the archive overnight.\n\nIn the new archive, 201 files had bad CRCs and a visual inspection revealed that the files were, indeed, corrupt. Since no one else has reported this issue, I assume, the corruption must have occurred at my end.\n\nFortunately, even though the files were corrupt, at least now I had full file paths to use with `kaggle competitions download -f -c siim-covid19-detection`. After running into a \"Too many Requests\" error after approx. 150 individual downloads yesterday, I've just finished downloading the remaining files.\n\nI now have 6334 & 1263 DCMs in the train and test directories, respectively." } ] }, { "id": 1315477, "postDate": "2021-05-19T20:30:41.337Z", "content": "

Thanks for the report. We are investigating this!

", "rawMarkdown": "Thanks for the report. We are investigating this!" }, { "id": 1315338, "postDate": "2021-05-19T18:10:20.993Z", "content": "

Hi, I simply checked my cases.

\n
len(glob.glob('../input/siim-covid19-detection/train/*'))\n\nlen(glob.glob('../input/siim-covid19-detection/test/*'))\n
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
In KaggleKaggle APIEtc. resized-to-256px-jpg
train / testtrain / testtrain / test
6,054 / 1,2145,819 / 1,1786,334 / 1,263
\n

There is same issue for me(maybe I'm wrong).
\nI hope it helps you.

\n

p.s. It seems the main cause of my submission score error - link.
\nšŸ˜…

", "rawMarkdown": "Hi, I simply checked my cases.\n```\nlen(glob.glob('../input/siim-covid19-detection/train/*'))\n\nlen(glob.glob('../input/siim-covid19-detection/test/*'))\n```\n\n| In Kaggle | Kaggle API| Etc. [resized-to-256px-jpg](https://www.kaggle.com/xhlulu/siim-covid19-resized-to-256px-jpg) | \n| --- | --- |--- |\n|train / test | train / test | train / test |\n|6,054 / 1,214 | 5,819 / 1,178 | 6,334 / 1,263 |\n\nThere is same issue for me(maybe I'm wrong).\nI hope it helps you.\n\np.s. It seems the main cause of my `submission score error` - [link](https://www.kaggle.com/c/siim-covid19-detection/discussion/240332).\nšŸ˜…\n" } ], "comments": [ { "id": 1318222, "author_name": "Julia Elliott", "author_url": "", "post_date": "2021-05-22T05:58:34.287000", "content": "

Final update: this has been fixed and verified. All bundles are now containing the correct full number of files.

", "votes": 3, "replies": [ { "id": 1318275, "author_name": "", "author_url": "", "post_date": "2021-05-22T07:03:24.557000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1315546, "author_name": "Julia Elliott", "author_url": "", "post_date": "2021-05-19T22:36:10.893000", "content": "

Update: We've replicated the error and are rebuilding the bundle. It appears there was some corruption that occurred in its .zip packaging. We expect the issue to be resolved in the next 24 hours. Sorry for the inconvenience, and we appreciate your patience as we correct it.

\n

In the meantime, the bundle available in Kaggle notebooks with the dataset loaded is in tact and working, so that is an alternative.

", "votes": 2, "replies": [] }, { "id": 1315318, "author_name": "OsciiArt", "author_url": "", "post_date": "2021-05-19T17:58:54.093000", "content": "

I've got the same problem. I got missing files through kaggle notebook.

", "votes": 2, "replies": [ { "id": 1315483, "author_name": "Phil Culliton", "author_url": "", "post_date": "2021-05-19T20:41:45.570000", "content": "

Thanks @osciiart! Which files are you missing? What are the counts?

", "votes": 0, "replies": [] } ] }, { "id": 1325751, "author_name": "Shiro", "author_url": "", "post_date": "2021-05-28T01:38:27.177000", "content": "

maybe try to uninstall and reinstall the kaggle api with the latest version. I had a similar issue in an another competition few weeks ago, it was related to an outdated kaggle api.

", "votes": 0, "replies": [ { "id": 1326887, "author_name": "Simonsob", "author_url": "", "post_date": "2021-05-28T17:58:42.203000", "content": "

hello thank you this worked!

", "votes": 1, "replies": [] } ] }, { "id": 1325539, "author_name": "Simonsob", "author_url": "", "post_date": "2021-05-27T20:18:12.703000", "content": "

I am having the same exact issue I only got train_study_level.csv, train_image_level.csv, sample submission and in addition I only got like 30 images and im not sure if it is from train or test

", "votes": 0, "replies": [ { "id": 1325571, "author_name": "Phil Culliton", "author_url": "", "post_date": "2021-05-27T20:47:48.620000", "content": "

How large was the file that downloaded for you? Should have been a single zip.

", "votes": 0, "replies": [] }, { "id": 1325577, "author_name": "Simonsob", "author_url": "", "post_date": "2021-05-27T20:50:56.823000", "content": "

Im not sure the size of the file, because it did not say when I downloaded it i got multiple zip files not a single one

", "votes": 0, "replies": [] }, { "id": 1325588, "author_name": "Phil Culliton", "author_url": "", "post_date": "2021-05-27T20:59:24.580000", "content": "

Ahh, okay. Were you intending to download separate files, or would you like the full archive? That should be kaggle competitions download -c siim-covid19-detection

\n

I'll look into why the files-level download might not be downloading everything.

", "votes": 0, "replies": [] }, { "id": 1325623, "author_name": "Simonsob", "author_url": "", "post_date": "2021-05-27T21:24:09.340000", "content": "

I put 'kaggle competitions download -c siim-covid19-detection' in colab and got seperate files but i do want the full archive

", "votes": 0, "replies": [] }, { "id": 1325631, "author_name": "Phil Culliton", "author_url": "", "post_date": "2021-05-27T21:43:19.320000", "content": "

Colab might have a (very) old version of the Kaggle API. Can you try updating it? (I don't recall if pip works on Colab, but pip install kaggle --upgrade should do it.) The command you mention should download the full archive.

", "votes": 0, "replies": [] }, { "id": 1326892, "author_name": "Simonsob", "author_url": "", "post_date": "2021-05-28T17:59:19.407000", "content": "

I uninstalled and reinstalled kaggle and it worked but thank you for your help

", "votes": 0, "replies": [] } ] }, { "id": 1323478, "author_name": "akki", "author_url": "", "post_date": "2021-05-26T09:00:56.043000", "content": "

Does anyone know of a way to download only the missing studies instead of re-downloading the entire 84GB archive? Unfortunately, on my connection, it takes around 8 hours just to download the archive. I've tried:

\n

kaggle competitions download -f <study_path> -c siim-covid19-detection

\n

but I get 404 - Not Found errors. I do understand that -f is meant to work with files and not directories but I don't see any other options in the API documentation.

\n

Besides, kaggle competitions files -c siim-covid19-detection only returns a handful of files. So, I'm not sure I'll be able to download even if I specify the full file paths. Has anyone tried this?

\n

Thanks in advance!

", "votes": 0, "replies": [ { "id": 1325548, "author_name": "Simonsob", "author_url": "", "post_date": "2021-05-27T20:33:17.893000", "content": "

I did 'kaggle competitions files -c siim-covid19-detection' aswell and I only got some of the items, have you found a way to get all of them?

", "votes": 0, "replies": [] }, { "id": 1326021, "author_name": "akki", "author_url": "", "post_date": "2021-05-28T07:01:41.023000", "content": "

I ended up re-downloading the archive overnight.

\n

In the new archive, 201 files had bad CRCs and a visual inspection revealed that the files were, indeed, corrupt. Since no one else has reported this issue, I assume, the corruption must have occurred at my end.

\n

Fortunately, even though the files were corrupt, at least now I had full file paths to use with kaggle competitions download -f <dcm_path> -c siim-covid19-detection. After running into a \"Too many Requests\" error after approx. 150 individual downloads yesterday, I've just finished downloading the remaining files.

\n

I now have 6334 & 1263 DCMs in the train and test directories, respectively.

", "votes": 0, "replies": [] } ] }, { "id": 1315477, "author_name": "Julia Elliott", "author_url": "", "post_date": "2021-05-19T20:30:41.337000", "content": "

Thanks for the report. We are investigating this!

", "votes": 0, "replies": [] }, { "id": 1315338, "author_name": "Heroseo", "author_url": "", "post_date": "2021-05-19T18:10:20.993000", "content": "

Hi, I simply checked my cases.

\n
len(glob.glob('../input/siim-covid19-detection/train/*'))\n\nlen(glob.glob('../input/siim-covid19-detection/test/*'))\n
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
In KaggleKaggle APIEtc. resized-to-256px-jpg
train / testtrain / testtrain / test
6,054 / 1,2145,819 / 1,1786,334 / 1,263
\n

There is same issue for me(maybe I'm wrong).
\nI hope it helps you.

\n

p.s. It seems the main cause of my submission score error - link.
\nšŸ˜…

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1315297": "I downloaded the competition data via the Kaggle API. It seems there are some missing studies. I only have 5,819/6,054 train studies and 1,178/1,214 test studies. Are other people having the same issue?", "1318222": "Final update: this has been fixed and verified. All bundles are now containing the correct full number of files.", "1315546": "Update: We've replicated the error and are rebuilding the bundle. It appears there was some corruption that occurred in its .zip packaging. We expect the issue to be resolved in the next 24 hours. Sorry for the inconvenience, and we appreciate your patience as we correct it.\n\nIn the meantime, the bundle available in Kaggle notebooks with the dataset loaded is in tact and working, so that is an alternative. ", "1315318": "I've got the same problem. I got missing files through kaggle notebook.", "1325751": "maybe try to uninstall and reinstall the kaggle api with the latest version. I had a similar issue in an another competition few weeks ago, it was related to an outdated kaggle api.", "1325539": "I am having the same exact issue I only got train_study_level.csv, train_image_level.csv, sample submission and in addition I only got like 30 images and im not sure if it is from train or test", "1323478": "Does anyone know of a way to download only the missing studies instead of re-downloading the entire 84GB archive? Unfortunately, on my connection, it takes around 8 hours just to download the archive. I've tried:\n\n`kaggle competitions download -f -c siim-covid19-detection`\n\nbut I get 404 - Not Found errors. I do understand that -f is meant to work with files and not directories but I don't see any other options in the [API documentation](https://github.com/Kaggle/kaggle-api).\n\nBesides, `kaggle competitions files -c siim-covid19-detection` only returns a handful of files. So, I'm not sure I'll be able to download even if I specify the full file paths. Has anyone tried this?\n\nThanks in advance!", "1315477": "Thanks for the report. We are investigating this!", "1315338": "Hi, I simply checked my cases.\n```\nlen(glob.glob('../input/siim-covid19-detection/train/*'))\n\nlen(glob.glob('../input/siim-covid19-detection/test/*'))\n```\n\n| In Kaggle | Kaggle API| Etc. [resized-to-256px-jpg](https://www.kaggle.com/xhlulu/siim-covid19-resized-to-256px-jpg) | \n| --- | --- |--- |\n|train / test | train / test | train / test |\n|6,054 / 1,214 | 5,819 / 1,178 | 6,334 / 1,263 |\n\nThere is same issue for me(maybe I'm wrong).\nI hope it helps you.\n\np.s. It seems the main cause of my `submission score error` - [link](https://www.kaggle.com/c/siim-covid19-detection/discussion/240332).\nšŸ˜…\n" } }