![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2F612b743b782a99bb09fd53c06b9aa69e%2FScreen%20Shot%202020-06-08%20at%209.40.20%20AM.png?generation=1591634441749767&alt=media\")
I have found another forum post from a few years back that said to preprocess in one notebook and save the processed images as output files and then train on a seperate notebook and use those preprocessed files as input. However, it's not clear if the 5gb working limit applies.
\n\nThe jpeg train images are 25gb originally and will be less when resized but even if the average rescaling is 50% then it'll still leave us with 6.25gb which is over the limit.
\n\nI don't have the capacity to run the model on my own setup, hence wanting to run it on Kaggle but doing this preprocessing everytime I want to test a small adjustment to the model is clearly not efficient or feasible with the 30 hour CPU limit.
\n\nThanks.
", "messages": [ { "id": "877713", "postDate": "06/07/2020 21:53:48", "content": "I have found another forum post from a few years back that said to preprocess in one notebook and save the processed images as output files and then train on a seperate notebook and use those preprocessed files as input. However, it's not clear if the 5gb working limit applies.
\n\nThe jpeg train images are 25gb originally and will be less when resized but even if the average rescaling is 50% then it'll still leave us with 6.25gb which is over the limit.
\n\nI don't have the capacity to run the model on my own setup, hence wanting to run it on Kaggle but doing this preprocessing everytime I want to test a small adjustment to the model is clearly not efficient or feasible with the 30 hour CPU limit.
\n\nThanks.
", "rawMarkdown": "I have found another forum post from a few years back that said to preprocess in one notebook and save the processed images as output files and then train on a seperate notebook and use those preprocessed files as input. However, it's not clear if the 5gb working limit applies.\n\nThe jpeg train images are 25gb originally and will be less when resized but even if the average rescaling is 50% then it'll still leave us with 6.25gb which is over the limit.\n\nI don't have the capacity to run the model on my own setup, hence wanting to run it on Kaggle but doing this preprocessing everytime I want to test a small adjustment to the model is clearly not efficient or feasible with the 30 hour CPU limit.\n\nThanks.", "votes": null }, { "id": "877794", "postDate": "06/08/2020 01:37:27", "content": "If you center square crop all the train and test images, and resize them to 704x704 and use default JPEG 95% compression, it all fits in under 5GB. I posted a 768x768 TFRecords dataset here which is only 5.3GB. And 512x512 TFRecords dataset here is only 2.6GB
", "rawMarkdown": "If you center square crop all the train and test images, and resize them to 704x704 and use default JPEG 95% compression, it all fits in under 5GB. I posted a 768x768 TFRecords dataset [here][1] which is only 5.3GB. And 512x512 TFRecords dataset [here][2] is only 2.6GB\n\n[1]: https://www.kaggle.com/cdeotte/melanoma-768x768\n[2]: https://www.kaggle.com/cdeotte/melanoma-512x512", "votes": null }, { "id": "877989", "postDate": "06/08/2020 07:14:42", "content": "@cdeotte i was doing some preprocessing and im saving it in zip file then i commit the notebook and i got the output with a file size of 100Mb.But still i didnt got a \"New Dataset\" icon on the output
", "rawMarkdown": "cdeotte i was doing some preprocessing and im saving it in zip file then i commit the notebook and i got the output with a file size of 100Mb.But still i didnt got a \"New Dataset\" icon on the output", "votes": null }, { "id": "878578", "postDate": "06/08/2020 16:41:58", "content": "Kaggle notebooks keep changing appearance. Have your scrolled down and looked for button in bottom left just above the Execution Info section? (In plot below you see a New Version button which would be New Dataset)
\n\n
Thanks @cdeotte.\nJust one more thing. I couldnt get a gcs path of the kernel output file, after i have created a dataset out of that kernel.Its taking too long to get that.
", "rawMarkdown": "Thanks @cdeotte.\nJust one more thing. I couldnt get a gcs path of the kernel output file, after i have created a dataset out of that kernel.Its taking too long to get that.", "votes": null }, { "id": "878604", "postDate": "06/08/2020 17:03:03", "content": "I've seen this before. There is a weird bug at Kaggle that comes and goes. We should tell Kaggle.
\n\nFor a few days last week you couldn't use
\n\nGCS_PATH = KaggleDatasets().get_gcs_path('siim-isic-melanoma-classification')\nAnd had to use
\n\nGCS_PATH = 'gs://kds-0b2c68d2b2fa4692fcffc1029c606b32dd6a88de8d6da08fcd30d0c4'\ninstead. However that weird link is temporary because it no longer works. But now the first one with siim-isic-melanoma-classification works again. Sometime is wrong with getting gcs_path. And the problem seems to come and go. I posted a discussion here
If you center square crop all the train and test images, and resize them to 704x704 and use default JPEG 95% compression, it all fits in under 5GB. I posted a 768x768 TFRecords dataset here which is only 5.3GB. And 512x512 TFRecords dataset here is only 2.6GB
", "votes": null, "replies": [ { "id": 877989, "author_name": "msharuk589", "author_url": "", "post_date": "06/08/2020 07:14:42", "content": "@cdeotte i was doing some preprocessing and im saving it in zip file then i commit the notebook and i got the output with a file size of 100Mb.But still i didnt got a \"New Dataset\" icon on the output
", "votes": null, "replies": [] }, { "id": 878578, "author_name": "cdeotte", "author_url": "", "post_date": "06/08/2020 16:41:58", "content": "Kaggle notebooks keep changing appearance. Have your scrolled down and looked for button in bottom left just above the Execution Info section? (In plot below you see a New Version button which would be New Dataset)
\n\n
Thanks @cdeotte.\nJust one more thing. I couldnt get a gcs path of the kernel output file, after i have created a dataset out of that kernel.Its taking too long to get that.
", "votes": null, "replies": [] }, { "id": 878604, "author_name": "cdeotte", "author_url": "", "post_date": "06/08/2020 17:03:03", "content": "I've seen this before. There is a weird bug at Kaggle that comes and goes. We should tell Kaggle.
\n\nFor a few days last week you couldn't use
\n\nGCS_PATH = KaggleDatasets().get_gcs_path('siim-isic-melanoma-classification')\nAnd had to use
\n\nGCS_PATH = 'gs://kds-0b2c68d2b2fa4692fcffc1029c606b32dd6a88de8d6da08fcd30d0c4'\ninstead. However that weird link is temporary because it no longer works. But now the first one with siim-isic-melanoma-classification works again. Sometime is wrong with getting gcs_path. And the problem seems to come and go. I posted a discussion here