![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1807042%2Fa9281eef0e9ad2f566803c4c0639c50c%2Fbatchsize64.svg?generation=1593885494697955&alt=media\")
Hello,
\n\nAs can be seen here: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/154281 loading images in a Kaggle kernel from the jpeg files is too slow. So you currently have a few options:
\n\nSo I've down the preprocess for my own baseline. *The published dataset (s*o if you also use 224x224 you don't need to run the notebook again !):\nhttps://www.kaggle.com/arroqc/siic-isic-224x224-images
\n\nHere is the very simple notebook that created it: \nhttps://www.kaggle.com/arroqc/siim-isic-preprocessing-jpeg-notebook
", "messages": [ { "id": "865591", "postDate": "05/28/2020 18:13:29", "content": "Hello,
\n\nAs can be seen here: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/154281 loading images in a Kaggle kernel from the jpeg files is too slow. So you currently have a few options:
\n\nSo I've down the preprocess for my own baseline. *The published dataset (s*o if you also use 224x224 you don't need to run the notebook again !):\nhttps://www.kaggle.com/arroqc/siic-isic-224x224-images
\n\nHere is the very simple notebook that created it: \nhttps://www.kaggle.com/arroqc/siim-isic-preprocessing-jpeg-notebook
", "rawMarkdown": "Hello,\n\nAs can be seen here: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/154281 loading images in a Kaggle kernel from the jpeg files is too slow. So you currently have a few options:\n\n* Use tfrecords instead with tensorflow/keras and TPU as per some public notebooks.\n* Do it on your own local/cloud hardware\n* Preprocess all images to make smaller files that don't take long to load in a kaggle kernel. \n\n\n\nSo I've down the preprocess for my own baseline. **The published dataset (s**o if you also use 224x224 you don't need to run the notebook again !):\nhttps://www.kaggle.com/arroqc/siic-isic-224x224-images\n\n Here is the very simple notebook that created it: \nhttps://www.kaggle.com/arroqc/siim-isic-preprocessing-jpeg-notebook", "votes": null }, { "id": "865606", "postDate": "05/28/2020 18:23:08", "content": "Thank you for this Arnaud! Cant wait to see your kernel!
", "rawMarkdown": "Thank you for this Arnaud! Cant wait to see your kernel!", "votes": null }, { "id": "915416", "postDate": "07/04/2020 18:01:06", "content": "I recently joined the competition. The raw images take so long to load. My epochs time increased 3 times. So I decided to investigate further. Quite alarming that reading a batch of Raw images is 3 times slower on Kaggle then on My Laptop (6 core i5 and ssd).
\n\n
Yes, the raw images are as large as 4000x3000. I don't recommend using raw images. You should use a folder of resized JPEGs or resized TFRecords.
", "rawMarkdown": "Yes, the raw images are as large as 4000x3000. I don't recommend using raw images. You should use a folder of resized JPEGs or resized TFRecords.", "votes": null }, { "id": "916205", "postDate": "07/05/2020 13:22:52", "content": "@cdeotte I am using your TFrecords dataset. Thanks a lot.
", "rawMarkdown": "cdeotte I am using your TFrecords dataset. Thanks a lot.", "votes": null } ], "comments": [ { "id": 865606, "author_name": "yannmajewski", "author_url": "", "post_date": "05/28/2020 18:23:08", "content": "Thank you for this Arnaud! Cant wait to see your kernel!
", "votes": null, "replies": [] }, { "id": 915416, "author_name": "spideysloth", "author_url": "", "post_date": "07/04/2020 18:01:06", "content": "I recently joined the competition. The raw images take so long to load. My epochs time increased 3 times. So I decided to investigate further. Quite alarming that reading a batch of Raw images is 3 times slower on Kaggle then on My Laptop (6 core i5 and ssd).
\n\n
Yes, the raw images are as large as 4000x3000. I don't recommend using raw images. You should use a folder of resized JPEGs or resized TFRecords.
", "votes": null, "replies": [] }, { "id": 916205, "author_name": "spideysloth", "author_url": "", "post_date": "07/05/2020 13:22:52", "content": "@cdeotte I am using your TFrecords dataset. Thanks a lot.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "865591": "Hello,\n\nAs can be seen here: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/154281 loading images in a Kaggle kernel from the jpeg files is too slow. So you currently have a few options:\n\n* Use tfrecords instead with tensorflow/keras and TPU as per some public notebooks.\n* Do it on your own local/cloud hardware\n* Preprocess all images to make smaller files that don't take long to load in a kaggle kernel. \n\n\n\nSo I've down the preprocess for my own baseline. **The published dataset (s**o if you also use 224x224 you don't need to run the notebook again !):\nhttps://www.kaggle.com/arroqc/siic-isic-224x224-images\n\n Here is the very simple notebook that created it: \nhttps://www.kaggle.com/arroqc/siim-isic-preprocessing-jpeg-notebook", "865606": "Thank you for this Arnaud! Cant wait to see your kernel!", "915416": "I recently joined the competition. The raw images take so long to load. My epochs time increased 3 times. So I decided to investigate further. Quite alarming that reading a batch of Raw images is 3 times slower on Kaggle then on My Laptop (6 core i5 and ssd). \n\n", "915539": "Yes, the raw images are as large as 4000x3000. I don't recommend using raw images. You should use a folder of resized JPEGs or resized TFRecords.", "916205": "cdeotte I am using your TFrecords dataset. Thanks a lot." }, "source": "meta" }