{ "id": 39583, "title": "378 GB of images in training dataset", "url": "/competitions/cdiscount-image-classification-challenge/discussion/39583", "author_name": "", "post_date": "2017-09-16T23:26:15.947763800Z", "votes": 9, "comment_count": 20, "views": 0, "content": "

BSON training file extracted produced:

\n\n", "messages": [ { "id": "221944", "postDate": "09/16/2017 23:26:15", "content": "

BSON training file extracted produced:

\n\n", "rawMarkdown": "BSON training file extracted produced:\n\n - 378 GB of image files\n - 12,371,293 files\n - 5,270 different categories", "votes": null }, { "id": "221980", "postDate": "09/17/2017 05:52:45", "content": "

How do you unarchive bson file?

", "rawMarkdown": "How do you unarchive bson file?", "votes": null }, { "id": "221997", "postDate": "09/17/2017 07:09:10", "content": "

Did you run through the bson and save each as a jpeg or png?

", "rawMarkdown": "Did you run through the bson and save each as a jpeg or png?", "votes": null }, { "id": "222004", "postDate": "09/17/2017 07:52:18", "content": "

I can confirm that training images are 12,371,293 files but my allocated storage disk size is about 81G after extracting images from training.bson. I have used JPEG format.

", "rawMarkdown": "I can confirm that training images are 12,371,293 files but my allocated storage disk size is about 81G after extracting images from training.bson. I have used JPEG format.", "votes": null }, { "id": "222005", "postDate": "09/17/2017 08:00:05", "content": "

Here's the code I used to extract the images: https://www.kaggle.com/carlossouza/extract-image-files-from-bson-and-save-to-disk/code

", "rawMarkdown": "Here's the code I used to extract the images: https://www.kaggle.com/carlossouza/extract-image-files-from-bson-and-save-to-disk/code", "votes": null }, { "id": "222042", "postDate": "09/17/2017 10:03:22", "content": "

How much time did you need to extract all images?

", "rawMarkdown": "How much time did you need to extract all images?", "votes": null }, { "id": "222096", "postDate": "09/17/2017 13:48:18", "content": "

Less than a day... Started to run the script Friday night, finished on Saturday around noon. \nI'm not happy with the result, my strategy is to use these image files as a fall back option.\nMy main objective today is to figure out a way to feed BSON generator directly into the training procedure. Let's see, if I manage to do it I will post the code :)\nCheers!

", "rawMarkdown": "Less than a day... Started to run the script Friday night, finished on Saturday around noon. \nI'm not happy with the result, my strategy is to use these image files as a fall back option.\nMy main objective today is to figure out a way to feed BSON generator directly into the training procedure. Let's see, if I manage to do it I will post the code :)\nCheers!", "votes": null }, { "id": "222151", "postDate": "09/17/2017 19:12:54", "content": "

My disk usage after creating jpeg format was 81G too.

", "rawMarkdown": "My disk usage after creating jpeg format was 81G too.", "votes": null }, { "id": "222169", "postDate": "09/17/2017 21:38:57", "content": "

Hi Carlos! Let me point a few issues in your script that are not resource optimal:

\n\n
    \n

  1. It decodes (imread) than encodes (plt.imsave) the image in order to work. It is better to read and write the RAW JPEG data from BSON file.

    2. \n

  2. You check folder (os.path.exists) and file (os.path.isfile) existence for every product and image. This is somehow slow and will lead to poor performance. A better approach, though not so conservative, is to read the categories from category_names.csv file and create the folders before saving images on it.

    3. \n
\n\n

With these modifications, I could save all train images in less than 30 minutes (on a SSD disk). It uses 82Gb of storage and has the same 12,371,293 files you find.

", "rawMarkdown": "Hi Carlos! Let me point a few issues in your script that are not resource optimal:\n\n1. It decodes (`imread`) than encodes (`plt.imsave`) the image in order to work. It is better to read and write the RAW JPEG data from BSON file.\n\n2. You check folder (`os.path.exists`) and file (`os.path.isfile`) existence for every product and image. This is somehow slow and will lead to poor performance. A better approach, though not so conservative, is to read the categories from `category_names.csv` file and create the folders before saving images on it.\n\nWith [these modifications,][1] I could save all train images in less than 30 minutes (on a SSD disk). It uses 82Gb of storage and has the same 12,371,293 files you find.\n\n\n [1]: https://www.kaggle.com/bguberfain/not-so-naive-way-to-convert-bson-to-files/", "votes": null }, { "id": "222182", "postDate": "09/18/2017 00:11:26", "content": "

You are right Bruno, thanks a lot!!!\nNevertheless, I figured out a way to feed BSON data straight into Keras, without the need of saving & reading data in the disk! I will post the code during the week :)

", "rawMarkdown": "You are right Bruno, thanks a lot!!!\nNevertheless, I figured out a way to feed BSON data straight into Keras, without the need of saving & reading data in the disk! I will post the code during the week :)", "votes": null }, { "id": "222245", "postDate": "09/18/2017 06:40:07", "content": "

OMG, Its a large database.

", "rawMarkdown": "OMG, Its a large database.", "votes": null }, { "id": "222363", "postDate": "09/18/2017 15:19:04", "content": "

@CarlosSouza, have you tried Model.fit_generator? using a generator function you can read and preprocess the images in small batches and then feed then into keras, without exausting your memory!

\n\n

Later I can post a sample script.

", "rawMarkdown": "CarlosSouza, have you tried [Model.fit_generator][1]? using a generator function you can read and preprocess the images in small batches and then feed then into keras, without exausting your memory!\n\nLater I can post a sample script. \n \n\n\n [1]: https://keras.io/models/model/", "votes": null }, { "id": "222368", "postDate": "09/18/2017 15:31:47", "content": "

Can you give me a piece of data that has been processed? I do not know why, running on my computer for nearly 24 hours but nothing to stop the sign. Can also be placed in the clouds! thank you very much!

", "rawMarkdown": "Can you give me a piece of data that has been processed? I do not know why, running on my computer for nearly 24 hours but nothing to stop the sign. Can also be placed in the clouds! thank you very much!", "votes": null }, { "id": "222592", "postDate": "09/19/2017 10:36:03", "content": "

Bad luck! Some Problems happened when I try to build a linux system.

", "rawMarkdown": "Bad luck! Some Problems happened when I try to build a linux system.", "votes": null }, { "id": "222989", "postDate": "09/20/2017 19:47:43", "content": "

i didn't extract anything. i'm piping it on the fly. get those cpu workers going ;-)

", "rawMarkdown": "i didn't extract anything. i'm piping it on the fly. get those cpu workers going ;-)", "votes": null }, { "id": "224092", "postDate": "09/25/2017 05:02:54", "content": "

I get 12,371,293 images as well in training data.

\n\n

I also got 3095080 images in test data.

\n\n

Is this what others got?

\n\n

I did not need 378gb of disk. I just converted the images into jpeg.

", "rawMarkdown": "I get 12,371,293 images as well in training data.\n\nI also got 3095080 images in test data.\n\nIs this what others got?\n\nI did not need 378gb of disk. I just converted the images into jpeg.", "votes": null }, { "id": "224204", "postDate": "09/25/2017 14:15:15", "content": "

I get 1,768,182 test images.

\n\n

I checked the number in File Description:\n\"test.bson - (Size: 14.5 GB) Contains a list of 1,768,182 products in the same format as train.bson...\"

\n\n

https://www.kaggle.com/c/cdiscount-image-classification-challenge/data

", "rawMarkdown": "I get 1,768,182 test images.\n\nI checked the number in File Description:\n\"test.bson - (Size: 14.5 GB) Contains a list of 1,768,182 products in the same format as train.bson...\"\n\nhttps://www.kaggle.com/c/cdiscount-image-classification-challenge/data", "votes": null }, { "id": "224328", "postDate": "09/25/2017 23:35:11", "content": "

Each product can have 1-4 images as it is same format as train.bson just no category_id key.\nAlso overview (https://www.kaggle.com/c/cdiscount-image-classification-challenge) says there is over 15 million images.

", "rawMarkdown": "Each product can have 1-4 images as it is same format as train.bson just no category_id key.\nAlso overview (https://www.kaggle.com/c/cdiscount-image-classification-challenge) says there is over 15 million images.", "votes": null }, { "id": "224484", "postDate": "09/26/2017 13:58:43", "content": "

You are right, now I get 3095080 test images.

\n\n

Thanks.

", "rawMarkdown": "You are right, now I get 3095080 test images. \n\nThanks.", "votes": null }, { "id": "225044", "postDate": "09/28/2017 04:45:05", "content": "

That sounds fun, will you mind showing your kernel for this logic only? I have an idea as we can make training image batches and feed those batches onto keras/tensor flow for training the network. But like to validate this understanding with someone like yours. thnx

", "rawMarkdown": "That sounds fun, will you mind showing your kernel for this logic only? I have an idea as we can make training image batches and feed those batches onto keras/tensor flow for training the network. But like to validate this understanding with someone like yours. thnx", "votes": null }, { "id": "225108", "postDate": "09/28/2017 09:21:53", "content": "

Hi CarlosSouza, Thank you for the code. I have copied it and using it currently on Windows 7 machine and getting the following error mesage. A quick question did you run this on a linux platform. The error message is as follows :

\n\n

RuntimeError: \n An attempt has been made to start a new process before the\n current process has finished its bootstrapping phase.

\n\n
    This probably means that you are not using fork to start your\n    child processes and you have forgotten to use the proper idiom\n    in the main module:\n\n        if __name__ == '__main__':\n            freeze_support()\n            ...\n\n    The \"freeze_support()\" line can be omitted if the program\n    is not going to be frozen to produce an executable.\n
\n\n

Do you have any ideas?

\n\n

Thanks\nMartina

", "rawMarkdown": "Hi CarlosSouza, Thank you for the code. I have copied it and using it currently on Windows 7 machine and getting the following error mesage. A quick question did you run this on a linux platform. The error message is as follows :\n\nRuntimeError: \n An attempt has been made to start a new process before the\n current process has finished its bootstrapping phase.\n\n This probably means that you are not using fork to start your\n child processes and you have forgotten to use the proper idiom\n in the main module:\n\n if __name__ == '__main__':\n freeze_support()\n ...\n\n The \"freeze_support()\" line can be omitted if the program\n is not going to be frozen to produce an executable.\n\n\nDo you have any ideas?\n\nThanks\nMartina", "votes": null } ], "comments": [ { "id": 221980, "author_name": "jeffzhan", "author_url": "", "post_date": "09/17/2017 05:52:45", "content": "

How do you unarchive bson file?

", "votes": null, "replies": [] }, { "id": 221997, "author_name": "mcamack", "author_url": "", "post_date": "09/17/2017 07:09:10", "content": "

Did you run through the bson and save each as a jpeg or png?

", "votes": null, "replies": [] }, { "id": 222004, "author_name": "andreaslup", "author_url": "", "post_date": "09/17/2017 07:52:18", "content": "

I can confirm that training images are 12,371,293 files but my allocated storage disk size is about 81G after extracting images from training.bson. I have used JPEG format.

", "votes": null, "replies": [] }, { "id": 222005, "author_name": "carlossouza", "author_url": "", "post_date": "09/17/2017 08:00:05", "content": "

Here's the code I used to extract the images: https://www.kaggle.com/carlossouza/extract-image-files-from-bson-and-save-to-disk/code

", "votes": null, "replies": [] }, { "id": 222042, "author_name": "thomasseleck", "author_url": "", "post_date": "09/17/2017 10:03:22", "content": "

How much time did you need to extract all images?

", "votes": null, "replies": [ { "id": 222096, "author_name": "carlossouza", "author_url": "", "post_date": "09/17/2017 13:48:18", "content": "

Less than a day... Started to run the script Friday night, finished on Saturday around noon. \nI'm not happy with the result, my strategy is to use these image files as a fall back option.\nMy main objective today is to figure out a way to feed BSON generator directly into the training procedure. Let's see, if I manage to do it I will post the code :)\nCheers!

", "votes": null, "replies": [] } ] }, { "id": 222151, "author_name": "riyaaz", "author_url": "", "post_date": "09/17/2017 19:12:54", "content": "

My disk usage after creating jpeg format was 81G too.

", "votes": null, "replies": [] }, { "id": 222169, "author_name": "bguberfain", "author_url": "", "post_date": "09/17/2017 21:38:57", "content": "

Hi Carlos! Let me point a few issues in your script that are not resource optimal:

\n\n
    \n

  1. It decodes (imread) than encodes (plt.imsave) the image in order to work. It is better to read and write the RAW JPEG data from BSON file.

    2. \n

  2. You check folder (os.path.exists) and file (os.path.isfile) existence for every product and image. This is somehow slow and will lead to poor performance. A better approach, though not so conservative, is to read the categories from category_names.csv file and create the folders before saving images on it.

    3. \n
\n\n

With these modifications, I could save all train images in less than 30 minutes (on a SSD disk). It uses 82Gb of storage and has the same 12,371,293 files you find.

", "votes": null, "replies": [ { "id": 222182, "author_name": "carlossouza", "author_url": "", "post_date": "09/18/2017 00:11:26", "content": "

You are right Bruno, thanks a lot!!!\nNevertheless, I figured out a way to feed BSON data straight into Keras, without the need of saving & reading data in the disk! I will post the code during the week :)

", "votes": null, "replies": [] }, { "id": 222363, "author_name": "aloisiodn", "author_url": "", "post_date": "09/18/2017 15:19:04", "content": "

@CarlosSouza, have you tried Model.fit_generator? using a generator function you can read and preprocess the images in small batches and then feed then into keras, without exausting your memory!

\n\n

Later I can post a sample script.

", "votes": null, "replies": [] } ] }, { "id": 222245, "author_name": "luizgustavomori", "author_url": "", "post_date": "09/18/2017 06:40:07", "content": "

OMG, Its a large database.

", "votes": null, "replies": [] }, { "id": 222368, "author_name": "leemathew", "author_url": "", "post_date": "09/18/2017 15:31:47", "content": "

Can you give me a piece of data that has been processed? I do not know why, running on my computer for nearly 24 hours but nothing to stop the sign. Can also be placed in the clouds! thank you very much!

", "votes": null, "replies": [] }, { "id": 222592, "author_name": "scumac", "author_url": "", "post_date": "09/19/2017 10:36:03", "content": "

Bad luck! Some Problems happened when I try to build a linux system.

", "votes": null, "replies": [] }, { "id": 222989, "author_name": "bavarianml", "author_url": "", "post_date": "09/20/2017 19:47:43", "content": "

i didn't extract anything. i'm piping it on the fly. get those cpu workers going ;-)

", "votes": null, "replies": [ { "id": 225044, "author_name": "raheelkhan", "author_url": "", "post_date": "09/28/2017 04:45:05", "content": "

That sounds fun, will you mind showing your kernel for this logic only? I have an idea as we can make training image batches and feed those batches onto keras/tensor flow for training the network. But like to validate this understanding with someone like yours. thnx

", "votes": null, "replies": [] } ] }, { "id": 224092, "author_name": "mhs123", "author_url": "", "post_date": "09/25/2017 05:02:54", "content": "

I get 12,371,293 images as well in training data.

\n\n

I also got 3095080 images in test data.

\n\n

Is this what others got?

\n\n

I did not need 378gb of disk. I just converted the images into jpeg.

", "votes": null, "replies": [ { "id": 224204, "author_name": "martinpella", "author_url": "", "post_date": "09/25/2017 14:15:15", "content": "

I get 1,768,182 test images.

\n\n

I checked the number in File Description:\n\"test.bson - (Size: 14.5 GB) Contains a list of 1,768,182 products in the same format as train.bson...\"

\n\n

https://www.kaggle.com/c/cdiscount-image-classification-challenge/data

", "votes": null, "replies": [] }, { "id": 224328, "author_name": "mhs123", "author_url": "", "post_date": "09/25/2017 23:35:11", "content": "

Each product can have 1-4 images as it is same format as train.bson just no category_id key.\nAlso overview (https://www.kaggle.com/c/cdiscount-image-classification-challenge) says there is over 15 million images.

", "votes": null, "replies": [] }, { "id": 224484, "author_name": "martinpella", "author_url": "", "post_date": "09/26/2017 13:58:43", "content": "

You are right, now I get 3095080 test images.

\n\n

Thanks.

", "votes": null, "replies": [] } ] }, { "id": 225108, "author_name": "martina1234", "author_url": "", "post_date": "09/28/2017 09:21:53", "content": "

Hi CarlosSouza, Thank you for the code. I have copied it and using it currently on Windows 7 machine and getting the following error mesage. A quick question did you run this on a linux platform. The error message is as follows :

\n\n

RuntimeError: \n An attempt has been made to start a new process before the\n current process has finished its bootstrapping phase.

\n\n
    This probably means that you are not using fork to start your\n    child processes and you have forgotten to use the proper idiom\n    in the main module:\n\n        if __name__ == '__main__':\n            freeze_support()\n            ...\n\n    The \"freeze_support()\" line can be omitted if the program\n    is not going to be frozen to produce an executable.\n
\n\n

Do you have any ideas?

\n\n

Thanks\nMartina

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "221944": "BSON training file extracted produced:\n\n - 378 GB of image files\n - 12,371,293 files\n - 5,270 different categories", "221980": "How do you unarchive bson file?", "221997": "Did you run through the bson and save each as a jpeg or png?", "222004": "I can confirm that training images are 12,371,293 files but my allocated storage disk size is about 81G after extracting images from training.bson. I have used JPEG format.", "222005": "Here's the code I used to extract the images: https://www.kaggle.com/carlossouza/extract-image-files-from-bson-and-save-to-disk/code", "222042": "How much time did you need to extract all images?", "222096": "Less than a day... Started to run the script Friday night, finished on Saturday around noon. \nI'm not happy with the result, my strategy is to use these image files as a fall back option.\nMy main objective today is to figure out a way to feed BSON generator directly into the training procedure. Let's see, if I manage to do it I will post the code :)\nCheers!", "222151": "My disk usage after creating jpeg format was 81G too.", "222169": "Hi Carlos! Let me point a few issues in your script that are not resource optimal:\n\n1. It decodes (`imread`) than encodes (`plt.imsave`) the image in order to work. It is better to read and write the RAW JPEG data from BSON file.\n\n2. You check folder (`os.path.exists`) and file (`os.path.isfile`) existence for every product and image. This is somehow slow and will lead to poor performance. A better approach, though not so conservative, is to read the categories from `category_names.csv` file and create the folders before saving images on it.\n\nWith [these modifications,][1] I could save all train images in less than 30 minutes (on a SSD disk). It uses 82Gb of storage and has the same 12,371,293 files you find.\n\n\n [1]: https://www.kaggle.com/bguberfain/not-so-naive-way-to-convert-bson-to-files/", "222182": "You are right Bruno, thanks a lot!!!\nNevertheless, I figured out a way to feed BSON data straight into Keras, without the need of saving & reading data in the disk! I will post the code during the week :)", "222245": "OMG, Its a large database.", "222363": "CarlosSouza, have you tried [Model.fit_generator][1]? using a generator function you can read and preprocess the images in small batches and then feed then into keras, without exausting your memory!\n\nLater I can post a sample script. \n \n\n\n [1]: https://keras.io/models/model/", "222368": "Can you give me a piece of data that has been processed? I do not know why, running on my computer for nearly 24 hours but nothing to stop the sign. Can also be placed in the clouds! thank you very much!", "222592": "Bad luck! Some Problems happened when I try to build a linux system.", "222989": "i didn't extract anything. i'm piping it on the fly. get those cpu workers going ;-)", "224092": "I get 12,371,293 images as well in training data.\n\nI also got 3095080 images in test data.\n\nIs this what others got?\n\nI did not need 378gb of disk. I just converted the images into jpeg.", "224204": "I get 1,768,182 test images.\n\nI checked the number in File Description:\n\"test.bson - (Size: 14.5 GB) Contains a list of 1,768,182 products in the same format as train.bson...\"\n\nhttps://www.kaggle.com/c/cdiscount-image-classification-challenge/data", "224328": "Each product can have 1-4 images as it is same format as train.bson just no category_id key.\nAlso overview (https://www.kaggle.com/c/cdiscount-image-classification-challenge) says there is over 15 million images.", "224484": "You are right, now I get 3095080 test images. \n\nThanks.", "225044": "That sounds fun, will you mind showing your kernel for this logic only? I have an idea as we can make training image batches and feed those batches onto keras/tensor flow for training the network. But like to validate this understanding with someone like yours. thnx", "225108": "Hi CarlosSouza, Thank you for the code. I have copied it and using it currently on Windows 7 machine and getting the following error mesage. A quick question did you run this on a linux platform. The error message is as follows :\n\nRuntimeError: \n An attempt has been made to start a new process before the\n current process has finished its bootstrapping phase.\n\n This probably means that you are not using fork to start your\n child processes and you have forgotten to use the proper idiom\n in the main module:\n\n if __name__ == '__main__':\n freeze_support()\n ...\n\n The \"freeze_support()\" line can be omitted if the program\n is not going to be frozen to produce an executable.\n\n\nDo you have any ideas?\n\nThanks\nMartina" }, "source": "meta" }