Please could you help me understand how to split the images available in the 'train_images' folder to separate folders for each class based on 'train.csv'?
", "messages": [ { "id": "1155256", "postDate": "01/16/2021 10:29:47", "content": "Please could you help me understand how to split the images available in the 'train_images' folder to separate folders for each class based on 'train.csv'?
", "rawMarkdown": "Please could you help me understand how to split the images available in the 'train_images' folder to separate folders for each class based on 'train.csv'?", "votes": null }, { "id": "1155486", "postDate": "01/16/2021 12:22:58", "content": "Try keras preprocessing ImageDataGenerator.flow_from_dataframe method.
\nMust visit Tensorflow doc and this tutorial article for same.
# This the quick implementation of ImageDataGenerator.flow_from_dataframe\n\nimport pandas as pd\nfrom keras_preprocessing.image import ImageDataGenerator\n\ndf = pd.read_csv('train.csv')\ndatagen = ImageDataGenerator(rescale=1./255)\n\ntrain_generator = datagen.flow_from_dataframe(dataframe=df, directory='.\\train_imgs', x_col='image_id', y_col='label', class_mode=\"categorical\", target_size=(256,256), batch_size=256)\nThis train_generator returns A DataFrameIterator yielding tuples of (x, y) where x is a NumPy array containing a batch of images with shape (batch_size, *target_size, channels) and y is a NumPy array of corresponding labels.
", "rawMarkdown": "Try keras preprocessing **ImageDataGenerator.flow_from_dataframe** method.\nMust visit [Tensorflow doc](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator#flow_from_directory) and [this tutorial article](https://vijayabhaskar96.medium.com/tutorial-on-keras-imagedatagenerator-with-flow-from-dataframe-8bd5776e45c1) for same.\n\n```\n# This the quick implementation of ImageDataGenerator.flow_from_dataframe\n\nimport pandas as pd\nfrom keras_preprocessing.image import ImageDataGenerator\n\ndf = pd.read_csv('train.csv')\ndatagen = ImageDataGenerator(rescale=1./255)\n\ntrain_generator = datagen.flow_from_dataframe(dataframe=df, directory='.\\train_imgs', x_col='image_id', y_col='label', class_mode=\"categorical\", target_size=(256,256), batch_size=256)\n```\n\nThis train_generator returns A **DataFrameIterator** yielding tuples of **(x, y)** where **x** is a NumPy array containing a batch of images with shape **(batch_size, *target_size, channels)** and **y** is a NumPy array of corresponding **labels**.", "votes": null }, { "id": "1156149", "postDate": "01/17/2021 00:38:59", "content": "There are many ways to split the images in train.csv. One of the basic ways is to simply split using sklearn's
\ntrain_test_split().
Let say you have 1000 images in your dataset. You do test_train_split set to 0.8. X_train will have 800 images and Valid will have 800 labels corresponding to the 800 images in it, X_test will have 200 images and Y_test will have 200 labels. That's your train and test set.
\nThere are many other ways such as stratified kfolding and such, but these are usually the basics of splitting your data. Hope this helps!
", "rawMarkdown": "There are many ways to split the images in train.csv. One of the basic ways is to simply split using sklearn's\n```train_test_split()```. \n\nLet say you have 1000 images in your dataset. You do test_train_split set to 0.8. X_train will have 800 images and Valid will have 800 labels corresponding to the 800 images in it, X_test will have 200 images and Y_test will have 200 labels. That's your train and test set.\n\nThere are many other ways such as stratified kfolding and such, but these are usually the basics of splitting your data. Hope this helps!", "votes": null }, { "id": "1156732", "postDate": "01/17/2021 11:22:14", "content": "You basically have to traverse through the 'train.csv' file and read the labels (from 0 to 4) .Create 5 different folders corresponding to each of the five categories in your directory. Then copy the images from the 'train_images' folder to the folders that you have created by comparing the labels using the code:
\n\n\nshutil.copy(src_path,dest_path)
\n
After you have done this, images would be split into five folders. Then to split train and test images run the code:
\n\n\nsplitfolders.ratio(src_path, output=dest_path, seed=1337, ratio=(.8, .2), group_prefix=None)
\n
Here the source path is where you have split your 'train_images' folder images into five folders . The destination path is of your choice. Hope this helps!!
", "rawMarkdown": "You basically have to traverse through the 'train.csv' file and read the labels (from 0 to 4) .Create 5 different folders corresponding to each of the five categories in your directory. Then copy the images from the 'train_images' folder to the folders that you have created by comparing the labels using the code:\n> shutil.copy(src_path,dest_path)\n\n\nAfter you have done this, images would be split into five folders. Then to split train and test images run the code:\n> splitfolders.ratio(src_path, output=dest_path, seed=1337, ratio=(.8, .2), group_prefix=None)\n\nHere the source path is where you have split your 'train_images' folder images into five folders . The destination path is of your choice. Hope this helps!!", "votes": null }, { "id": "1161118", "postDate": "01/20/2021 10:42:23", "content": "Thank you all, you guys are really awesome!!!
\nI was able to read the 'train.csv' file using a custom function and was able to create separate folders for each class.
\nAlso, tried using flow_from_dataframe() and it works as well.
Then, used splitfolders.ratio() to split the images into train, val, and test.
\nI've used training_test_split() for non-image data before. Maybe, I'll try this for image data next time.
Have just completed preprocessing and started to train the model. Good luck with your models. :)
\nThanks again,
\nArun
Try keras preprocessing ImageDataGenerator.flow_from_dataframe method.
\nMust visit Tensorflow doc and this tutorial article for same.
# This the quick implementation of ImageDataGenerator.flow_from_dataframe\n\nimport pandas as pd\nfrom keras_preprocessing.image import ImageDataGenerator\n\ndf = pd.read_csv('train.csv')\ndatagen = ImageDataGenerator(rescale=1./255)\n\ntrain_generator = datagen.flow_from_dataframe(dataframe=df, directory='.\\train_imgs', x_col='image_id', y_col='label', class_mode=\"categorical\", target_size=(256,256), batch_size=256)\nThis train_generator returns A DataFrameIterator yielding tuples of (x, y) where x is a NumPy array containing a batch of images with shape (batch_size, *target_size, channels) and y is a NumPy array of corresponding labels.
", "votes": null, "replies": [] }, { "id": 1156149, "author_name": "andyjianzhou", "author_url": "", "post_date": "01/17/2021 00:38:59", "content": "There are many ways to split the images in train.csv. One of the basic ways is to simply split using sklearn's
\ntrain_test_split().
Let say you have 1000 images in your dataset. You do test_train_split set to 0.8. X_train will have 800 images and Valid will have 800 labels corresponding to the 800 images in it, X_test will have 200 images and Y_test will have 200 labels. That's your train and test set.
\nThere are many other ways such as stratified kfolding and such, but these are usually the basics of splitting your data. Hope this helps!
", "votes": null, "replies": [] }, { "id": 1156732, "author_name": "shameemarmy", "author_url": "", "post_date": "01/17/2021 11:22:14", "content": "You basically have to traverse through the 'train.csv' file and read the labels (from 0 to 4) .Create 5 different folders corresponding to each of the five categories in your directory. Then copy the images from the 'train_images' folder to the folders that you have created by comparing the labels using the code:
\n\n\nshutil.copy(src_path,dest_path)
\n
After you have done this, images would be split into five folders. Then to split train and test images run the code:
\n\n\nsplitfolders.ratio(src_path, output=dest_path, seed=1337, ratio=(.8, .2), group_prefix=None)
\n
Here the source path is where you have split your 'train_images' folder images into five folders . The destination path is of your choice. Hope this helps!!
", "votes": null, "replies": [] }, { "id": 1161118, "author_name": "arun728", "author_url": "", "post_date": "01/20/2021 10:42:23", "content": "Thank you all, you guys are really awesome!!!
\nI was able to read the 'train.csv' file using a custom function and was able to create separate folders for each class.
\nAlso, tried using flow_from_dataframe() and it works as well.
Then, used splitfolders.ratio() to split the images into train, val, and test.
\nI've used training_test_split() for non-image data before. Maybe, I'll try this for image data next time.
Have just completed preprocessing and started to train the model. Good luck with your models. :)
\nThanks again,
\nArun