I have just begun solving the problem, and I feel that since the image sizes are 640 X 480, there are too many features too learn. In examples on internet where deep learning is used, typical image size is 26 X 26. I was trying to run SGD, and it seems to be taking a lot of time. Any idea how the number of features could be reduced?
\n\nHere's the code snippet I am using to process an image:
\n\nimage_height = 640\nimage_width = 480# Pixel width and height.\npixel_depth = 255.0 # Number of levels per pixel.\nimage_files = os.listdir(folder)\ndataset = np.ndarray(shape=(len(image_files), image_width, image_height,3), dtype=np.float32) \n\n\nfor image_index, image in enumerate(image_files):\n image_data = (ndimage.imread(image_file).astype(float) - \n pixel_depth / 2) / pixel_depth #normalize\ndataset[image_index, :, :] = image_data\nreturn dataset\n@asmith26 The SUBSET flag will generate a pickled file with just a small portion of the dataset. The model won't perform well when trained on the subset but it's useful for testing the whole script without waiting too long.
@Mustyy Now that you have a dataset, its time to train the model. :)
", "rawMarkdown": "@asmith26 The `SUBSET` flag will generate a pickled file with just a small portion of the dataset. The model won't perform well when trained on the subset but it's useful for testing the whole script without waiting too long.\r\n\r\n@Mustyy Now that you have a dataset, it's time to train the model. :)", "votes": 1 }, { "id": 115203, "postDate": "2016-04-17T03:37:57.453Z", "content": "Hey Jim\nSo the data_20.pkl file is generated but before that it generated pics labeled c0-c9 which are really tiny.\nWon't it be hard to train because of the really small image size?
\n\nSecondly I loaded the pkl file into run_keras_cv.py posted by ZFTurbo but the final log_loss score still stands at 0.138.
\n\nThanks for your reply and help :)
", "rawMarkdown": "Hey Jim\r\nSo the data_20.pkl file is generated but before that it generated pics labeled c0-c9 which are really tiny.\r\nWon't it be hard to train because of the really small image size?\r\n\r\nSecondly I loaded the pkl file into run_keras_cv.py posted by ZFTurbo but the final log_loss score still stands at 0.138. \r\n\r\nThanks for your reply and help :)" }, { "id": 115196, "postDate": "2016-04-17T02:32:48.287Z", "content": "Hey Gauss\nThanks I made it work.\nSo my final log_loss score turns out to be 0.138\nI don't know if this is good or bad with K=10 folds\nSecondly it doesn't save the final score to a csv..any ideas ?
", "rawMarkdown": "Hey Gauss\r\nThanks I made it work.\r\nSo my final log_loss score turns out to be 0.138\r\nI don't know if this is good or bad with K=10 folds\r\nSecondly it doesn't save the final score to a csv..any ideas ?" }, { "id": 115191, "postDate": "2016-04-17T00:37:25.643Z", "content": "Cannot have number of folds n_folds=10 greater than the number of samples: 0.\nMake sure you have put the images in the right folder. The script is probably not finding them and so the "number of samples" ends up being zero.
", "rawMarkdown": " Cannot have number of folds n_folds=10 greater than the number of samples: 0.\r\n\r\nMake sure you have put the images in the right folder. The script is probably not finding them and so the \"number of samples\" ends up being zero.\r\n\r\n" }, { "id": 115190, "postDate": "2016-04-17T00:23:24.910Z", "content": "Yeah I'm trying to train the model following the script ZfTurbo posted but I'm getting an error in the cross validation.\nValueError: Cannot have number of folds n_folds=10 greater than the number of samples: 0.
", "rawMarkdown": "Yeah I'm trying to train the model following the script ZfTurbo posted but I'm getting an error in the cross validation.\r\nValueError: Cannot have number of folds n_folds=10 greater than the number of samples: 0." }, { "id": 115180, "postDate": "2016-04-16T22:44:25.673Z", "content": "Hey thanks for the great insight\nSo what would you recommend as next steps after this stage?
", "rawMarkdown": "Hey thanks for the great insight\r\nSo what would you recommend as next steps after this stage?\r\n" }, { "id": 115130, "postDate": "2016-04-16T14:38:54.027Z", "content": "Downsampling is used to calculate the width and height used to resize the image inside load_image:
WIDTH, HEIGHT = 640 // DOWNSAMPLE, 480 // DOWNSAMPLE\nThe double backslashes represent division with integers.
\n\nHere's the full script: https://github.com/fomorians/distracted-drivers-keras/blob/master/dataset/prep_dataset.py
\n\nIt produces numpy arrays ready for batching, then pickles them. The file main.py loads the pickled object.
Thanks for your reply, but I wanted to know where have you done the resizing in the code i.e used the value of downsample?
", "rawMarkdown": "Thanks for your reply, but I wanted to know where have you done the resizing in the code i.e used the value of downsample?" }, { "id": 115041, "postDate": "2016-04-15T21:14:17.950Z", "content": "Typically with deep learning you want to use either strided convolutions or max pooling to reduce the input size as you go deeper. That said you can also downsample the images significantly (I'm seeing decent results with a factor of 20) using scipy's imresize:
from skimage.io import imread\nfrom scipy.misc import imresize\n\nDOWNSAMPLE = 20\nWIDTH, HEIGHT = 640 // DOWNSAMPLE, 480 // DOWNSAMPLE\n\ndef load_image(path):\n img = imread(path)\n img = imresize(img, (HEIGHT, WIDTH))\n return img\nI have just begun solving the problem, and I feel that since the image sizes are 640 X 480, there are too many features too learn. In examples on internet where deep learning is used, typical image size is 26 X 26. I was trying to run SGD, and it seems to be taking a lot of time. Any idea how the number of features could be reduced?
\n\nHere's the code snippet I am using to process an image:
\n\nimage_height = 640\nimage_width = 480# Pixel width and height.\npixel_depth = 255.0 # Number of levels per pixel.\nimage_files = os.listdir(folder)\ndataset = np.ndarray(shape=(len(image_files), image_width, image_height,3), dtype=np.float32) \n\n\nfor image_index, image in enumerate(image_files):\n image_data = (ndimage.imread(image_file).astype(float) - \n pixel_depth / 2) / pixel_depth #normalize\ndataset[image_index, :, :] = image_data\nreturn dataset\n[quote=Jim Fleming;115130]
\n\nDownsampling is used to calculate the width and height used to resize the image inside load_image:
WIDTH, HEIGHT = 640 // DOWNSAMPLE, 480 // DOWNSAMPLE\nThe double backslashes represent division with integers.
\n\nHere's the full script: https://github.com/fomorians/distracted-drivers-keras/blob/master/dataset/prep_dataset.py
\n\nIt produces numpy arrays ready for batching, then pickles them. The file main.py loads the pickled object.
[/quote]
\n\n@ Jim Fleming - Thanks for this! Just wondering what the "SUBSET=False" is used for in your code?
", "rawMarkdown": "[quote=Jim Fleming;115130]\r\n\r\nDownsampling is used to calculate the width and height used to resize the image inside `load_image`:\r\n\r\n WIDTH, HEIGHT = 640 // DOWNSAMPLE, 480 // DOWNSAMPLE\r\n\r\nThe double backslashes represent division with integers.\r\n\r\nHere's the full script: https://github.com/fomorians/distracted-drivers-keras/blob/master/dataset/prep_dataset.py\r\n\r\nIt produces numpy arrays ready for batching, then pickles them. The file `main.py` loads the pickled object.\r\n\r\n[/quote]\r\n\r\n@ Jim Fleming - Thanks for this! Just wondering what the \"SUBSET=False\" is used for in your code?", "isDeleted": true } ], "comments": [ { "id": 115182, "author_name": "Jim Fleming", "author_url": "", "post_date": "2016-04-16T22:58:26.303000", "content": "@asmith26 The SUBSET flag will generate a pickled file with just a small portion of the dataset. The model won't perform well when trained on the subset but it's useful for testing the whole script without waiting too long.
@Mustyy Now that you have a dataset, its time to train the model. :)
", "votes": 1, "replies": [] }, { "id": 115203, "author_name": "Mustyy", "author_url": "", "post_date": "2016-04-17T03:37:57.453000", "content": "Hey Jim\nSo the data_20.pkl file is generated but before that it generated pics labeled c0-c9 which are really tiny.\nWon't it be hard to train because of the really small image size?
\n\nSecondly I loaded the pkl file into run_keras_cv.py posted by ZFTurbo but the final log_loss score still stands at 0.138.
\n\nThanks for your reply and help :)
", "votes": 0, "replies": [] }, { "id": 115196, "author_name": "Mustyy", "author_url": "", "post_date": "2016-04-17T02:32:48.287000", "content": "Hey Gauss\nThanks I made it work.\nSo my final log_loss score turns out to be 0.138\nI don't know if this is good or bad with K=10 folds\nSecondly it doesn't save the final score to a csv..any ideas ?
", "votes": 0, "replies": [] }, { "id": 115191, "author_name": "gauss256", "author_url": "", "post_date": "2016-04-17T00:37:25.643000", "content": "Cannot have number of folds n_folds=10 greater than the number of samples: 0.\nMake sure you have put the images in the right folder. The script is probably not finding them and so the "number of samples" ends up being zero.
", "votes": 0, "replies": [] }, { "id": 115190, "author_name": "Mustyy", "author_url": "", "post_date": "2016-04-17T00:23:24.910000", "content": "Yeah I'm trying to train the model following the script ZfTurbo posted but I'm getting an error in the cross validation.\nValueError: Cannot have number of folds n_folds=10 greater than the number of samples: 0.
", "votes": 0, "replies": [] }, { "id": 115180, "author_name": "Mustyy", "author_url": "", "post_date": "2016-04-16T22:44:25.673000", "content": "Hey thanks for the great insight\nSo what would you recommend as next steps after this stage?
", "votes": 0, "replies": [] }, { "id": 115130, "author_name": "Jim Fleming", "author_url": "", "post_date": "2016-04-16T14:38:54.027000", "content": "Downsampling is used to calculate the width and height used to resize the image inside load_image:
WIDTH, HEIGHT = 640 // DOWNSAMPLE, 480 // DOWNSAMPLE\nThe double backslashes represent division with integers.
\n\nHere's the full script: https://github.com/fomorians/distracted-drivers-keras/blob/master/dataset/prep_dataset.py
\n\nIt produces numpy arrays ready for batching, then pickles them. The file main.py loads the pickled object.
Thanks for your reply, but I wanted to know where have you done the resizing in the code i.e used the value of downsample?
", "votes": 0, "replies": [] }, { "id": 115041, "author_name": "Jim Fleming", "author_url": "", "post_date": "2016-04-15T21:14:17.950000", "content": "Typically with deep learning you want to use either strided convolutions or max pooling to reduce the input size as you go deeper. That said you can also downsample the images significantly (I'm seeing decent results with a factor of 20) using scipy's imresize:
from skimage.io import imread\nfrom scipy.misc import imresize\n\nDOWNSAMPLE = 20\nWIDTH, HEIGHT = 640 // DOWNSAMPLE, 480 // DOWNSAMPLE\n\ndef load_image(path):\n img = imread(path)\n img = imresize(img, (HEIGHT, WIDTH))\n return img\n[quote=Jim Fleming;115130]
\n\nDownsampling is used to calculate the width and height used to resize the image inside load_image:
WIDTH, HEIGHT = 640 // DOWNSAMPLE, 480 // DOWNSAMPLE\nThe double backslashes represent division with integers.
\n\nHere's the full script: https://github.com/fomorians/distracted-drivers-keras/blob/master/dataset/prep_dataset.py
\n\nIt produces numpy arrays ready for batching, then pickles them. The file main.py loads the pickled object.
[/quote]
\n\n@ Jim Fleming - Thanks for this! Just wondering what the "SUBSET=False" is used for in your code?
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "115182": "@asmith26 The `SUBSET` flag will generate a pickled file with just a small portion of the dataset. The model won't perform well when trained on the subset but it's useful for testing the whole script without waiting too long.\r\n\r\n@Mustyy Now that you have a dataset, it's time to train the model. :)", "115203": "Hey Jim\r\nSo the data_20.pkl file is generated but before that it generated pics labeled c0-c9 which are really tiny.\r\nWon't it be hard to train because of the really small image size?\r\n\r\nSecondly I loaded the pkl file into run_keras_cv.py posted by ZFTurbo but the final log_loss score still stands at 0.138. \r\n\r\nThanks for your reply and help :)", "115196": "Hey Gauss\r\nThanks I made it work.\r\nSo my final log_loss score turns out to be 0.138\r\nI don't know if this is good or bad with K=10 folds\r\nSecondly it doesn't save the final score to a csv..any ideas ?", "115191": " Cannot have number of folds n_folds=10 greater than the number of samples: 0.\r\n\r\nMake sure you have put the images in the right folder. The script is probably not finding them and so the \"number of samples\" ends up being zero.\r\n\r\n", "115190": "Yeah I'm trying to train the model following the script ZfTurbo posted but I'm getting an error in the cross validation.\r\nValueError: Cannot have number of folds n_folds=10 greater than the number of samples: 0.", "115180": "Hey thanks for the great insight\r\nSo what would you recommend as next steps after this stage?\r\n", "115130": "Downsampling is used to calculate the width and height used to resize the image inside `load_image`:\r\n\r\n WIDTH, HEIGHT = 640 // DOWNSAMPLE, 480 // DOWNSAMPLE\r\n\r\nThe double backslashes represent division with integers.\r\n\r\nHere's the full script: https://github.com/fomorians/distracted-drivers-keras/blob/master/dataset/prep_dataset.py\r\n\r\nIt produces numpy arrays ready for batching, then pickles them. The file `main.py` loads the pickled object.", "115084": "Thanks for your reply, but I wanted to know where have you done the resizing in the code i.e used the value of downsample?", "115041": "Typically with deep learning you want to use either strided convolutions or max pooling to reduce the input size as you go deeper. That said you can also downsample the images significantly (I'm seeing decent results with a factor of 20) using scipy's `imresize`:\r\n\r\n from skimage.io import imread\r\n from scipy.misc import imresize\r\n \r\n DOWNSAMPLE = 20\r\n WIDTH, HEIGHT = 640 // DOWNSAMPLE, 480 // DOWNSAMPLE\r\n\r\n def load_image(path):\r\n img = imread(path)\r\n img = imresize(img, (HEIGHT, WIDTH))\r\n return img\r\n\r\n", "115032": "I have just begun solving the problem, and I feel that since the image sizes are 640 X 480, there are too many features too learn. In examples on internet where deep learning is used, typical image size is 26 X 26. I was trying to run SGD, and it seems to be taking a lot of time. Any idea how the number of features could be reduced? \r\n\r\nHere's the code snippet I am using to process an image:\r\n\r\n image_height = 640\r\n image_width = 480# Pixel width and height.\r\n pixel_depth = 255.0 # Number of levels per pixel.\r\n image_files = os.listdir(folder)\r\n dataset = np.ndarray(shape=(len(image_files), image_width, image_height,3), dtype=np.float32) \r\n \r\n\r\n for image_index, image in enumerate(image_files):\r\n image_data = (ndimage.imread(image_file).astype(float) - \r\n pixel_depth / 2) / pixel_depth #normalize\r\n dataset[image_index, :, :] = image_data\r\n return dataset\r\n\r\n", "115173": "[quote=Jim Fleming;115130]\r\n\r\nDownsampling is used to calculate the width and height used to resize the image inside `load_image`:\r\n\r\n WIDTH, HEIGHT = 640 // DOWNSAMPLE, 480 // DOWNSAMPLE\r\n\r\nThe double backslashes represent division with integers.\r\n\r\nHere's the full script: https://github.com/fomorians/distracted-drivers-keras/blob/master/dataset/prep_dataset.py\r\n\r\nIt produces numpy arrays ready for batching, then pickles them. The file `main.py` loads the pickled object.\r\n\r\n[/quote]\r\n\r\n@ Jim Fleming - Thanks for this! Just wondering what the \"SUBSET=False\" is used for in your code?" } }