Hello,\nI am struggling with getting the data into a form that I can get my model to accept. I’m on my phone, and my text has already been deleted by my phone rotating once, so I will try to describe the problem in as much detail as possible but I can’t provide my exact code right now.
\n\nI’m trying to build a very simple cnn with TensorFlow/Keras(as that’s the extent of my learning at this point).\nI know can reshape the data into an array of values that represents the image, but after I do this and attempt to train my model I get an error regarding the dimensionality of my input(I think I’m describing the input values to the model incorrectly).\nI have done extensive google searching of this problem and have found nothing that actually helps me fix the particular problem I am having.\nDoes anyone know a good way to understand how to define input shape to a model? \nIf this question isn’t clear enough I can provide my code when I get home this evening.
", "messages": [ { "id": "701546", "postDate": "12/23/2019 15:42:40", "content": "Hello,\nI am struggling with getting the data into a form that I can get my model to accept. I’m on my phone, and my text has already been deleted by my phone rotating once, so I will try to describe the problem in as much detail as possible but I can’t provide my exact code right now.
\n\nI’m trying to build a very simple cnn with TensorFlow/Keras(as that’s the extent of my learning at this point).\nI know can reshape the data into an array of values that represents the image, but after I do this and attempt to train my model I get an error regarding the dimensionality of my input(I think I’m describing the input values to the model incorrectly).\nI have done extensive google searching of this problem and have found nothing that actually helps me fix the particular problem I am having.\nDoes anyone know a good way to understand how to define input shape to a model? \nIf this question isn’t clear enough I can provide my code when I get home this evening.
", "rawMarkdown": "Hello,\nI am struggling with getting the data into a form that I can get my model to accept. I’m on my phone, and my text has already been deleted by my phone rotating once, so I will try to describe the problem in as much detail as possible but I can’t provide my exact code right now.\n\nI’m trying to build a very simple cnn with TensorFlow/Keras(as that’s the extent of my learning at this point).\nI know can reshape the data into an array of values that represents the image, but after I do this and attempt to train my model I get an error regarding the dimensionality of my input(I think I’m describing the input values to the model incorrectly).\nI have done extensive google searching of this problem and have found nothing that actually helps me fix the particular problem I am having.\nDoes anyone know a good way to understand how to define input shape to a model? \nIf this question isn’t clear enough I can provide my code when I get home this evening.", "votes": null }, { "id": "701563", "postDate": "12/23/2019 16:00:18", "content": "The problem might be with the number of channels. The common simple image classification examples have 3 channels. The bengali grapheme has one, I think.
", "rawMarkdown": "The problem might be with the number of channels. The common simple image classification examples have 3 channels. The bengali grapheme has one, I think.", "votes": null }, { "id": "701616", "postDate": "12/23/2019 17:08:27", "content": "Try to post also the error you get and:\n- theinput_shape of your model \n- dimensions of X, y tensors that go to model.fit(X,y,...)
I have realized now that my original problem involved not feeding the model image shaped arrays, as in I wasn't passing those inputs into the model, but now that I have a list of image shaped arrays I am getting a different error.
\n\nI currently have a list of arrays that are 64x64(converted using code from a public notebook on this competition. i try to define input shape(64, 64, 1)
\n\nand I get the error:\nValueError: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 1 array(s), but instead got the following list of 50210 arrays: [array([[253, 252, 253, ..., 254, 254, 253],
\n\nis there a way to tell my model to expect a list of arrays?
", "rawMarkdown": "I have realized now that my original problem involved not feeding the model image shaped arrays, as in I wasn't passing those inputs into the model, but now that I have a list of image shaped arrays I am getting a different error.\n\nI currently have a list of arrays that are 64x64(converted using code from a public notebook on this competition. i try to define input shape(64, 64, 1)\n\nand I get the error:\nValueError: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 1 array(s), but instead got the following list of 50210 arrays: [array([[253, 252, 253, ..., 254, 254, 253],\n\nis there a way to tell my model to expect a list of arrays?", "votes": null }, { "id": "701945", "postDate": "12/24/2019 04:56:41", "content": "It looks like it's feeding the whole dataset. The array you are looping through might be an array in an array like [[[1,2,3],[4,5,6],[7,8,9]]]. Instead of looping [1,2,3] then [4,5,6] then [7,8,9], it loops only loops the [[1,2,3],[4,5,6],[7,8,9]] array. If this is the case, you can change the for loop from:\nfor array in dataset:\nto\nfor array in dataset[0]:
So I need to manually unpack my array before I try to feed anything to my model? Do you happen to know of an article or something that can demonstrate this process? One item say,\nx_train[0] is:\narray([[252, 254, 254, ..., 251, 249, 250],\n [252, 254, 253, ..., 253, 250, 248],\n [253, 253, 253, ..., 252, 251, 249],\n ...,\n [251, 253, 253, ..., 252, 251, 251],\n [251, 253, 253, ..., 251, 250, 250],\n [251, 253, 253, ..., 251, 251, 250]], dtype=uint8)\nand currently I'm feeding all of my array, so I understand why that is wrong, but I don't understand how to feed all of the data to the model.
", "rawMarkdown": "So I need to manually unpack my array before I try to feed anything to my model? Do you happen to know of an article or something that can demonstrate this process? One item say,\nx_train[0] is:\narray([[252, 254, 254, ..., 251, 249, 250],\n [252, 254, 253, ..., 253, 250, 248],\n [253, 253, 253, ..., 252, 251, 249],\n ...,\n [251, 253, 253, ..., 252, 251, 251],\n [251, 253, 253, ..., 251, 250, 250],\n [251, 253, 253, ..., 251, 251, 250]], dtype=uint8)\nand currently I'm feeding all of my array, so I understand why that is wrong, but I don't understand how to feed all of the data to the model.", "votes": null }, { "id": "703275", "postDate": "12/25/2019 23:06:18", "content": "not sure but maybenp.concatenate(x_train) will do the work. \nBut how you end up there? check that first, or try to post the input and dimensions you are giving to the reshapefunction.
A snippet that works and you might find it helpful:
\n\n```
\n\nX = train_df.drop(['grapheme_root', 'vowel_diacritic', 'consonant_diacritic'], axis=1)
\n\nX = resize_img(X) # custom function to resize my images (returns images IMG_SIZExIMG_SIZE)\nX = X/255 # normalize pixel values to 0-1
\n\nX = X.values.reshape(-1, IMG_SIZE, IMG_SIZE, NO_CHANNELS) # reshape to 4d\n``\nwhere,IMG_SIZEset yours andNO_CHANNELS=1`
Something similar to this(I think) actually worked for me. when I called reshape like this:\nX.values.reshape(n_images, img_size, img_size, 1) (n_images is the number of images)\nI was able to set the arguments to input shape as (img_size, img_size, 1) and it accepted the data. \nI found this out mainly by experimenting and reading advice given to people trying to solve similar problems, and I think your solution is doing something similar, thank you for the assistance.
", "rawMarkdown": "Something similar to this(I think) actually worked for me. when I called reshape like this:\nX.values.reshape(n_images, img_size, img_size, 1) (n_images is the number of images)\nI was able to set the arguments to input shape as (img_size, img_size, 1) and it accepted the data. \nI found this out mainly by experimenting and reading advice given to people trying to solve similar problems, and I think your solution is doing something similar, thank you for the assistance.", "votes": null }, { "id": "703882", "postDate": "12/26/2019 18:55:13", "content": "It is pretty simple. What I did was : model.fit([train_X],[train_y]) .
The Order of dims does matter when you feed the images to the model. In pytorch, [batch_size, channels, width,height] is valid order to feed. If the order is different, then you can permute these by adding .permute()\nkeras and tensorflow are having different order that i am knowing. Just make sure it is valid order.
", "rawMarkdown": "The Order of dims does matter when you feed the images to the model. In pytorch, [batch_size, channels, width,height] is valid order to feed. If the order is different, then you can permute these by adding .permute()\nkeras and tensorflow are having different order that i am knowing. Just make sure it is valid order.", "votes": null } ], "comments": [ { "id": 701563, "author_name": "notjohnsmith", "author_url": "", "post_date": "12/23/2019 16:00:18", "content": "The problem might be with the number of channels. The common simple image classification examples have 3 channels. The bengali grapheme has one, I think.
", "votes": null, "replies": [] }, { "id": 701616, "author_name": "imeintanis", "author_url": "", "post_date": "12/23/2019 17:08:27", "content": "Try to post also the error you get and:\n- theinput_shape of your model \n- dimensions of X, y tensors that go to model.fit(X,y,...)
I have realized now that my original problem involved not feeding the model image shaped arrays, as in I wasn't passing those inputs into the model, but now that I have a list of image shaped arrays I am getting a different error.
\n\nI currently have a list of arrays that are 64x64(converted using code from a public notebook on this competition. i try to define input shape(64, 64, 1)
\n\nand I get the error:\nValueError: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 1 array(s), but instead got the following list of 50210 arrays: [array([[253, 252, 253, ..., 254, 254, 253],
\n\nis there a way to tell my model to expect a list of arrays?
", "votes": null, "replies": [ { "id": 701945, "author_name": "notjohnsmith", "author_url": "", "post_date": "12/24/2019 04:56:41", "content": "It looks like it's feeding the whole dataset. The array you are looping through might be an array in an array like [[[1,2,3],[4,5,6],[7,8,9]]]. Instead of looping [1,2,3] then [4,5,6] then [7,8,9], it loops only loops the [[1,2,3],[4,5,6],[7,8,9]] array. If this is the case, you can change the for loop from:\nfor array in dataset:\nto\nfor array in dataset[0]:
So I need to manually unpack my array before I try to feed anything to my model? Do you happen to know of an article or something that can demonstrate this process? One item say,\nx_train[0] is:\narray([[252, 254, 254, ..., 251, 249, 250],\n [252, 254, 253, ..., 253, 250, 248],\n [253, 253, 253, ..., 252, 251, 249],\n ...,\n [251, 253, 253, ..., 252, 251, 251],\n [251, 253, 253, ..., 251, 250, 250],\n [251, 253, 253, ..., 251, 251, 250]], dtype=uint8)\nand currently I'm feeding all of my array, so I understand why that is wrong, but I don't understand how to feed all of the data to the model.
", "votes": null, "replies": [] }, { "id": 703275, "author_name": "imeintanis", "author_url": "", "post_date": "12/25/2019 23:06:18", "content": "not sure but maybenp.concatenate(x_train) will do the work. \nBut how you end up there? check that first, or try to post the input and dimensions you are giving to the reshapefunction.
A snippet that works and you might find it helpful:
\n\n```
\n\nX = train_df.drop(['grapheme_root', 'vowel_diacritic', 'consonant_diacritic'], axis=1)
\n\nX = resize_img(X) # custom function to resize my images (returns images IMG_SIZExIMG_SIZE)\nX = X/255 # normalize pixel values to 0-1
\n\nX = X.values.reshape(-1, IMG_SIZE, IMG_SIZE, NO_CHANNELS) # reshape to 4d\n``\nwhere,IMG_SIZEset yours andNO_CHANNELS=1`
Something similar to this(I think) actually worked for me. when I called reshape like this:\nX.values.reshape(n_images, img_size, img_size, 1) (n_images is the number of images)\nI was able to set the arguments to input shape as (img_size, img_size, 1) and it accepted the data. \nI found this out mainly by experimenting and reading advice given to people trying to solve similar problems, and I think your solution is doing something similar, thank you for the assistance.
", "votes": null, "replies": [] }, { "id": 703882, "author_name": "unkownhihi", "author_url": "", "post_date": "12/26/2019 18:55:13", "content": "It is pretty simple. What I did was : model.fit([train_X],[train_y]) .
The Order of dims does matter when you feed the images to the model. In pytorch, [batch_size, channels, width,height] is valid order to feed. If the order is different, then you can permute these by adding .permute()\nkeras and tensorflow are having different order that i am knowing. Just make sure it is valid order.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "701546": "Hello,\nI am struggling with getting the data into a form that I can get my model to accept. I’m on my phone, and my text has already been deleted by my phone rotating once, so I will try to describe the problem in as much detail as possible but I can’t provide my exact code right now.\n\nI’m trying to build a very simple cnn with TensorFlow/Keras(as that’s the extent of my learning at this point).\nI know can reshape the data into an array of values that represents the image, but after I do this and attempt to train my model I get an error regarding the dimensionality of my input(I think I’m describing the input values to the model incorrectly).\nI have done extensive google searching of this problem and have found nothing that actually helps me fix the particular problem I am having.\nDoes anyone know a good way to understand how to define input shape to a model? \nIf this question isn’t clear enough I can provide my code when I get home this evening.", "701563": "The problem might be with the number of channels. The common simple image classification examples have 3 channels. The bengali grapheme has one, I think.", "701616": "Try to post also the error you get and:\n- the` input_shape` of your model \n- dimensions of `X, y` tensors that go to `model.fit(X,y,...)`", "701879": "I have realized now that my original problem involved not feeding the model image shaped arrays, as in I wasn't passing those inputs into the model, but now that I have a list of image shaped arrays I am getting a different error.\n\nI currently have a list of arrays that are 64x64(converted using code from a public notebook on this competition. i try to define input shape(64, 64, 1)\n\nand I get the error:\nValueError: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 1 array(s), but instead got the following list of 50210 arrays: [array([[253, 252, 253, ..., 254, 254, 253],\n\nis there a way to tell my model to expect a list of arrays?", "701945": "It looks like it's feeding the whole dataset. The array you are looping through might be an array in an array like [[[1,2,3],[4,5,6],[7,8,9]]]. Instead of looping [1,2,3] then [4,5,6] then [7,8,9], it loops only loops the [[1,2,3],[4,5,6],[7,8,9]] array. If this is the case, you can change the for loop from:\n`for array in dataset:`\nto\n`for array in dataset[0]:`", "703269": "So I need to manually unpack my array before I try to feed anything to my model? Do you happen to know of an article or something that can demonstrate this process? One item say,\nx_train[0] is:\narray([[252, 254, 254, ..., 251, 249, 250],\n [252, 254, 253, ..., 253, 250, 248],\n [253, 253, 253, ..., 252, 251, 249],\n ...,\n [251, 253, 253, ..., 252, 251, 251],\n [251, 253, 253, ..., 251, 250, 250],\n [251, 253, 253, ..., 251, 251, 250]], dtype=uint8)\nand currently I'm feeding all of my array, so I understand why that is wrong, but I don't understand how to feed all of the data to the model.", "703275": "not sure but maybe` np.concatenate(x_train)` will do the work. \nBut how you end up there? check that first, or try to post the input and dimensions you are giving to the `reshape `function.\n\nA snippet that works and you might find it helpful: \n\n```\n# train_df is the df loaded from train.csv\nX = train_df.drop(['grapheme_root', 'vowel_diacritic', 'consonant_diacritic'], axis=1)\n\nX = resize_img(X) # custom function to resize my images (returns images IMG_SIZExIMG_SIZE)\nX = X/255 # normalize pixel values to 0-1\n\nX = X.values.reshape(-1, IMG_SIZE, IMG_SIZE, NO_CHANNELS) # reshape to 4d\n```\nwhere, `IMG_SIZE` set yours and `NO_CHANNELS=1`", "703620": "Something similar to this(I think) actually worked for me. when I called reshape like this:\nX.values.reshape(n_images, img_size, img_size, 1) (n_images is the number of images)\nI was able to set the arguments to input shape as (img_size, img_size, 1) and it accepted the data. \nI found this out mainly by experimenting and reading advice given to people trying to solve similar problems, and I think your solution is doing something similar, thank you for the assistance.", "703882": "It is pretty simple. What I did was : `model.fit([train_X],[train_y])` .", "703942": "The Order of dims does matter when you feed the images to the model. In pytorch, [batch_size, channels, width,height] is valid order to feed. If the order is different, then you can permute these by adding .permute()\nkeras and tensorflow are having different order that i am knowing. Just make sure it is valid order." }, "source": "meta" }