Hello everyone,
\n\nThis is my first competition on Kaggle and I'm quite excited about it, however I'm running into problems which I think are quite elementary and any help would be more than welcome.
\n\nThis is the link to my kernel -- \nhttps://www.kaggle.com/whatvermawhat/resnet50-keras-implementation
\n\nPlease tell me if the link's not working.
\n\nNow coming back to the issue, this is what I've tried so far --
\n\nTried training with the new_whale class -- Horribly overfit, got validation accuracy 1.00 and all it did was predict new_whale class for everything. Do note that I only trained the dense layers and let the ResNet50 convolutional base to non trainable. Tried with and without Dropout layer.
Removed the new_whale class. Made the last block of ResNet50 trainable, removed Dropout layer entirely. Tried training on 10 epochs, got around 0.0x top5 accuracy and it rarely increases. Highest I got was 0.05x
Currently reading Siamese Neural networks implementation along with the paper, interested in it because this is very much a One-Shot learning task as the average examples per class are around 2-3.
Is there anything obvious that I'm doing wrong?
\n\nThank you all for your help in advance!
", "messages": [ { "id": 443428, "postDate": "2018-12-21T16:06:16.510Z", "content": "I didn't go too deep into the code but you are doing wrong preprocessing. Keras resnet50 requires removal of imagenet means and not \"featurewise_center=True, rescale=1./255\"
", "rawMarkdown": "I didn't go too deep into the code but you are doing wrong preprocessing. Keras resnet50 requires removal of imagenet means and not \"featurewise_center=True, rescale=1./255\"\n", "votes": 1, "replies": [ { "id": 443526, "postDate": "2018-12-21T20:02:21.667Z", "content": "Hello Dennis,
\n\nAfter using preprocessing function of Resnet50 provided by Keras I'm getting cleaner results now (12-20% top5_accuracy). Will keep digging more, thank you so much for your response!
", "rawMarkdown": "Hello Dennis,\n\nAfter using preprocessing function of Resnet50 provided by Keras I'm getting cleaner results now (12-20% top5_accuracy). Will keep digging more, thank you so much for your response!" }, { "id": 444843, "postDate": "2018-12-25T01:04:39.387Z", "content": "Hi Abhinav, \nI got the same problem of issue 2, i am using resnet50 to train my classification model, and using the preprocess built in resnet50 of keras. I'm getting almost 20% top_5_acc. So, Have you got a better top_5_acc after removal of imagenet means?????
", "rawMarkdown": "Hi Abhinav, \nI got the same problem of issue 2, i am using resnet50 to train my classification model, and using the preprocess built in resnet50 of keras. I'm getting almost 20% top_5_acc. So, Have you got a better top_5_acc after removal of imagenet means?????" }, { "id": 444847, "postDate": "2018-12-25T01:21:01.637Z", "content": "Hi TsungHan,
\n\nI have tried a number of approaches since my questions and you could try them too if you'd like to,
\n\nI saw a lot GitHub issues where people complained that calling the preprocess_function as argument in ImageDataGenerator wasn't working so instead I create a Keras Lambda layer right before the ResNet50 base which takes in the batch and applies preprocessing to it --
\n\nmodel.add(Lambda(preprocess_input, name='preprocessing', input_shape=(128, 128, 3)))
Instead of using ImageDataGenerator I sampled the entire training dataset in the numpy array and used that, this method gave me my first actual result (26.7% if I'm not wrong).
Used bounding box method and got 28.2 %
If you're getting very low top5 accuracy (on train_set, not on validation_set), I'd suggest that it's highly likely that preprocessing is not being done. I'm not sure what you mean by preprocessing built in resent 50 but I'm assuming you're using preprocess_input() in keras.applications.resnet50.
", "rawMarkdown": "Hi TsungHan,\n\nI have tried a number of approaches since my questions and you could try them too if you'd like to,\n\n1. I saw a lot GitHub issues where people complained that calling the preprocess_function as argument in ImageDataGenerator wasn't working so instead I create a Keras Lambda layer right before the ResNet50 base which takes in the batch and applies preprocessing to it -- \n\n model.add(Lambda(preprocess_input, name='preprocessing', input_shape=(128, 128, 3)))\n\n2. Instead of using ImageDataGenerator I sampled the entire training dataset in the numpy array and used that, this method gave me my first actual result (26.7% if I'm not wrong).\n\n3. Used bounding box method and got 28.2 %\n\nIf you're getting very low top5 accuracy (on train_set, not on validation_set), I'd suggest that it's highly likely that preprocessing is not being done. I'm not sure what you mean by preprocessing built in resent 50 but I'm assuming you're using preprocess_input() in keras.applications.resnet50." }, { "id": 444876, "postDate": "2018-12-25T03:16:54.120Z", "content": "Yes. i'm using preprocess in keras.application.resnet50(). I have a strange problem and have no idea how to solve it, that is: when using learning_rate with 1e-4 or less, the top_5_acc is very low, but train_loss is go down. But when i adjust lr to 1e-3 or higher, top_5_acc is very high(98%), but val_acc is low, and train_loss go up. Is the problem i'm talked above appeared in your code? By the way, i'm not using ImageDataGenerator in my code, i write data_generator by my own.
", "rawMarkdown": "Yes. i'm using preprocess in keras.application.resnet50(). I have a strange problem and have no idea how to solve it, that is: when using learning_rate with 1e-4 or less, the top_5_acc is very low, but train_loss is go down. But when i adjust lr to 1e-3 or higher, top_5_acc is very high(98%), but val_acc is low, and train_loss go up. Is the problem i'm talked above appeared in your code? By the way, i'm not using ImageDataGenerator in my code, i write data_generator by my own." }, { "id": 445320, "postDate": "2018-12-26T07:58:06.380Z", "rawMarkdown": "", "isDeleted": true }, { "id": 445321, "postDate": "2018-12-26T07:58:57.070Z", "content": "I'm not sure about the learning rate problems ( I use a fixed learning rate of 1e-3 throughout ) and since your top5acc is high while the valacc is a prime example of overfitting so you might wanna add a dropout layer and use data augmentation.
", "rawMarkdown": "I'm not sure about the learning rate problems ( I use a fixed learning rate of 1e-3 throughout ) and since your top5acc is high while the valacc is a prime example of overfitting so you might wanna add a dropout layer and use data augmentation." } ] }, { "id": 443291, "postDate": "2018-12-21T11:14:33.723Z", "content": "Hello everyone,
\n\nThis is my first competition on Kaggle and I'm quite excited about it, however I'm running into problems which I think are quite elementary and any help would be more than welcome.
\n\nThis is the link to my kernel -- \nhttps://www.kaggle.com/whatvermawhat/resnet50-keras-implementation
\n\nPlease tell me if the link's not working.
\n\nNow coming back to the issue, this is what I've tried so far --
\n\nTried training with the new_whale class -- Horribly overfit, got validation accuracy 1.00 and all it did was predict new_whale class for everything. Do note that I only trained the dense layers and let the ResNet50 convolutional base to non trainable. Tried with and without Dropout layer.
Removed the new_whale class. Made the last block of ResNet50 trainable, removed Dropout layer entirely. Tried training on 10 epochs, got around 0.0x top5 accuracy and it rarely increases. Highest I got was 0.05x
Currently reading Siamese Neural networks implementation along with the paper, interested in it because this is very much a One-Shot learning task as the average examples per class are around 2-3.
Is there anything obvious that I'm doing wrong?
\n\nThank you all for your help in advance!
", "rawMarkdown": "Hello everyone, \n\nThis is my first competition on Kaggle and I'm quite excited about it, however I'm running into problems which I think are quite elementary and any help would be more than welcome. \n\nThis is the link to my kernel -- \nhttps://www.kaggle.com/whatvermawhat/resnet50-keras-implementation\n\nPlease tell me if the link's not working.\n\nNow coming back to the issue, this is what I've tried so far -- \n\n1. Tried training with the new_whale class -- Horribly overfit, got validation accuracy 1.00 and all it did was predict new_whale class for everything. Do note that I only trained the dense layers and let the ResNet50 convolutional base to non trainable. Tried with and without Dropout layer.\n\n2. Removed the new_whale class. Made the last block of ResNet50 trainable, removed Dropout layer entirely. Tried training on 10 epochs, got around 0.0x top5 accuracy and it rarely increases. Highest I got was 0.05x\n\n3. Currently reading Siamese Neural networks implementation along with the paper, interested in it because this is very much a One-Shot learning task as the average examples per class are around 2-3.\n\nIs there anything obvious that I'm doing wrong?\n\nThank you all for your help in advance!", "votes": -1 } ], "comments": [ { "id": 443428, "author_name": "DennisSakva", "author_url": "", "post_date": "2018-12-21T16:06:16.510000", "content": "I didn't go too deep into the code but you are doing wrong preprocessing. Keras resnet50 requires removal of imagenet means and not \"featurewise_center=True, rescale=1./255\"
", "votes": 1, "replies": [ { "id": 443526, "author_name": "Abhinav Verma", "author_url": "", "post_date": "2018-12-21T20:02:21.667000", "content": "Hello Dennis,
\n\nAfter using preprocessing function of Resnet50 provided by Keras I'm getting cleaner results now (12-20% top5_accuracy). Will keep digging more, thank you so much for your response!
", "votes": 0, "replies": [] }, { "id": 444843, "author_name": "TsungHan", "author_url": "", "post_date": "2018-12-25T01:04:39.387000", "content": "Hi Abhinav, \nI got the same problem of issue 2, i am using resnet50 to train my classification model, and using the preprocess built in resnet50 of keras. I'm getting almost 20% top_5_acc. So, Have you got a better top_5_acc after removal of imagenet means?????
", "votes": 0, "replies": [] }, { "id": 444847, "author_name": "Abhinav Verma", "author_url": "", "post_date": "2018-12-25T01:21:01.637000", "content": "Hi TsungHan,
\n\nI have tried a number of approaches since my questions and you could try them too if you'd like to,
\n\nI saw a lot GitHub issues where people complained that calling the preprocess_function as argument in ImageDataGenerator wasn't working so instead I create a Keras Lambda layer right before the ResNet50 base which takes in the batch and applies preprocessing to it --
\n\nmodel.add(Lambda(preprocess_input, name='preprocessing', input_shape=(128, 128, 3)))
Instead of using ImageDataGenerator I sampled the entire training dataset in the numpy array and used that, this method gave me my first actual result (26.7% if I'm not wrong).
Used bounding box method and got 28.2 %
If you're getting very low top5 accuracy (on train_set, not on validation_set), I'd suggest that it's highly likely that preprocessing is not being done. I'm not sure what you mean by preprocessing built in resent 50 but I'm assuming you're using preprocess_input() in keras.applications.resnet50.
", "votes": 0, "replies": [] }, { "id": 444876, "author_name": "TsungHan", "author_url": "", "post_date": "2018-12-25T03:16:54.120000", "content": "Yes. i'm using preprocess in keras.application.resnet50(). I have a strange problem and have no idea how to solve it, that is: when using learning_rate with 1e-4 or less, the top_5_acc is very low, but train_loss is go down. But when i adjust lr to 1e-3 or higher, top_5_acc is very high(98%), but val_acc is low, and train_loss go up. Is the problem i'm talked above appeared in your code? By the way, i'm not using ImageDataGenerator in my code, i write data_generator by my own.
", "votes": 0, "replies": [] }, { "id": 445320, "author_name": "", "author_url": "", "post_date": "2018-12-26T07:58:06.380000", "content": "", "votes": 0, "replies": [] }, { "id": 445321, "author_name": "Abhinav Verma", "author_url": "", "post_date": "2018-12-26T07:58:57.070000", "content": "I'm not sure about the learning rate problems ( I use a fixed learning rate of 1e-3 throughout ) and since your top5acc is high while the valacc is a prime example of overfitting so you might wanna add a dropout layer and use data augmentation.
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "443428": "I didn't go too deep into the code but you are doing wrong preprocessing. Keras resnet50 requires removal of imagenet means and not \"featurewise_center=True, rescale=1./255\"\n", "443291": "Hello everyone, \n\nThis is my first competition on Kaggle and I'm quite excited about it, however I'm running into problems which I think are quite elementary and any help would be more than welcome. \n\nThis is the link to my kernel -- \nhttps://www.kaggle.com/whatvermawhat/resnet50-keras-implementation\n\nPlease tell me if the link's not working.\n\nNow coming back to the issue, this is what I've tried so far -- \n\n1. Tried training with the new_whale class -- Horribly overfit, got validation accuracy 1.00 and all it did was predict new_whale class for everything. Do note that I only trained the dense layers and let the ResNet50 convolutional base to non trainable. Tried with and without Dropout layer.\n\n2. Removed the new_whale class. Made the last block of ResNet50 trainable, removed Dropout layer entirely. Tried training on 10 epochs, got around 0.0x top5 accuracy and it rarely increases. Highest I got was 0.05x\n\n3. Currently reading Siamese Neural networks implementation along with the paper, interested in it because this is very much a One-Shot learning task as the average examples per class are around 2-3.\n\nIs there anything obvious that I'm doing wrong?\n\nThank you all for your help in advance!" } }