{ "id": 316882, "title": "Test accuracy remain below 5% - Efficientnet B1 Keras", "url": "/competitions/sorghum-id-fgvc-9/discussion/316882", "author_name": "", "post_date": "2022-04-04T12:04:01.724756500Z", "votes": 3, "comment_count": 13, "views": 0, "content": "

Hello to everyone,

\n

What a nice topic, predicting plant species over this huge dataset of images.
\nIt is a competition, but as there is no points or money involved, I hope I can find a little bit of help.

\n

I started with confidence, but after a few attempts, I am not sure why my model prediction accuracy remains so low when the training and validation accuracy are over 80% …

\n

I would like to use ImageDataGenerator and flow_from_dataframe to get the simplest pipeline possible with one the fly data augmentation.

\n

I wish you could guide me with some comments or advice, so I shared the code for my preprocessing, training and prediction here :

\n

Kaggle Notebook

\n

Looking forward for your feedback.

\n

Edit : Found the problem, it was in the treatment of the prediction classes, the notebook is now achieving 57% accuracy. Don't hesitate if you have other comments or questions.

", "messages": [ { "id": "1744884", "postDate": "04/04/2022 12:04:01", "content": "

Hello to everyone,

\n

What a nice topic, predicting plant species over this huge dataset of images.
\nIt is a competition, but as there is no points or money involved, I hope I can find a little bit of help.

\n

I started with confidence, but after a few attempts, I am not sure why my model prediction accuracy remains so low when the training and validation accuracy are over 80% …

\n

I would like to use ImageDataGenerator and flow_from_dataframe to get the simplest pipeline possible with one the fly data augmentation.

\n

I wish you could guide me with some comments or advice, so I shared the code for my preprocessing, training and prediction here :

\n

Kaggle Notebook

\n

Looking forward for your feedback.

\n

Edit : Found the problem, it was in the treatment of the prediction classes, the notebook is now achieving 57% accuracy. Don't hesitate if you have other comments or questions.

", "rawMarkdown": "Hello to everyone,\n\nWhat a nice topic, predicting plant species over this huge dataset of images.\nIt is a competition, but as there is no points or money involved, I hope I can find a little bit of help.\n\nI started with confidence, but after a few attempts, I am not sure why my model prediction accuracy remains so low when the training and validation accuracy are over 80% ...\n\nI would like to use ImageDataGenerator and flow_from_dataframe to get the simplest pipeline possible with one the fly data augmentation.\n\nI wish you could guide me with some comments or advice, so I shared the code for my preprocessing, training and prediction here :\n\n[Kaggle Notebook](url[https://www.kaggle.com/code/laurentpoyet/sorghum-100-preprocessing-and-training])\n\nLooking forward for your feedback. \n\nEdit : Found the problem, it was in the treatment of the prediction classes, the notebook is now achieving 57% accuracy. Don't hesitate if you have other comments or questions.", "votes": null }, { "id": "1747058", "postDate": "04/06/2022 09:58:03", "content": "

So do I. 😭

", "rawMarkdown": "So do I. 😭", "votes": null }, { "id": "1747138", "postDate": "04/06/2022 12:09:30", "content": "

Thanks for sharing your code also, I can't achieve more than 0.05 accuracy using ImageDataGenerator and flow_from_dataframe method …

\n

With other methods the model performs quite good however …

", "rawMarkdown": "Thanks for sharing your code also, I can't achieve more than 0.05 accuracy using ImageDataGenerator and flow_from_dataframe method ...\n\nWith other methods the model performs quite good however ...", "votes": null }, { "id": "1754363", "postDate": "04/13/2022 15:43:40", "content": "

You can try to use a higher learning rate to help the train_loss to drop faster(maybe… below 2.0 in the first 5 epochs), and a smaller batch size to help your model perform better on the test dataset.

", "rawMarkdown": "You can try to use a higher learning rate to help the train_loss to drop faster(maybe... below 2.0 in the first 5 epochs), and a smaller batch size to help your model perform better on the test dataset.", "votes": null }, { "id": "1754474", "postDate": "04/13/2022 17:56:02", "content": "

Thanks for your advice, I reduced the batch size and will set the initial LR at 0.1 with reduceLRonplateau. Just waiting to recover my GPU to commit the notebook. What image size do you use for optimal results ?

\n

Edit : From first trails, accuracy remains low with LR higher than 0,01 with effnetB1 and batchsize 16.

", "rawMarkdown": "Thanks for your advice, I reduced the batch size and will set the initial LR at 0.1 with reduceLRonplateau. Just waiting to recover my GPU to commit the notebook. What image size do you use for optimal results ?\n\nEdit : From first trails, accuracy remains low with LR higher than 0,01 with effnetB1 and batchsize 16.", "votes": null }, { "id": "1755888", "postDate": "04/15/2022 04:51:35", "content": "

lr: it depends on your model. if you find it is too high for your model, you can try to set a lower one. I tried 1e-4 with effnetb4, and got more than 0.9 train_acc and val_acc, but only 0.72 test_acc. With lr warm up and cosine annealing, the test_acc can reach about 0.8.
\nimage_size=512
\nbatch_size=16

", "rawMarkdown": "lr: it depends on your model. if you find it is too high for your model, you can try to set a lower one. I tried 1e-4 with effnetb4, and got more than 0.9 train_acc and val_acc, but only 0.72 test_acc. With lr warm up and cosine annealing, the test_acc can reach about 0.8.\nimage_size=512\nbatch_size=16", "votes": null }, { "id": "1759282", "postDate": "04/18/2022 13:54:43", "content": "

What does the score mean on the leaderboard? I used eff v2 large with a few tricks, 99.5% ACC on Val, but got only 0.845 on the test set.

", "rawMarkdown": "What does the score mean on the leaderboard? I used eff v2 large with a few tricks, 99.5% ACC on Val, but got only 0.845 on the test set.", "votes": null }, { "id": "1760883", "postDate": "04/19/2022 15:09:39", "content": "

Overfitting probably ; plus are you using kfolds or simple train val split ?

", "rawMarkdown": "Overfitting probably ; plus are you using kfolds or simple train val split ?", "votes": null }, { "id": "1783337", "postDate": "05/10/2022 10:01:14", "content": "

I have the same problem you described here. My val_acc is around 85% but on the test set I get very poor results. I'm trying to figure out what I'm doing wrong during prediction

", "rawMarkdown": "I have the same problem you described here. My val_acc is around 85% but on the test set I get very poor results. I'm trying to figure out what I'm doing wrong during prediction", "votes": null }, { "id": "1783551", "postDate": "05/10/2022 13:23:29", "content": "

I was not associating the correct labels to my predictions, and was obtaining around 0.05 score. So first step is to be sure that your predictions are associated to the correct label.
\nI updated my notebook accordingly to solve this problem using flow_from_dataframe() method with Tensorflow

", "rawMarkdown": "I was not associating the correct labels to my predictions, and was obtaining around 0.05 score. So first step is to be sure that your predictions are associated to the correct label.\nI updated my notebook accordingly to solve this problem using flow_from_dataframe() method with Tensorflow", "votes": null }, { "id": "1784674", "postDate": "05/11/2022 11:20:45", "content": "

Hi, is the ImageDataGenerator and flow_from_dataframe method working for you?
\nI have tried this method but I getting very low initial accuracy in training and validation equal to .01 in the 1st epoch. I am not able to figure out the problem here.

", "rawMarkdown": "Hi, is the ImageDataGenerator and flow_from_dataframe method working for you?\nI have tried this method but I getting very low initial accuracy in training and validation equal to .01 in the 1st epoch. I am not able to figure out the problem here.", "votes": null }, { "id": "1790914", "postDate": "05/15/2022 13:10:53", "content": "

Hi @laurentpoyet . Pls, share the link to your notebook. I too got accuracy above 90% on the train and Val data. Initially, i was too getting 0.01 on test data but then I indexed labels alphabetically and specified class_names in flow_from_dataframe. Since then I'm getting max of 50% accuracy on test data. Since I'm new to this, I have few doubts. It'll be really helpful if u help me with this:
\n1) What will be the major changes in code for FGVC and normal Image classification?
\n2) Why are we training the whole model instead of converging on top layers and then fine-tuning?
\n3) Is the high bias or overfitting due to training all layers of model to the dataset?

", "rawMarkdown": "Hi @laurentpoyet . Pls, share the link to your notebook. I too got accuracy above 90% on the train and Val data. Initially, i was too getting 0.01 on test data but then I indexed labels alphabetically and specified class_names in flow_from_dataframe. Since then I'm getting max of 50% accuracy on test data. Since I'm new to this, I have few doubts. It'll be really helpful if u help me with this:\n1) What will be the major changes in code for FGVC and normal Image classification?\n2) Why are we training the whole model instead of converging on top layers and then fine-tuning?\n3) Is the high bias or overfitting due to training all layers of model to the dataset?", "votes": null }, { "id": "1790918", "postDate": "05/15/2022 13:13:36", "content": "

Hi @mersotc . could you pls share the link to your notebook. I too got train and val accuracy >90% but not able to get test accuracy more than 0.5

", "rawMarkdown": "Hi @mersotc . could you pls share the link to your notebook. I too got train and val accuracy >90% but not able to get test accuracy more than 0.5", "votes": null }, { "id": "1791913", "postDate": "05/16/2022 13:24:24", "content": "

Hi,

\n

You can find the notebook here :

\n

www.kaggle.com/code/laurentpoyet/sorghum-100-effnetb1
\n

\n

1) it is a normal image classification problem
\n2) There is no need to train the model completely, fine tuning would be to train some layers, then freeze these layers and train only last layers, which I didn't do in this notebook. However I didn't achieve better results doing this.
\n3) Personally I didn't experience overfitting in this case, neither did I reach better results by reducing the trainable layers in the model.

", "rawMarkdown": "Hi,\n\nYou can find the notebook here :\n\n[www.kaggle.com/code/laurentpoyet/sorghum-100-effnetb1\n](url)\n\n1) it is a normal image classification problem\n2) There is no need to train the model completely, fine tuning would be to train some layers, then freeze these layers and train only last layers, which I didn't do in this notebook. However I didn't achieve better results doing this.\n3) Personally I didn't experience overfitting in this case, neither did I reach better results by reducing the trainable layers in the model.", "votes": null } ], "comments": [ { "id": 1747058, "author_name": "waynewhying", "author_url": "", "post_date": "04/06/2022 09:58:03", "content": "

So do I. 😭

", "votes": null, "replies": [ { "id": 1747138, "author_name": "laurentpoyet", "author_url": "", "post_date": "04/06/2022 12:09:30", "content": "

Thanks for sharing your code also, I can't achieve more than 0.05 accuracy using ImageDataGenerator and flow_from_dataframe method …

\n

With other methods the model performs quite good however …

", "votes": null, "replies": [ { "id": 1754363, "author_name": "mersotc", "author_url": "", "post_date": "04/13/2022 15:43:40", "content": "

You can try to use a higher learning rate to help the train_loss to drop faster(maybe… below 2.0 in the first 5 epochs), and a smaller batch size to help your model perform better on the test dataset.

", "votes": null, "replies": [] } ] }, { "id": 1754474, "author_name": "laurentpoyet", "author_url": "", "post_date": "04/13/2022 17:56:02", "content": "

Thanks for your advice, I reduced the batch size and will set the initial LR at 0.1 with reduceLRonplateau. Just waiting to recover my GPU to commit the notebook. What image size do you use for optimal results ?

\n

Edit : From first trails, accuracy remains low with LR higher than 0,01 with effnetB1 and batchsize 16.

", "votes": null, "replies": [] }, { "id": 1755888, "author_name": "mersotc", "author_url": "", "post_date": "04/15/2022 04:51:35", "content": "

lr: it depends on your model. if you find it is too high for your model, you can try to set a lower one. I tried 1e-4 with effnetb4, and got more than 0.9 train_acc and val_acc, but only 0.72 test_acc. With lr warm up and cosine annealing, the test_acc can reach about 0.8.
\nimage_size=512
\nbatch_size=16

", "votes": null, "replies": [] }, { "id": 1790918, "author_name": "aayushdeswal", "author_url": "", "post_date": "05/15/2022 13:13:36", "content": "

Hi @mersotc . could you pls share the link to your notebook. I too got train and val accuracy >90% but not able to get test accuracy more than 0.5

", "votes": null, "replies": [] } ] }, { "id": 1759282, "author_name": "ngyzly", "author_url": "", "post_date": "04/18/2022 13:54:43", "content": "

What does the score mean on the leaderboard? I used eff v2 large with a few tricks, 99.5% ACC on Val, but got only 0.845 on the test set.

", "votes": null, "replies": [ { "id": 1760883, "author_name": "mithilsalunkhe", "author_url": "", "post_date": "04/19/2022 15:09:39", "content": "

Overfitting probably ; plus are you using kfolds or simple train val split ?

", "votes": null, "replies": [] } ] }, { "id": 1783337, "author_name": "matteob88", "author_url": "", "post_date": "05/10/2022 10:01:14", "content": "

I have the same problem you described here. My val_acc is around 85% but on the test set I get very poor results. I'm trying to figure out what I'm doing wrong during prediction

", "votes": null, "replies": [ { "id": 1783551, "author_name": "laurentpoyet", "author_url": "", "post_date": "05/10/2022 13:23:29", "content": "

I was not associating the correct labels to my predictions, and was obtaining around 0.05 score. So first step is to be sure that your predictions are associated to the correct label.
\nI updated my notebook accordingly to solve this problem using flow_from_dataframe() method with Tensorflow

", "votes": null, "replies": [] } ] }, { "id": 1784674, "author_name": "aroonima12", "author_url": "", "post_date": "05/11/2022 11:20:45", "content": "

Hi, is the ImageDataGenerator and flow_from_dataframe method working for you?
\nI have tried this method but I getting very low initial accuracy in training and validation equal to .01 in the 1st epoch. I am not able to figure out the problem here.

", "votes": null, "replies": [] }, { "id": 1790914, "author_name": "aayushdeswal", "author_url": "", "post_date": "05/15/2022 13:10:53", "content": "

Hi @laurentpoyet . Pls, share the link to your notebook. I too got accuracy above 90% on the train and Val data. Initially, i was too getting 0.01 on test data but then I indexed labels alphabetically and specified class_names in flow_from_dataframe. Since then I'm getting max of 50% accuracy on test data. Since I'm new to this, I have few doubts. It'll be really helpful if u help me with this:
\n1) What will be the major changes in code for FGVC and normal Image classification?
\n2) Why are we training the whole model instead of converging on top layers and then fine-tuning?
\n3) Is the high bias or overfitting due to training all layers of model to the dataset?

", "votes": null, "replies": [ { "id": 1791913, "author_name": "laurentpoyet", "author_url": "", "post_date": "05/16/2022 13:24:24", "content": "

Hi,

\n

You can find the notebook here :

\n

www.kaggle.com/code/laurentpoyet/sorghum-100-effnetb1
\n

\n

1) it is a normal image classification problem
\n2) There is no need to train the model completely, fine tuning would be to train some layers, then freeze these layers and train only last layers, which I didn't do in this notebook. However I didn't achieve better results doing this.
\n3) Personally I didn't experience overfitting in this case, neither did I reach better results by reducing the trainable layers in the model.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1744884": "Hello to everyone,\n\nWhat a nice topic, predicting plant species over this huge dataset of images.\nIt is a competition, but as there is no points or money involved, I hope I can find a little bit of help.\n\nI started with confidence, but after a few attempts, I am not sure why my model prediction accuracy remains so low when the training and validation accuracy are over 80% ...\n\nI would like to use ImageDataGenerator and flow_from_dataframe to get the simplest pipeline possible with one the fly data augmentation.\n\nI wish you could guide me with some comments or advice, so I shared the code for my preprocessing, training and prediction here :\n\n[Kaggle Notebook](url[https://www.kaggle.com/code/laurentpoyet/sorghum-100-preprocessing-and-training])\n\nLooking forward for your feedback. \n\nEdit : Found the problem, it was in the treatment of the prediction classes, the notebook is now achieving 57% accuracy. Don't hesitate if you have other comments or questions.", "1747058": "So do I. 😭", "1747138": "Thanks for sharing your code also, I can't achieve more than 0.05 accuracy using ImageDataGenerator and flow_from_dataframe method ...\n\nWith other methods the model performs quite good however ...", "1754363": "You can try to use a higher learning rate to help the train_loss to drop faster(maybe... below 2.0 in the first 5 epochs), and a smaller batch size to help your model perform better on the test dataset.", "1754474": "Thanks for your advice, I reduced the batch size and will set the initial LR at 0.1 with reduceLRonplateau. Just waiting to recover my GPU to commit the notebook. What image size do you use for optimal results ?\n\nEdit : From first trails, accuracy remains low with LR higher than 0,01 with effnetB1 and batchsize 16.", "1755888": "lr: it depends on your model. if you find it is too high for your model, you can try to set a lower one. I tried 1e-4 with effnetb4, and got more than 0.9 train_acc and val_acc, but only 0.72 test_acc. With lr warm up and cosine annealing, the test_acc can reach about 0.8.\nimage_size=512\nbatch_size=16", "1759282": "What does the score mean on the leaderboard? I used eff v2 large with a few tricks, 99.5% ACC on Val, but got only 0.845 on the test set.", "1760883": "Overfitting probably ; plus are you using kfolds or simple train val split ?", "1783337": "I have the same problem you described here. My val_acc is around 85% but on the test set I get very poor results. I'm trying to figure out what I'm doing wrong during prediction", "1783551": "I was not associating the correct labels to my predictions, and was obtaining around 0.05 score. So first step is to be sure that your predictions are associated to the correct label.\nI updated my notebook accordingly to solve this problem using flow_from_dataframe() method with Tensorflow", "1784674": "Hi, is the ImageDataGenerator and flow_from_dataframe method working for you?\nI have tried this method but I getting very low initial accuracy in training and validation equal to .01 in the 1st epoch. I am not able to figure out the problem here.", "1790914": "Hi @laurentpoyet . Pls, share the link to your notebook. I too got accuracy above 90% on the train and Val data. Initially, i was too getting 0.01 on test data but then I indexed labels alphabetically and specified class_names in flow_from_dataframe. Since then I'm getting max of 50% accuracy on test data. Since I'm new to this, I have few doubts. It'll be really helpful if u help me with this:\n1) What will be the major changes in code for FGVC and normal Image classification?\n2) Why are we training the whole model instead of converging on top layers and then fine-tuning?\n3) Is the high bias or overfitting due to training all layers of model to the dataset?", "1790918": "Hi @mersotc . could you pls share the link to your notebook. I too got train and val accuracy >90% but not able to get test accuracy more than 0.5", "1791913": "Hi,\n\nYou can find the notebook here :\n\n[www.kaggle.com/code/laurentpoyet/sorghum-100-effnetb1\n](url)\n\n1) it is a normal image classification problem\n2) There is no need to train the model completely, fine tuning would be to train some layers, then freeze these layers and train only last layers, which I didn't do in this notebook. However I didn't achieve better results doing this.\n3) Personally I didn't experience overfitting in this case, neither did I reach better results by reducing the trainable layers in the model." }, "source": "meta" }