I have used several transfer learning models and got different accuracies for each model on validation set while running on my local machine. But when I upload the models into my notebook and submit for score, I am getting 0.139 for every model I submit. Is there something I need to handle? It looks like a lot of people on leaderboard got 0.139 accuracy. There must be some mistake we are all committing.
", "messages": [ { "id": "1159006", "postDate": "01/19/2021 01:01:23", "content": "I have used several transfer learning models and got different accuracies for each model on validation set while running on my local machine. But when I upload the models into my notebook and submit for score, I am getting 0.139 for every model I submit. Is there something I need to handle? It looks like a lot of people on leaderboard got 0.139 accuracy. There must be some mistake we are all committing.
", "rawMarkdown": "I have used several transfer learning models and got different accuracies for each model on validation set while running on my local machine. But when I upload the models into my notebook and submit for score, I am getting 0.139 for every model I submit. Is there something I need to handle? It looks like a lot of people on leaderboard got 0.139 accuracy. There must be some mistake we are all committing.", "votes": null }, { "id": "1159123", "postDate": "01/19/2021 04:04:06", "content": "You are not probable rescaling your data in the training pipeline or inference pipeline
", "rawMarkdown": "You are not probable rescaling your data in the training pipeline or inference pipeline", "votes": null }, { "id": "1159170", "postDate": "01/19/2021 04:48:18", "content": "I got 0.139 on leaderboard, in beginning of the competition.
\nBecause of an error, a cell(model inferencing) couldn't run, and skipping this cell the final submission file generation code was ran.
\nTry to debug your inferencing notebook, read the logs.
A form of leader board probing is to create script that always selects one of the labels. Your score than tells you the percentage of that class in the test images that are being scored.
\nI believe that a number of posts exist that suggest that a .048 score is the result of always have 0 as the class - so 4.8% of the test images that are scored are this class. I am one of those that had a model that turned out to always predict class 0 due to an error in my script, so I believe those posts to be accurate. I have also had errors that always predicted a class 4 - but never submitted any of those models to the LB to see what percentage of the scored test images are 4.
\n0.106, 0.139 and 0.602 are also scores on the leader board that have a lot of folks. Based on the train distribution, the 0.602 are likely submissions that always predicted class 3.
\nSo my assumption would be that your model is always predicting the same class value.
\nRun your script using \"Run All\" in the kernel and evaluate images in the train set rather than the test. Look at your submission - very strong probability that your predicting the same value all the time.
", "rawMarkdown": "A form of leader board probing is to create script that always selects one of the labels. Your score than tells you the percentage of that class in the test images that are being scored. \n\nI believe that a number of posts exist that suggest that a .048 score is the result of always have 0 as the class - so 4.8% of the test images that are scored are this class. I am one of those that had a model that turned out to always predict class 0 due to an error in my script, so I believe those posts to be accurate. I have also had errors that always predicted a class 4 - but never submitted any of those models to the LB to see what percentage of the scored test images are 4.\n\n0.106, 0.139 and 0.602 are also scores on the leader board that have a lot of folks. Based on the train distribution, the 0.602 are likely submissions that always predicted class 3.\n\nSo my assumption would be that your model is always predicting the same class value.\n\nRun your script using \"Run All\" in the kernel and evaluate images in the train set rather than the test. Look at your submission - very strong probability that your predicting the same value all the time.", "votes": null }, { "id": "1159198", "postDate": "01/19/2021 05:13:57", "content": "https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/202943
\nDid a quick search of the posts - the above is post where someone has reported the distribution he believes exists in the scored data.
\nIf that post is correct than 0.139 score is not the result of the same class always being predicted - so sorry - you got some other error in your script!
", "rawMarkdown": "https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/202943\n\nDid a quick search of the posts - the above is post where someone has reported the distribution he believes exists in the scored data.\n\nIf that post is correct than 0.139 score is not the result of the same class always being predicted - so sorry - you got some other error in your script!", "votes": null }, { "id": "1159974", "postDate": "01/19/2021 15:27:57", "content": "No, you might be correct in your previous reply. Data distribution in test data will have 13.9% of data for one label. But my model is working fine with validation data.
\nI have developed and trained the model on my local system and saved to .h5 file. Then I loaded the model into kaggle notebook and submitted for predictions. Does it affect my submission in anyway?
", "rawMarkdown": "No, you might be correct in your previous reply. Data distribution in test data will have 13.9% of data for one label. But my model is working fine with validation data.\n\nI have developed and trained the model on my local system and saved to .h5 file. Then I loaded the model into kaggle notebook and submitted for predictions. Does it affect my submission in anyway?", "votes": null }, { "id": "1160047", "postDate": "01/19/2021 16:30:35", "content": "When you say that your model is working fine with validation data - does that mean you have code that is a duplicate of your test submission code on your local system that uses some of the train images and the accuracy is similar to what is seen during training? I created a small \"test\" images folder on my local machine with 1000 of the train images - did not really care if they were \"validation\" or not - I just wanted a simple \"does this make sense\" verification that my submission code is correct. Did this of course after the horse had left the barn and I had made my 0.048 submission.
\nI load all my local models into a data set and bring them into a Kaggle kernel as you mentioned. At times in past competitions I did have some issues when my local machine was running a much older version of one or more key libraries. Aside from version issues I don't recall any other errors related to local saved models.
", "rawMarkdown": "When you say that your model is working fine with validation data - does that mean you have code that is a duplicate of your test submission code on your local system that uses some of the train images and the accuracy is similar to what is seen during training? I created a small \"test\" images folder on my local machine with 1000 of the train images - did not really care if they were \"validation\" or not - I just wanted a simple \"does this make sense\" verification that my submission code is correct. Did this of course after the horse had left the barn and I had made my 0.048 submission.\n\nI load all my local models into a data set and bring them into a Kaggle kernel as you mentioned. At times in past competitions I did have some issues when my local machine was running a much older version of one or more key libraries. Aside from version issues I don't recall any other errors related to local saved models.", "votes": null }, { "id": "1160172", "postDate": "01/19/2021 17:59:10", "content": "I actually downloaded whole data into my local machine, and then I developed my code in my loacal machine only. I split the train data into 80:20(train:validattion) ratio and trained my model in loacal machine only. I got 84% training accuracy and 78% test accuracy. I saved this model, uploaded that model to kaggle notebook and I submitted the model.
\nMy doubt is, how does submission system know the input shape of my model? And how does it know the output shape of prediction? Am I missing something here?
", "rawMarkdown": "I actually downloaded whole data into my local machine, and then I developed my code in my loacal machine only. I split the train data into 80:20(train:validattion) ratio and trained my model in loacal machine only. I got 84% training accuracy and 78% test accuracy. I saved this model, uploaded that model to kaggle notebook and I submitted the model. \n\nMy doubt is, how does submission system know the input shape of my model? And how does it know the output shape of prediction? Am I missing something here?", "votes": null }, { "id": "1160474", "postDate": "01/20/2021 00:09:37", "content": "Shapes are all contained in the saved h5 file.
\nWhen you run the kaggle kernel with the model did it successfully classify the single public test image and did every cell in the script run without errors? My very low scoring submissions all have at least one cell that has error. Does the Log view of the saved kernel show any error messages?
", "rawMarkdown": "Shapes are all contained in the saved h5 file.\n\nWhen you run the kaggle kernel with the model did it successfully classify the single public test image and did every cell in the script run without errors? My very low scoring submissions all have at least one cell that has error. Does the Log view of the saved kernel show any error messages?", "votes": null }, { "id": "1161590", "postDate": "01/20/2021 16:27:46", "content": "I did not see any errors in logs. My model predicted public test image with out any issues. below is my notebook. I have shared it with public. Will you be able to have a look at it?
\nhttps://www.kaggle.com/venkat2ram/cassava-efficientnet/log
", "rawMarkdown": "I did not see any errors in logs. My model predicted public test image with out any issues. below is my notebook. I have shared it with public. Will you be able to have a look at it?\n\nhttps://www.kaggle.com/venkat2ram/cassava-efficientnet/log", "votes": null }, { "id": "1161621", "postDate": "01/20/2021 16:43:27", "content": "Tried to fork and run the notebook. Can't do that as you have at least one private file in your input.
\nWill download it later today to my local machine using one of my saved models and run - but at first glance I don't see how your looping thru all the images in the hidden test set. Change the test image location to the train images and \"Run All\" - does your code generate a submission file with all of the train images?
\nIt will be a few hours before I get to try this step on my local machine.
\nChange this cell to look at the train rather than the test images.
\nimg=tf.keras.preprocessing.image.load_img('/kaggle/input/cassava-leaf-disease-classification/test_images/'+img_id[0])
You also need to change this cell inorder to use the train images. Use the train.csv instead. However, even with some errors a submission still gets written so this is not a fool proof way of validating the code.
\nsubm=pd.read_csv('/kaggle/input/cassava-leaf-disease-classification/sample_submission.csv')
It looks to me that your script only evaluates the first image and does not loop/predict the remaining images in the 15K test set. Not sure how you end up with the 0.139 score - I was using a model from a saved data set of mine.
\nI could not run this cell on my keras model.
\nmodel.add(tf.keras.layers.Lambda(lambda x:tf.math.argmax(x,axis=1)))
The pred dataframe only has a single prediction after running the script.
\nHow about making your dataset with the model public also. You will probably need to remove it and than add it back for the public status to take effect. Than I and others can look using your model - I may be getting errors using my keras model that distorts what I see happening.
", "rawMarkdown": "It looks to me that your script only evaluates the first image and does not loop/predict the remaining images in the 15K test set. Not sure how you end up with the 0.139 score - I was using a model from a saved data set of mine.\n\nI could not run this cell on my keras model.\n`model.add(tf.keras.layers.Lambda(lambda x:tf.math.argmax(x,axis=1)))`\n\nThe pred dataframe only has a single prediction after running the script.\n\nHow about making your dataset with the model public also. You will probably need to remove it and than add it back for the public status to take effect. Than I and others can look using your model - I may be getting errors using my keras model that distorts what I see happening.", "votes": null }, { "id": "1161878", "postDate": "01/20/2021 19:37:56", "content": "I guess I understand the problem now. I have seen 'for' loop used in several other notebooks and did not understand why they used for loop to process single public test set image. Will try to use for loop and let you know. Thanks for your time and effort.
", "rawMarkdown": "I guess I understand the problem now. I have seen 'for' loop used in several other notebooks and did not understand why they used for loop to process single public test set image. Will try to use for loop and let you know. Thanks for your time and effort.", "votes": null }, { "id": "1161899", "postDate": "01/20/2021 20:07:36", "content": "Yes - IMO the host/kaggle did a huge dis-service to us by only including a single image in the public test set when the private set had 15K worth of images. A single image caused lots of confusion - many posts asking about this and it does not let you easily confirm that your code will correctly loop. If they had given us a hundred or so than most methods for feeding images into the prediction would have had a full batch, and those methods and those using a simple loop could have confirmed both timing and accuracy issues.
\nGood luck - your more than welcome. Glad I might have been of some help.
", "rawMarkdown": "Yes - IMO the host/kaggle did a huge dis-service to us by only including a single image in the public test set when the private set had 15K worth of images. A single image caused lots of confusion - many posts asking about this and it does not let you easily confirm that your code will correctly loop. If they had given us a hundred or so than most methods for feeding images into the prediction would have had a full batch, and those methods and those using a simple loop could have confirmed both timing and accuracy issues.\n\nGood luck - your more than welcome. Glad I might have been of some help.", "votes": null }, { "id": "1163891", "postDate": "01/22/2021 03:03:31", "content": "Hello,do you find out the root cause?I have the same problem as you,I got 0.8+ accuracy on my training set and validation set,but only 0.083 or 0.066 accuracy for submission.
", "rawMarkdown": "Hello,do you find out the root cause?I have the same problem as you,I got 0.8+ accuracy on my training set and validation set,but only 0.083 or 0.066 accuracy for submission.", "votes": null }, { "id": "1163932", "postDate": "01/22/2021 04:15:06", "content": "Image name does not match prediction category
", "rawMarkdown": "Image name does not match prediction category", "votes": null }, { "id": "1164691", "postDate": "01/22/2021 14:20:01", "content": "Hi Carl, My submission was taking only few minutes when I got 0.139 accuracy. It clearly shows my code was not processing all test images. After adding for loop in my code, my submission is taking 3 hours to complete. But the accuracy I am getting is 0.155. I need to look into it more. I predicted few batches of images on my validation set, It's giving more than 80% accuracy on validation set.
", "rawMarkdown": "Hi Carl, My submission was taking only few minutes when I got 0.139 accuracy. It clearly shows my code was not processing all test images. After adding for loop in my code, my submission is taking 3 hours to complete. But the accuracy I am getting is 0.155. I need to look into it more. I predicted few batches of images on my validation set, It's giving more than 80% accuracy on validation set.", "votes": null }, { "id": "1165024", "postDate": "01/22/2021 16:47:43", "content": "save and download your trained model to local machine ,and then add the model to input directory in kaggle notebook
\nafter that load your model from there
Hi Chanukya, I have trained my model on my local machine. Then exported the trained model to kaggle notebook and submitted. Below is my notebook link. Can you check once?
\nhttps://www.kaggle.com/venkat2ram/cassava-efficientnet
", "rawMarkdown": "Hi Chanukya, I have trained my model on my local machine. Then exported the trained model to kaggle notebook and submitted. Below is my notebook link. Can you check once?\n\nhttps://www.kaggle.com/venkat2ram/cassava-efficientnet", "votes": null }, { "id": "1165709", "postDate": "01/23/2021 07:21:27", "content": "Very interesting that several days after offering some advice on this post I also had a submission that scored 0.139.
\nMy error was pretty easy to root cause. I added the most recent new model to my data set. Loaded the model but than used the wrong model name to make the prediction. Obvious errors in log that pointed to this mistake.
", "rawMarkdown": "Very interesting that several days after offering some advice on this post I also had a submission that scored 0.139.\n\nMy error was pretty easy to root cause. I added the most recent new model to my data set. Loaded the model but than used the wrong model name to make the prediction. Obvious errors in log that pointed to this mistake.", "votes": null }, { "id": "1166256", "postDate": "01/23/2021 13:27:53", "content": "My issue resolved when I added for loop. But got 0.15 accuracy. Then I identified that I have rescaled the training set and validation set, but did not rescale the test set. So I applied rescaling and submitted it. Now I got 0.845. It's below average accuracy.. But I am happy that I learnt few things here.
", "rawMarkdown": "My issue resolved when I added for loop. But got 0.15 accuracy. Then I identified that I have rescaled the training set and validation set, but did not rescale the test set. So I applied rescaling and submitted it. Now I got 0.845. It's below average accuracy.. But I am happy that I learnt few things here.", "votes": null }, { "id": "1166266", "postDate": "01/23/2021 13:36:14", "content": "Hi All, Thanks for your answers. There are two causes why I got 0.139 as accuracy.
\n1) I have not used for loop to predict images from sample submission. Once I added for loop, my submission actually took 3 hours to complete. Then I knew that my submission is actually getting evaluated with large test data. And I got 0.15 accuracy.
\n2) I identified another cause. i.e) I have rescaled my train set and validation set while training, but forgot to rescale my input while working on sample_submission.csv. Obviously my model weights are calculated for rescaled input but I am trying to predict with input which is not rescaled. To resolve this issue, I applied rescaling to my submission input.
Finally I got 0.845 accuracy. This is with MobileNetV2. I am going to try few other models and see. Good luck.
", "rawMarkdown": "Hi All, Thanks for your answers. There are two causes why I got 0.139 as accuracy.\n\n1) I have not used for loop to predict images from sample submission. Once I added for loop, my submission actually took 3 hours to complete. Then I knew that my submission is actually getting evaluated with large test data. And I got 0.15 accuracy.\n2) I identified another cause. i.e) I have rescaled my train set and validation set while training, but forgot to rescale my input while working on sample_submission.csv. Obviously my model weights are calculated for rescaled input but I am trying to predict with input which is not rescaled. To resolve this issue, I applied rescaling to my submission input. \n\nFinally I got 0.845 accuracy. This is with MobileNetV2. I am going to try few other models and see. Good luck.", "votes": null }, { "id": "1166543", "postDate": "01/23/2021 17:11:42", "content": "Glad to hear - good luck
", "rawMarkdown": "Glad to hear - good luck", "votes": null } ], "comments": [ { "id": 1159123, "author_name": "mithilsalunkhe", "author_url": "", "post_date": "01/19/2021 04:04:06", "content": "You are not probable rescaling your data in the training pipeline or inference pipeline
", "votes": null, "replies": [] }, { "id": 1159170, "author_name": "jabertuhin", "author_url": "", "post_date": "01/19/2021 04:48:18", "content": "I got 0.139 on leaderboard, in beginning of the competition.
\nBecause of an error, a cell(model inferencing) couldn't run, and skipping this cell the final submission file generation code was ran.
\nTry to debug your inferencing notebook, read the logs.
A form of leader board probing is to create script that always selects one of the labels. Your score than tells you the percentage of that class in the test images that are being scored.
\nI believe that a number of posts exist that suggest that a .048 score is the result of always have 0 as the class - so 4.8% of the test images that are scored are this class. I am one of those that had a model that turned out to always predict class 0 due to an error in my script, so I believe those posts to be accurate. I have also had errors that always predicted a class 4 - but never submitted any of those models to the LB to see what percentage of the scored test images are 4.
\n0.106, 0.139 and 0.602 are also scores on the leader board that have a lot of folks. Based on the train distribution, the 0.602 are likely submissions that always predicted class 3.
\nSo my assumption would be that your model is always predicting the same class value.
\nRun your script using \"Run All\" in the kernel and evaluate images in the train set rather than the test. Look at your submission - very strong probability that your predicting the same value all the time.
", "votes": null, "replies": [ { "id": 1159974, "author_name": "venkat2ram", "author_url": "", "post_date": "01/19/2021 15:27:57", "content": "No, you might be correct in your previous reply. Data distribution in test data will have 13.9% of data for one label. But my model is working fine with validation data.
\nI have developed and trained the model on my local system and saved to .h5 file. Then I loaded the model into kaggle notebook and submitted for predictions. Does it affect my submission in anyway?
", "votes": null, "replies": [] }, { "id": 1160047, "author_name": "pcjimmmy", "author_url": "", "post_date": "01/19/2021 16:30:35", "content": "When you say that your model is working fine with validation data - does that mean you have code that is a duplicate of your test submission code on your local system that uses some of the train images and the accuracy is similar to what is seen during training? I created a small \"test\" images folder on my local machine with 1000 of the train images - did not really care if they were \"validation\" or not - I just wanted a simple \"does this make sense\" verification that my submission code is correct. Did this of course after the horse had left the barn and I had made my 0.048 submission.
\nI load all my local models into a data set and bring them into a Kaggle kernel as you mentioned. At times in past competitions I did have some issues when my local machine was running a much older version of one or more key libraries. Aside from version issues I don't recall any other errors related to local saved models.
", "votes": null, "replies": [] }, { "id": 1160172, "author_name": "venkat2ram", "author_url": "", "post_date": "01/19/2021 17:59:10", "content": "I actually downloaded whole data into my local machine, and then I developed my code in my loacal machine only. I split the train data into 80:20(train:validattion) ratio and trained my model in loacal machine only. I got 84% training accuracy and 78% test accuracy. I saved this model, uploaded that model to kaggle notebook and I submitted the model.
\nMy doubt is, how does submission system know the input shape of my model? And how does it know the output shape of prediction? Am I missing something here?
", "votes": null, "replies": [] }, { "id": 1160474, "author_name": "pcjimmmy", "author_url": "", "post_date": "01/20/2021 00:09:37", "content": "Shapes are all contained in the saved h5 file.
\nWhen you run the kaggle kernel with the model did it successfully classify the single public test image and did every cell in the script run without errors? My very low scoring submissions all have at least one cell that has error. Does the Log view of the saved kernel show any error messages?
", "votes": null, "replies": [] }, { "id": 1161590, "author_name": "venkat2ram", "author_url": "", "post_date": "01/20/2021 16:27:46", "content": "I did not see any errors in logs. My model predicted public test image with out any issues. below is my notebook. I have shared it with public. Will you be able to have a look at it?
\nhttps://www.kaggle.com/venkat2ram/cassava-efficientnet/log
", "votes": null, "replies": [] }, { "id": 1161621, "author_name": "pcjimmmy", "author_url": "", "post_date": "01/20/2021 16:43:27", "content": "Tried to fork and run the notebook. Can't do that as you have at least one private file in your input.
\nWill download it later today to my local machine using one of my saved models and run - but at first glance I don't see how your looping thru all the images in the hidden test set. Change the test image location to the train images and \"Run All\" - does your code generate a submission file with all of the train images?
\nIt will be a few hours before I get to try this step on my local machine.
\nChange this cell to look at the train rather than the test images.
\nimg=tf.keras.preprocessing.image.load_img('/kaggle/input/cassava-leaf-disease-classification/test_images/'+img_id[0])
You also need to change this cell inorder to use the train images. Use the train.csv instead. However, even with some errors a submission still gets written so this is not a fool proof way of validating the code.
\nsubm=pd.read_csv('/kaggle/input/cassava-leaf-disease-classification/sample_submission.csv')
It looks to me that your script only evaluates the first image and does not loop/predict the remaining images in the 15K test set. Not sure how you end up with the 0.139 score - I was using a model from a saved data set of mine.
\nI could not run this cell on my keras model.
\nmodel.add(tf.keras.layers.Lambda(lambda x:tf.math.argmax(x,axis=1)))
The pred dataframe only has a single prediction after running the script.
\nHow about making your dataset with the model public also. You will probably need to remove it and than add it back for the public status to take effect. Than I and others can look using your model - I may be getting errors using my keras model that distorts what I see happening.
", "votes": null, "replies": [] }, { "id": 1161878, "author_name": "venkat2ram", "author_url": "", "post_date": "01/20/2021 19:37:56", "content": "I guess I understand the problem now. I have seen 'for' loop used in several other notebooks and did not understand why they used for loop to process single public test set image. Will try to use for loop and let you know. Thanks for your time and effort.
", "votes": null, "replies": [] }, { "id": 1161899, "author_name": "pcjimmmy", "author_url": "", "post_date": "01/20/2021 20:07:36", "content": "Yes - IMO the host/kaggle did a huge dis-service to us by only including a single image in the public test set when the private set had 15K worth of images. A single image caused lots of confusion - many posts asking about this and it does not let you easily confirm that your code will correctly loop. If they had given us a hundred or so than most methods for feeding images into the prediction would have had a full batch, and those methods and those using a simple loop could have confirmed both timing and accuracy issues.
\nGood luck - your more than welcome. Glad I might have been of some help.
", "votes": null, "replies": [] } ] }, { "id": 1159198, "author_name": "pcjimmmy", "author_url": "", "post_date": "01/19/2021 05:13:57", "content": "https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/202943
\nDid a quick search of the posts - the above is post where someone has reported the distribution he believes exists in the scored data.
\nIf that post is correct than 0.139 score is not the result of the same class always being predicted - so sorry - you got some other error in your script!
", "votes": null, "replies": [] }, { "id": 1163891, "author_name": "carlwuuu", "author_url": "", "post_date": "01/22/2021 03:03:31", "content": "Hello,do you find out the root cause?I have the same problem as you,I got 0.8+ accuracy on my training set and validation set,but only 0.083 or 0.066 accuracy for submission.
", "votes": null, "replies": [ { "id": 1164691, "author_name": "venkat2ram", "author_url": "", "post_date": "01/22/2021 14:20:01", "content": "Hi Carl, My submission was taking only few minutes when I got 0.139 accuracy. It clearly shows my code was not processing all test images. After adding for loop in my code, my submission is taking 3 hours to complete. But the accuracy I am getting is 0.155. I need to look into it more. I predicted few batches of images on my validation set, It's giving more than 80% accuracy on validation set.
", "votes": null, "replies": [] } ] }, { "id": 1163932, "author_name": "zzzhangzz", "author_url": "", "post_date": "01/22/2021 04:15:06", "content": "Image name does not match prediction category
", "votes": null, "replies": [] }, { "id": 1165024, "author_name": "chanu0109", "author_url": "", "post_date": "01/22/2021 16:47:43", "content": "save and download your trained model to local machine ,and then add the model to input directory in kaggle notebook
\nafter that load your model from there
Hi Chanukya, I have trained my model on my local machine. Then exported the trained model to kaggle notebook and submitted. Below is my notebook link. Can you check once?
\nhttps://www.kaggle.com/venkat2ram/cassava-efficientnet
", "votes": null, "replies": [] } ] }, { "id": 1165709, "author_name": "pcjimmmy", "author_url": "", "post_date": "01/23/2021 07:21:27", "content": "Very interesting that several days after offering some advice on this post I also had a submission that scored 0.139.
\nMy error was pretty easy to root cause. I added the most recent new model to my data set. Loaded the model but than used the wrong model name to make the prediction. Obvious errors in log that pointed to this mistake.
", "votes": null, "replies": [ { "id": 1166256, "author_name": "venkat2ram", "author_url": "", "post_date": "01/23/2021 13:27:53", "content": "My issue resolved when I added for loop. But got 0.15 accuracy. Then I identified that I have rescaled the training set and validation set, but did not rescale the test set. So I applied rescaling and submitted it. Now I got 0.845. It's below average accuracy.. But I am happy that I learnt few things here.
", "votes": null, "replies": [] }, { "id": 1166543, "author_name": "pcjimmmy", "author_url": "", "post_date": "01/23/2021 17:11:42", "content": "Glad to hear - good luck
", "votes": null, "replies": [] } ] }, { "id": 1166266, "author_name": "venkat2ram", "author_url": "", "post_date": "01/23/2021 13:36:14", "content": "Hi All, Thanks for your answers. There are two causes why I got 0.139 as accuracy.
\n1) I have not used for loop to predict images from sample submission. Once I added for loop, my submission actually took 3 hours to complete. Then I knew that my submission is actually getting evaluated with large test data. And I got 0.15 accuracy.
\n2) I identified another cause. i.e) I have rescaled my train set and validation set while training, but forgot to rescale my input while working on sample_submission.csv. Obviously my model weights are calculated for rescaled input but I am trying to predict with input which is not rescaled. To resolve this issue, I applied rescaling to my submission input.
Finally I got 0.845 accuracy. This is with MobileNetV2. I am going to try few other models and see. Good luck.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1159006": "I have used several transfer learning models and got different accuracies for each model on validation set while running on my local machine. But when I upload the models into my notebook and submit for score, I am getting 0.139 for every model I submit. Is there something I need to handle? It looks like a lot of people on leaderboard got 0.139 accuracy. There must be some mistake we are all committing.", "1159123": "You are not probable rescaling your data in the training pipeline or inference pipeline", "1159170": "I got 0.139 on leaderboard, in beginning of the competition.\nBecause of an error, a cell(model inferencing) couldn't run, and skipping this cell the final submission file generation code was ran.\nTry to debug your inferencing notebook, read the logs.", "1159188": "A form of leader board probing is to create script that always selects one of the labels. Your score than tells you the percentage of that class in the test images that are being scored. \n\nI believe that a number of posts exist that suggest that a .048 score is the result of always have 0 as the class - so 4.8% of the test images that are scored are this class. I am one of those that had a model that turned out to always predict class 0 due to an error in my script, so I believe those posts to be accurate. I have also had errors that always predicted a class 4 - but never submitted any of those models to the LB to see what percentage of the scored test images are 4.\n\n0.106, 0.139 and 0.602 are also scores on the leader board that have a lot of folks. Based on the train distribution, the 0.602 are likely submissions that always predicted class 3.\n\nSo my assumption would be that your model is always predicting the same class value.\n\nRun your script using \"Run All\" in the kernel and evaluate images in the train set rather than the test. Look at your submission - very strong probability that your predicting the same value all the time.", "1159198": "https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/202943\n\nDid a quick search of the posts - the above is post where someone has reported the distribution he believes exists in the scored data.\n\nIf that post is correct than 0.139 score is not the result of the same class always being predicted - so sorry - you got some other error in your script!", "1159974": "No, you might be correct in your previous reply. Data distribution in test data will have 13.9% of data for one label. But my model is working fine with validation data.\n\nI have developed and trained the model on my local system and saved to .h5 file. Then I loaded the model into kaggle notebook and submitted for predictions. Does it affect my submission in anyway?", "1160047": "When you say that your model is working fine with validation data - does that mean you have code that is a duplicate of your test submission code on your local system that uses some of the train images and the accuracy is similar to what is seen during training? I created a small \"test\" images folder on my local machine with 1000 of the train images - did not really care if they were \"validation\" or not - I just wanted a simple \"does this make sense\" verification that my submission code is correct. Did this of course after the horse had left the barn and I had made my 0.048 submission.\n\nI load all my local models into a data set and bring them into a Kaggle kernel as you mentioned. At times in past competitions I did have some issues when my local machine was running a much older version of one or more key libraries. Aside from version issues I don't recall any other errors related to local saved models.", "1160172": "I actually downloaded whole data into my local machine, and then I developed my code in my loacal machine only. I split the train data into 80:20(train:validattion) ratio and trained my model in loacal machine only. I got 84% training accuracy and 78% test accuracy. I saved this model, uploaded that model to kaggle notebook and I submitted the model. \n\nMy doubt is, how does submission system know the input shape of my model? And how does it know the output shape of prediction? Am I missing something here?", "1160474": "Shapes are all contained in the saved h5 file.\n\nWhen you run the kaggle kernel with the model did it successfully classify the single public test image and did every cell in the script run without errors? My very low scoring submissions all have at least one cell that has error. Does the Log view of the saved kernel show any error messages?", "1161590": "I did not see any errors in logs. My model predicted public test image with out any issues. below is my notebook. I have shared it with public. Will you be able to have a look at it?\n\nhttps://www.kaggle.com/venkat2ram/cassava-efficientnet/log", "1161621": "Tried to fork and run the notebook. Can't do that as you have at least one private file in your input.\n\nWill download it later today to my local machine using one of my saved models and run - but at first glance I don't see how your looping thru all the images in the hidden test set. Change the test image location to the train images and \"Run All\" - does your code generate a submission file with all of the train images?\n\nIt will be a few hours before I get to try this step on my local machine. \n\nChange this cell to look at the train rather than the test images.\n\n`img=tf.keras.preprocessing.image.load_img('/kaggle/input/cassava-leaf-disease-classification/test_images/'+img_id[0])`", "1161843": "You also need to change this cell inorder to use the train images. Use the train.csv instead. However, even with some errors a submission still gets written so this is not a fool proof way of validating the code.\n\ns`ubm=pd.read_csv('/kaggle/input/cassava-leaf-disease-classification/sample_submission.csv')`", "1161873": "It looks to me that your script only evaluates the first image and does not loop/predict the remaining images in the 15K test set. Not sure how you end up with the 0.139 score - I was using a model from a saved data set of mine.\n\nI could not run this cell on my keras model.\n`model.add(tf.keras.layers.Lambda(lambda x:tf.math.argmax(x,axis=1)))`\n\nThe pred dataframe only has a single prediction after running the script.\n\nHow about making your dataset with the model public also. You will probably need to remove it and than add it back for the public status to take effect. Than I and others can look using your model - I may be getting errors using my keras model that distorts what I see happening.", "1161878": "I guess I understand the problem now. I have seen 'for' loop used in several other notebooks and did not understand why they used for loop to process single public test set image. Will try to use for loop and let you know. Thanks for your time and effort.", "1161899": "Yes - IMO the host/kaggle did a huge dis-service to us by only including a single image in the public test set when the private set had 15K worth of images. A single image caused lots of confusion - many posts asking about this and it does not let you easily confirm that your code will correctly loop. If they had given us a hundred or so than most methods for feeding images into the prediction would have had a full batch, and those methods and those using a simple loop could have confirmed both timing and accuracy issues.\n\nGood luck - your more than welcome. Glad I might have been of some help.", "1163891": "Hello,do you find out the root cause?I have the same problem as you,I got 0.8+ accuracy on my training set and validation set,but only 0.083 or 0.066 accuracy for submission.", "1163932": "Image name does not match prediction category", "1164691": "Hi Carl, My submission was taking only few minutes when I got 0.139 accuracy. It clearly shows my code was not processing all test images. After adding for loop in my code, my submission is taking 3 hours to complete. But the accuracy I am getting is 0.155. I need to look into it more. I predicted few batches of images on my validation set, It's giving more than 80% accuracy on validation set.", "1165024": "save and download your trained model to local machine ,and then add the model to input directory in kaggle notebook \nafter that load your model from there", "1165041": "Hi Chanukya, I have trained my model on my local machine. Then exported the trained model to kaggle notebook and submitted. Below is my notebook link. Can you check once?\n\nhttps://www.kaggle.com/venkat2ram/cassava-efficientnet", "1165709": "Very interesting that several days after offering some advice on this post I also had a submission that scored 0.139.\n\nMy error was pretty easy to root cause. I added the most recent new model to my data set. Loaded the model but than used the wrong model name to make the prediction. Obvious errors in log that pointed to this mistake.", "1166256": "My issue resolved when I added for loop. But got 0.15 accuracy. Then I identified that I have rescaled the training set and validation set, but did not rescale the test set. So I applied rescaling and submitted it. Now I got 0.845. It's below average accuracy.. But I am happy that I learnt few things here.", "1166266": "Hi All, Thanks for your answers. There are two causes why I got 0.139 as accuracy.\n\n1) I have not used for loop to predict images from sample submission. Once I added for loop, my submission actually took 3 hours to complete. Then I knew that my submission is actually getting evaluated with large test data. And I got 0.15 accuracy.\n2) I identified another cause. i.e) I have rescaled my train set and validation set while training, but forgot to rescale my input while working on sample_submission.csv. Obviously my model weights are calculated for rescaled input but I am trying to predict with input which is not rescaled. To resolve this issue, I applied rescaling to my submission input. \n\nFinally I got 0.845 accuracy. This is with MobileNetV2. I am going to try few other models and see. Good luck.", "1166543": "Glad to hear - good luck" }, "source": "meta" }