Hey kagglers,
\nFrom the past few days, I have been trying to submit multiple models, but all of them are giving me a weird accuracy of 0.139 whereas they are doing fine when I validated on some part of trainset(I was doing kfold validation). I tried with efficientnets, ViT, but that figure seems to be constant for all these model on lb. Does anybody experienced such a thing while you are trying to submit. If so, do let me know how you fixed it!
\nThanks you in advance!
Hey kagglers,
\nFrom the past few days, I have been trying to submit multiple models, but all of them are giving me a weird accuracy of 0.139 whereas they are doing fine when I validated on some part of trainset(I was doing kfold validation). I tried with efficientnets, ViT, but that figure seems to be constant for all these model on lb. Does anybody experienced such a thing while you are trying to submit. If so, do let me know how you fixed it!
\nThanks you in advance!
Debug your inference kernel, I had this error in the beginning, and it was due to a bad indentation in my inference method.
", "rawMarkdown": "Debug your inference kernel, I had this error in the beginning, and it was due to a bad indentation in my inference method.", "votes": null }, { "id": "1177343", "postDate": "01/30/2021 08:09:36", "content": "Hello!
\nAccording to this class distribution of the public test set, it seems that your model mostly predicts label 4. I suppose you've just forgotten to normalize the test set before calling predict on it.
hey @nickuzmenkov , thanks for your reply,
\nBut I don't think my model is biased towards to label 4, because it's predicting label 2 for the one single image we got in test dataset. Moreover, my albumentation pipeline includes normalization as well. If possible have a look at my kernel here.
I have thoroughly checked it but yet didn't find any such issues!
\nIf possible, have a look at it here
Getting a 404
", "rawMarkdown": "Getting a 404", "votes": null }, { "id": "1177381", "postDate": "01/30/2021 08:37:57", "content": "Thanks for sharing, @saimanojakondi
\nHowever, the link is broken
apologies,
\ntry this one.
apologies,
\ntry this one
Did you read this post?
\nhttps://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/212455
\nWill look at your model shortly but 139 seems to be a score when your model does no predicting.
", "rawMarkdown": "Did you read this post?\n\n[https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/212455](url)\n\nWill look at your model shortly but 139 seems to be a score when your model does no predicting.", "votes": null }, { "id": "1177446", "postDate": "01/30/2021 09:38:09", "content": "I forked your model - since you have private data for the model file, it of course fails for me with this error. (YES YES - I know it should fail when I run it)
\nFileNotFoundError: [Errno 2] No such file or directory: '../input/effnetb4/vit_large_patch16_384_best_fold1.pth'
But guess what value I predicted for the single test image - yep a big old 2.
\nYour generating some type of similar error.
", "rawMarkdown": "I forked your model - since you have private data for the model file, it of course fails for me with this error. (YES YES - I know it should fail when I run it)\n\n`FileNotFoundError: [Errno 2] No such file or directory: '../input/effnetb4/vit_large_patch16_384_best_fold1.pth'`\n\nBut guess what value I predicted for the single test image - yep a big old 2.\n\nYour generating some type of similar error.", "votes": null }, { "id": "1177469", "postDate": "01/30/2021 09:55:35", "content": "Thanks for sharing the thread, I had a similar doubt whether my model is predicting or not, so i ran the same inference kernel on few images of train data…model is predicting those properly.
", "rawMarkdown": "Thanks for sharing the thread, I had a similar doubt whether my model is predicting or not, so i ran the same inference kernel on few images of train data...model is predicting those properly.", "votes": null }, { "id": "1177509", "postDate": "01/30/2021 10:27:05", "content": "You should be getting 0 i hope….because i tried commenting out the line which is loading the weights and tried to predict and I got 0 as my label, anyways let me check it again ;)
\nBut still, it didn't understand why model is not predicting, like the code is pretty much straight forward
There probably a nice long list of things that kill the model on a submit that don’t kill it during a commit. How long after you submit can you view the submission score? Should be several hours.
", "rawMarkdown": "There probably a nice long list of things that kill the model on a submit that don’t kill it during a commit. How long after you submit can you view the submission score? Should be several hours.", "votes": null }, { "id": "1177579", "postDate": "01/30/2021 11:47:37", "content": "😂that's one more tricky thing here, it took couple of mins(<60) to get the score on lb
", "rawMarkdown": "😂that's one more tricky thing here, it took couple of mins(<60) to get the score on lb", "votes": null }, { "id": "1178059", "postDate": "01/30/2021 16:22:44", "content": "Don't due Pytorch so not able to read the code and don't have a Pytorch model that I could use in place of your private.
\nI would use GPU rather than cpu - was the model trained with GPU? Pretty sure it's an error loading the model or killing it early.
", "rawMarkdown": "Don't due Pytorch so not able to read the code and don't have a Pytorch model that I could use in place of your private. \n\nI would use GPU rather than cpu - was the model trained with GPU? Pretty sure it's an error loading the model or killing it early.", "votes": null }, { "id": "1178174", "postDate": "01/30/2021 17:45:27", "content": "yes the model is trained on GPU…If it's due to loading it should not predict stuff while committing the kernel also, isn't it?
", "rawMarkdown": "yes the model is trained on GPU...If it's due to loading it should not predict stuff while committing the kernel also, isn't it?", "votes": null }, { "id": "1178365", "postDate": "01/30/2021 20:21:44", "content": "When I had my error that predicted 0.139 I had attempted to run a model I had not loaded - so something gets written to a submission file with the hidden test process even if no model existing - if you trained on GPU I would submit with GPU turned on and change the right code for GPU predicting.
", "rawMarkdown": "When I had my error that predicted 0.139 I had attempted to run a model I had not loaded - so something gets written to a submission file with the hidden test process even if no model existing - if you trained on GPU I would submit with GPU turned on and change the right code for GPU predicting.", "votes": null }, { "id": "1178932", "postDate": "01/31/2021 07:36:17", "content": "Just now, I tried with a tpu model and it took lot of time…finally a big notebook timed out error!!😂
", "rawMarkdown": "Just now, I tried with a tpu model and it took lot of time...finally a big notebook timed out error!!😂", "votes": null }, { "id": "1179125", "postDate": "01/31/2021 10:42:52", "content": "Whats that saying about 1 step forward and 2 steps back ? Good luck 👍👍👍
", "rawMarkdown": "Whats that saying about 1 step forward and 2 steps back ? Good luck 👍👍👍", "votes": null } ], "comments": [ { "id": 1177342, "author_name": "zowlex", "author_url": "", "post_date": "01/30/2021 08:08:46", "content": "Debug your inference kernel, I had this error in the beginning, and it was due to a bad indentation in my inference method.
", "votes": null, "replies": [ { "id": 1177364, "author_name": "saimanojakondi", "author_url": "", "post_date": "01/30/2021 08:28:55", "content": "I have thoroughly checked it but yet didn't find any such issues!
\nIf possible, have a look at it here
Getting a 404
", "votes": null, "replies": [] }, { "id": 1177385, "author_name": "saimanojakondi", "author_url": "", "post_date": "01/30/2021 08:44:00", "content": "apologies,
\ntry this one
Hello!
\nAccording to this class distribution of the public test set, it seems that your model mostly predicts label 4. I suppose you've just forgotten to normalize the test set before calling predict on it.
hey @nickuzmenkov , thanks for your reply,
\nBut I don't think my model is biased towards to label 4, because it's predicting label 2 for the one single image we got in test dataset. Moreover, my albumentation pipeline includes normalization as well. If possible have a look at my kernel here.
Thanks for sharing, @saimanojakondi
\nHowever, the link is broken
apologies,
\ntry this one.
Did you read this post?
\nhttps://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/212455
\nWill look at your model shortly but 139 seems to be a score when your model does no predicting.
", "votes": null, "replies": [ { "id": 1177469, "author_name": "saimanojakondi", "author_url": "", "post_date": "01/30/2021 09:55:35", "content": "Thanks for sharing the thread, I had a similar doubt whether my model is predicting or not, so i ran the same inference kernel on few images of train data…model is predicting those properly.
", "votes": null, "replies": [] } ] }, { "id": 1177446, "author_name": "pcjimmmy", "author_url": "", "post_date": "01/30/2021 09:38:09", "content": "I forked your model - since you have private data for the model file, it of course fails for me with this error. (YES YES - I know it should fail when I run it)
\nFileNotFoundError: [Errno 2] No such file or directory: '../input/effnetb4/vit_large_patch16_384_best_fold1.pth'
But guess what value I predicted for the single test image - yep a big old 2.
\nYour generating some type of similar error.
", "votes": null, "replies": [ { "id": 1177509, "author_name": "saimanojakondi", "author_url": "", "post_date": "01/30/2021 10:27:05", "content": "You should be getting 0 i hope….because i tried commenting out the line which is loading the weights and tried to predict and I got 0 as my label, anyways let me check it again ;)
\nBut still, it didn't understand why model is not predicting, like the code is pretty much straight forward
There probably a nice long list of things that kill the model on a submit that don’t kill it during a commit. How long after you submit can you view the submission score? Should be several hours.
", "votes": null, "replies": [] }, { "id": 1177579, "author_name": "saimanojakondi", "author_url": "", "post_date": "01/30/2021 11:47:37", "content": "😂that's one more tricky thing here, it took couple of mins(<60) to get the score on lb
", "votes": null, "replies": [] }, { "id": 1178059, "author_name": "pcjimmmy", "author_url": "", "post_date": "01/30/2021 16:22:44", "content": "Don't due Pytorch so not able to read the code and don't have a Pytorch model that I could use in place of your private.
\nI would use GPU rather than cpu - was the model trained with GPU? Pretty sure it's an error loading the model or killing it early.
", "votes": null, "replies": [] }, { "id": 1178174, "author_name": "saimanojakondi", "author_url": "", "post_date": "01/30/2021 17:45:27", "content": "yes the model is trained on GPU…If it's due to loading it should not predict stuff while committing the kernel also, isn't it?
", "votes": null, "replies": [] }, { "id": 1178365, "author_name": "pcjimmmy", "author_url": "", "post_date": "01/30/2021 20:21:44", "content": "When I had my error that predicted 0.139 I had attempted to run a model I had not loaded - so something gets written to a submission file with the hidden test process even if no model existing - if you trained on GPU I would submit with GPU turned on and change the right code for GPU predicting.
", "votes": null, "replies": [] }, { "id": 1178932, "author_name": "saimanojakondi", "author_url": "", "post_date": "01/31/2021 07:36:17", "content": "Just now, I tried with a tpu model and it took lot of time…finally a big notebook timed out error!!😂
", "votes": null, "replies": [] }, { "id": 1179125, "author_name": "pcjimmmy", "author_url": "", "post_date": "01/31/2021 10:42:52", "content": "Whats that saying about 1 step forward and 2 steps back ? Good luck 👍👍👍
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1177242": "Hey kagglers,\nFrom the past few days, I have been trying to submit multiple models, but all of them are giving me a weird accuracy of 0.139 whereas they are doing fine when I validated on some part of trainset(I was doing kfold validation). I tried with efficientnets, ViT, but that figure seems to be constant for all these model on lb. Does anybody experienced such a thing while you are trying to submit. If so, do let me know how you fixed it!\nThanks you in advance!", "1177342": "Debug your inference kernel, I had this error in the beginning, and it was due to a bad indentation in my inference method.", "1177343": "Hello!\n\nAccording to this **[class distribution](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/202943)** of the public test set, it seems that your model mostly predicts label 4. I suppose you've just forgotten to normalize the test set before calling `predict` on it.", "1177362": "hey @nickuzmenkov , thanks for your reply,\nBut I don't think my model is biased towards to label 4, because it's predicting label 2 for the one single image we got in test dataset. Moreover, my albumentation pipeline includes normalization as well. If possible have a look at my kernel [here](https://www.kaggle.com/saimanojakondi/inference?scriptVersionId=53074627).", "1177364": "I have thoroughly checked it but yet didn't find any such issues!\nIf possible, have a look at it [here](https://www.kaggle.com/saimanojakondi/inference?scriptVersionId=53074627)", "1177379": "Getting a 404", "1177381": "Thanks for sharing, @saimanojakondi \nHowever, the link is broken", "1177384": "apologies,\ntry this [one](https://www.kaggle.com/saimanojakondi/inference).", "1177385": "apologies,\ntry this [one](https://www.kaggle.com/saimanojakondi/inference)", "1177433": "Did you read this post?\n\n[https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/212455](url)\n\nWill look at your model shortly but 139 seems to be a score when your model does no predicting.", "1177446": "I forked your model - since you have private data for the model file, it of course fails for me with this error. (YES YES - I know it should fail when I run it)\n\n`FileNotFoundError: [Errno 2] No such file or directory: '../input/effnetb4/vit_large_patch16_384_best_fold1.pth'`\n\nBut guess what value I predicted for the single test image - yep a big old 2.\n\nYour generating some type of similar error.", "1177469": "Thanks for sharing the thread, I had a similar doubt whether my model is predicting or not, so i ran the same inference kernel on few images of train data...model is predicting those properly.", "1177509": "You should be getting 0 i hope....because i tried commenting out the line which is loading the weights and tried to predict and I got 0 as my label, anyways let me check it again ;)\nBut still, it didn't understand why model is not predicting, like the code is pretty much straight forward", "1177560": "There probably a nice long list of things that kill the model on a submit that don’t kill it during a commit. How long after you submit can you view the submission score? Should be several hours.", "1177579": "😂that's one more tricky thing here, it took couple of mins(<60) to get the score on lb", "1178059": "Don't due Pytorch so not able to read the code and don't have a Pytorch model that I could use in place of your private. \n\nI would use GPU rather than cpu - was the model trained with GPU? Pretty sure it's an error loading the model or killing it early.", "1178174": "yes the model is trained on GPU...If it's due to loading it should not predict stuff while committing the kernel also, isn't it?", "1178365": "When I had my error that predicted 0.139 I had attempted to run a model I had not loaded - so something gets written to a submission file with the hidden test process even if no model existing - if you trained on GPU I would submit with GPU turned on and change the right code for GPU predicting.", "1178932": "Just now, I tried with a tpu model and it took lot of time...finally a big notebook timed out error!!😂", "1179125": "Whats that saying about 1 step forward and 2 steps back ? Good luck 👍👍👍" }, "source": "meta" }