In some cases I am encountering an error where I would provide a pretrained (e.g. from a previous run) model and once I run the notebook and submit the csv file, I would end up getting 0.0000 score despite the model showing promising validation accuracy. Has anyone else encountered this problem? Any clues on what is happening exactly?
", "messages": [ { "id": "1009209", "postDate": "09/13/2020 18:26:17", "content": "In some cases I am encountering an error where I would provide a pretrained (e.g. from a previous run) model and once I run the notebook and submit the csv file, I would end up getting 0.0000 score despite the model showing promising validation accuracy. Has anyone else encountered this problem? Any clues on what is happening exactly?
", "rawMarkdown": "In some cases I am encountering an error where I would provide a pretrained (e.g. from a previous run) model and once I run the notebook and submit the csv file, I would end up getting 0.0000 score despite the model showing promising validation accuracy. Has anyone else encountered this problem? Any clues on what is happening exactly?", "votes": null }, { "id": "1009278", "postDate": "09/13/2020 20:02:01", "content": "Try reordering your submission file. The initial order of the predicted samples should be as per the order given in sample_submission.csv.
", "rawMarkdown": "Try reordering your submission file. The initial order of the predicted samples should be as per the order given in sample_submission.csv.", "votes": null }, { "id": "1009825", "postDate": "09/14/2020 09:16:46", "content": "I have double checked and the produced submission.csv file is in the same order as the sample. I am not sure how important this is.
", "rawMarkdown": "I have double checked and the produced submission.csv file is in the same order as the sample. I am not sure how important this is.", "votes": null }, { "id": "1013640", "postDate": "09/16/2020 20:58:58", "content": "@pevogam .. getting a 0.0000 submission score in combination with a good validation score is what typically happens if you train with a low number of classes. If you only select the top 1000 or 2000 classes then this could happen. Can you check if that would be the case?
\nIf so the solution would then be easy…increases the number of classes. Good luck!
", "rawMarkdown": "pevogam .. getting a 0.0000 submission score in combination with a good validation score is what typically happens if you train with a low number of classes. If you only select the top 1000 or 2000 classes then this could happen. Can you check if that would be the case?\n\nIf so the solution would then be easy...increases the number of classes. Good luck!", "votes": null }, { "id": "1016377", "postDate": "09/18/2020 22:33:35", "content": "Hi @rsmits, the problem is that I get both the 0.0 score and higher scores. If I train a model loaded from added dataset and submit, I get a nonzero (e.g. 10%) GAP score. However, if I only load the model from the added dataset (trained model and even checked with some validation before generation the submission file), I get the 0.0 score. I suspect there is something in the training that I might be missing by so far everything I have is pretty standard.
", "rawMarkdown": "Hi @rsmits, the problem is that I get both the 0.0 score and higher scores. If I train a model loaded from added dataset and submit, I get a nonzero (e.g. 10%) GAP score. However, if I only load the model from the added dataset (trained model and even checked with some validation before generation the submission file), I get the 0.0 score. I suspect there is something in the training that I might be missing by so far everything I have is pretty standard.", "votes": null } ], "comments": [ { "id": 1009278, "author_name": "dsnil87", "author_url": "", "post_date": "09/13/2020 20:02:01", "content": "Try reordering your submission file. The initial order of the predicted samples should be as per the order given in sample_submission.csv.
", "votes": null, "replies": [ { "id": 1009825, "author_name": "pevogam", "author_url": "", "post_date": "09/14/2020 09:16:46", "content": "I have double checked and the produced submission.csv file is in the same order as the sample. I am not sure how important this is.
", "votes": null, "replies": [] } ] }, { "id": 1013640, "author_name": "rsmits", "author_url": "", "post_date": "09/16/2020 20:58:58", "content": "@pevogam .. getting a 0.0000 submission score in combination with a good validation score is what typically happens if you train with a low number of classes. If you only select the top 1000 or 2000 classes then this could happen. Can you check if that would be the case?
\nIf so the solution would then be easy…increases the number of classes. Good luck!
", "votes": null, "replies": [ { "id": 1016377, "author_name": "pevogam", "author_url": "", "post_date": "09/18/2020 22:33:35", "content": "Hi @rsmits, the problem is that I get both the 0.0 score and higher scores. If I train a model loaded from added dataset and submit, I get a nonzero (e.g. 10%) GAP score. However, if I only load the model from the added dataset (trained model and even checked with some validation before generation the submission file), I get the 0.0 score. I suspect there is something in the training that I might be missing by so far everything I have is pretty standard.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1009209": "In some cases I am encountering an error where I would provide a pretrained (e.g. from a previous run) model and once I run the notebook and submit the csv file, I would end up getting 0.0000 score despite the model showing promising validation accuracy. Has anyone else encountered this problem? Any clues on what is happening exactly?", "1009278": "Try reordering your submission file. The initial order of the predicted samples should be as per the order given in sample_submission.csv.", "1009825": "I have double checked and the produced submission.csv file is in the same order as the sample. I am not sure how important this is.", "1013640": "pevogam .. getting a 0.0000 submission score in combination with a good validation score is what typically happens if you train with a low number of classes. If you only select the top 1000 or 2000 classes then this could happen. Can you check if that would be the case?\n\nIf so the solution would then be easy...increases the number of classes. Good luck!", "1016377": "Hi @rsmits, the problem is that I get both the 0.0 score and higher scores. If I train a model loaded from added dataset and submit, I get a nonzero (e.g. 10%) GAP score. However, if I only load the model from the added dataset (trained model and even checked with some validation before generation the submission file), I get the 0.0 score. I suspect there is something in the training that I might be missing by so far everything I have is pretty standard." }, "source": "meta" }