I have QC-ed the output and everything. My notebook generates an output submission file which looks fine. But when I am trying to make a submission, the kernel runs for 9/10 hrs and gives me \"Submission Scoring Error\". I am not able to figure out a way to debug it. Any help is appreciated.
", "messages": [ { "id": "1108544", "postDate": "12/10/2020 19:16:10", "content": "I have QC-ed the output and everything. My notebook generates an output submission file which looks fine. But when I am trying to make a submission, the kernel runs for 9/10 hrs and gives me \"Submission Scoring Error\". I am not able to figure out a way to debug it. Any help is appreciated.
", "rawMarkdown": "I have QC-ed the output and everything. My notebook generates an output submission file which looks fine. But when I am trying to make a submission, the kernel runs for 9/10 hrs and gives me \"Submission Scoring Error\". I am not able to figure out a way to debug it. Any help is appreciated.", "votes": null }, { "id": "1108586", "postDate": "12/10/2020 20:08:21", "content": "It seems to be memory problem. If it so try to delete high memory features , one at a time, and retry.Don't forget that the run time is obfuscated
", "rawMarkdown": "It seems to be memory problem. If it so try to delete high memory features , one at a time, and retry.Don't forget that the run time is obfuscated", "votes": null }, { "id": "1108614", "postDate": "12/10/2020 20:39:26", "content": "My normal kernel isn't taking that much to save & commit. Seems pretty weird. Lets see if this step by step thing works. Even I thought of that as the only way of debugging.
", "rawMarkdown": "My normal kernel isn't taking that much to save & commit. Seems pretty weird. Lets see if this step by step thing works. Even I thought of that as the only way of debugging.", "votes": null }, { "id": "1108657", "postDate": "12/10/2020 21:31:41", "content": "Nornal kernel resources have nothing to do with the submission ones.With no logs available only trial and error are on the table.
\nFinally ,submission times of 8-9 hours, even derived from obfuscation, must be unacceptable in kaggle competitions
Maybe your code does not account for all cases, then in the testing set, it encounters these cases and leads to some errors (e.g. index out of range error). For example, in my case, I embed a some categorical numbers, say from 0 to 9, then I set nn.Embedding(10, embed_dim). In the training set, all values from that feature range from 0 to 9, but in the testing set, somehow a value of 10 pops up and the code stops running. So, try to clip everything, or make sure that, some values out of the range can not be happened.
\nAt least, in my case, it works. I hope it helps…
", "rawMarkdown": "Maybe your code does not account for all cases, then in the testing set, it encounters these cases and leads to some errors (e.g. index out of range error). For example, in my case, I embed a some categorical numbers, say from 0 to 9, then I set nn.Embedding(10, embed_dim). In the training set, all values from that feature range from 0 to 9, but in the testing set, somehow a value of 10 pops up and the code stops running. So, try to clip everything, or make sure that, some values out of the range can not be happened.\n\nAt least, in my case, it works. I hope it helps...", "votes": null }, { "id": "1108710", "postDate": "12/10/2020 23:43:32", "content": "I use @its7171 's iter_env simulation to double check. It helped me debug several errors in my submission code's feature eng.
", "rawMarkdown": "I use @its7171 's iter_env simulation to double check. It helped me debug several errors in my submission code's feature eng.", "votes": null }, { "id": "1109938", "postDate": "12/12/2020 08:26:45", "content": "Figured out the problem. There was a pd.merging part which was taking lot of time to run, eventually leading to timeout and hence submission file had NaNs
", "rawMarkdown": "Figured out the problem. There was a pd.merging part which was taking lot of time to run, eventually leading to timeout and hence submission file had NaNs", "votes": null }, { "id": "1109981", "postDate": "12/12/2020 09:19:08", "content": "If I can give you an advice is to forget about pandas in this competition for feature creation. It will be slow and memory costly.
\nYou rather go with a simple \"for\" loop and numpy arrays / dictionnaries 🙂
", "rawMarkdown": "If I can give you an advice is to forget about pandas in this competition for feature creation. It will be slow and memory costly.\n\nYou rather go with a simple \"for\" loop and numpy arrays / dictionnaries 🙂", "votes": null } ], "comments": [ { "id": 1108586, "author_name": "georgem", "author_url": "", "post_date": "12/10/2020 20:08:21", "content": "It seems to be memory problem. If it so try to delete high memory features , one at a time, and retry.Don't forget that the run time is obfuscated
", "votes": null, "replies": [ { "id": 1108614, "author_name": "duttadebadri", "author_url": "", "post_date": "12/10/2020 20:39:26", "content": "My normal kernel isn't taking that much to save & commit. Seems pretty weird. Lets see if this step by step thing works. Even I thought of that as the only way of debugging.
", "votes": null, "replies": [] }, { "id": 1108657, "author_name": "georgem", "author_url": "", "post_date": "12/10/2020 21:31:41", "content": "Nornal kernel resources have nothing to do with the submission ones.With no logs available only trial and error are on the table.
\nFinally ,submission times of 8-9 hours, even derived from obfuscation, must be unacceptable in kaggle competitions
Maybe your code does not account for all cases, then in the testing set, it encounters these cases and leads to some errors (e.g. index out of range error). For example, in my case, I embed a some categorical numbers, say from 0 to 9, then I set nn.Embedding(10, embed_dim). In the training set, all values from that feature range from 0 to 9, but in the testing set, somehow a value of 10 pops up and the code stops running. So, try to clip everything, or make sure that, some values out of the range can not be happened.
\nAt least, in my case, it works. I hope it helps…
", "votes": null, "replies": [] }, { "id": 1108710, "author_name": "scaomath", "author_url": "", "post_date": "12/10/2020 23:43:32", "content": "I use @its7171 's iter_env simulation to double check. It helped me debug several errors in my submission code's feature eng.
", "votes": null, "replies": [] }, { "id": 1109938, "author_name": "duttadebadri", "author_url": "", "post_date": "12/12/2020 08:26:45", "content": "Figured out the problem. There was a pd.merging part which was taking lot of time to run, eventually leading to timeout and hence submission file had NaNs
", "votes": null, "replies": [ { "id": 1109981, "author_name": "bowaka", "author_url": "", "post_date": "12/12/2020 09:19:08", "content": "If I can give you an advice is to forget about pandas in this competition for feature creation. It will be slow and memory costly.
\nYou rather go with a simple \"for\" loop and numpy arrays / dictionnaries 🙂
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1108544": "I have QC-ed the output and everything. My notebook generates an output submission file which looks fine. But when I am trying to make a submission, the kernel runs for 9/10 hrs and gives me \"Submission Scoring Error\". I am not able to figure out a way to debug it. Any help is appreciated.", "1108586": "It seems to be memory problem. If it so try to delete high memory features , one at a time, and retry.Don't forget that the run time is obfuscated", "1108614": "My normal kernel isn't taking that much to save & commit. Seems pretty weird. Lets see if this step by step thing works. Even I thought of that as the only way of debugging.", "1108657": "Nornal kernel resources have nothing to do with the submission ones.With no logs available only trial and error are on the table.\nFinally ,submission times of 8-9 hours, even derived from obfuscation, must be unacceptable in kaggle competitions", "1108681": "Maybe your code does not account for all cases, then in the testing set, it encounters these cases and leads to some errors (e.g. index out of range error). For example, in my case, I embed a some categorical numbers, say from 0 to 9, then I set nn.Embedding(10, embed_dim). In the training set, all values from that feature range from 0 to 9, but in the testing set, somehow a value of 10 pops up and the code stops running. So, try to clip everything, or make sure that, some values out of the range can not be happened.\n\nAt least, in my case, it works. I hope it helps...", "1108710": "I use @its7171 's iter_env simulation to double check. It helped me debug several errors in my submission code's feature eng.", "1109938": "Figured out the problem. There was a pd.merging part which was taking lot of time to run, eventually leading to timeout and hence submission file had NaNs", "1109981": "If I can give you an advice is to forget about pandas in this competition for feature creation. It will be slow and memory costly.\n\nYou rather go with a simple \"for\" loop and numpy arrays / dictionnaries 🙂" }, "source": "meta" }