![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7801821%2Fce87054191bf147c39b8c558a489bc93%2FCapture%20dcran%202023-06-24%20090220.png?generation=1687611749409403&alt=media\")
I am an active participant in this competition. I am writing to express my deep concern and dissatisfaction regarding the persistent issues we, as participants, are experiencing during the submission process.
\nWe are dedicating substantial time and effort in crafting advanced, innovative models. However, the roadblock of a difficult and error-prone submission process has been a consistent pain point, creating an unnecessarily stressful and disheartening competition experience.
\nA competition participant should, in principle, not have to invest an inordinate amount of effort and time into just being able to submit their work correctly. The time we invest in this competition is valuable, and it is deeply disconcerting to see it wasted on battling submission issues rather than improving our models and fostering learning.
\nThe gravity of this issue is intensified considering the limited number of submission attempts we have each day. Every erroneous submission squanders a precious attempt and deprives us of the opportunity to test, learn, and enhance our models.
\nThe cornerstone of such competitions should be the pursuit of innovation and improvement, not the navigation of submission intricacies. When we have to divert our energy away from model development and towards solving submission issues, the learning objective of this competition is not being met.
\nWhile I do recognize and appreciate the hard work that goes into organizing a competition of this scale, it's essential that these long-standing issues are addressed with urgency. The experience and motivation of numerous participants, including myself, are being adversely affected.
\nI kindly ask the organizing team to address these problems promptly, or at least provide clear communication about the actions being taken towards resolving these issues. Ensuring a smooth, user-friendly submission process is vital to let us focus on what truly matters - developing the best possible models and learning in the process.
\nThank you for your immediate attention to this matter. I trust in your commitment to improve our experience and look forward to a swift and effective resolution of these issues.
", "messages": [ { "id": "2315331", "postDate": "06/24/2023 04:11:23", "content": "I am an active participant in this competition. I am writing to express my deep concern and dissatisfaction regarding the persistent issues we, as participants, are experiencing during the submission process.
\nWe are dedicating substantial time and effort in crafting advanced, innovative models. However, the roadblock of a difficult and error-prone submission process has been a consistent pain point, creating an unnecessarily stressful and disheartening competition experience.
\nA competition participant should, in principle, not have to invest an inordinate amount of effort and time into just being able to submit their work correctly. The time we invest in this competition is valuable, and it is deeply disconcerting to see it wasted on battling submission issues rather than improving our models and fostering learning.
\nThe gravity of this issue is intensified considering the limited number of submission attempts we have each day. Every erroneous submission squanders a precious attempt and deprives us of the opportunity to test, learn, and enhance our models.
\nThe cornerstone of such competitions should be the pursuit of innovation and improvement, not the navigation of submission intricacies. When we have to divert our energy away from model development and towards solving submission issues, the learning objective of this competition is not being met.
\nWhile I do recognize and appreciate the hard work that goes into organizing a competition of this scale, it's essential that these long-standing issues are addressed with urgency. The experience and motivation of numerous participants, including myself, are being adversely affected.
\nI kindly ask the organizing team to address these problems promptly, or at least provide clear communication about the actions being taken towards resolving these issues. Ensuring a smooth, user-friendly submission process is vital to let us focus on what truly matters - developing the best possible models and learning in the process.
\nThank you for your immediate attention to this matter. I trust in your commitment to improve our experience and look forward to a swift and effective resolution of these issues.
", "rawMarkdown": "I am an active participant in this competition. I am writing to express my deep concern and dissatisfaction regarding the persistent issues we, as participants, are experiencing during the submission process.\n\nWe are dedicating substantial time and effort in crafting advanced, innovative models. However, the roadblock of a difficult and error-prone submission process has been a consistent pain point, creating an unnecessarily stressful and disheartening competition experience.\n\nA competition participant should, in principle, not have to invest an inordinate amount of effort and time into just being able to submit their work correctly. The time we invest in this competition is valuable, and it is deeply disconcerting to see it wasted on battling submission issues rather than improving our models and fostering learning.\n\nThe gravity of this issue is intensified considering the limited number of submission attempts we have each day. Every erroneous submission squanders a precious attempt and deprives us of the opportunity to test, learn, and enhance our models.\n\nThe cornerstone of such competitions should be the pursuit of innovation and improvement, not the navigation of submission intricacies. When we have to divert our energy away from model development and towards solving submission issues, the learning objective of this competition is not being met.\n\nWhile I do recognize and appreciate the hard work that goes into organizing a competition of this scale, it's essential that these long-standing issues are addressed with urgency. The experience and motivation of numerous participants, including myself, are being adversely affected.\n\nI kindly ask the organizing team to address these problems promptly, or at least provide clear communication about the actions being taken towards resolving these issues. Ensuring a smooth, user-friendly submission process is vital to let us focus on what truly matters - developing the best possible models and learning in the process. \n\nThank you for your immediate attention to this matter. I trust in your commitment to improve our experience and look forward to a swift and effective resolution of these issues.", "votes": null }, { "id": "2315525", "postDate": "06/24/2023 07:51:08", "content": "I suggest you could escalate these issues to Kaggle by writing this on the product feedback page. I am sure this will be addressed better there.
", "rawMarkdown": "I suggest you could escalate these issues to Kaggle by writing this on the **product feedback** page. I am sure this will be addressed better there.", "votes": null }, { "id": "2315563", "postDate": "06/24/2023 08:25:31", "content": "I appreciate your suggestion, I went ahead and contacted the hosts and a person from Kaggle stuff. I received an answer from the hosts, although they have dedicated some time to answer me, the proposals they gave me didn't really work sadly.
", "rawMarkdown": "I appreciate your suggestion, I went ahead and contacted the hosts and a person from Kaggle stuff. I received an answer from the hosts, although they have dedicated some time to answer me, the proposals they gave me didn't really work sadly.", "votes": null }, { "id": "2315821", "postDate": "06/24/2023 12:57:01", "content": "i agree with you - i had a number of submissions that ran fine over open test set yet failed on hidden test set. In my opinion, this should NEVER be allowed to happen - open test set should be representative of the hidden test set, so that it can actually be used to debug your model. Otherwise, what is the point of it?
\nI think it is the responsibility of competition organizers to select open test set that fully represents the features of the hidden test set.
\nHopefully in future competitions organizers will do a better job selecting open test set.
", "rawMarkdown": "i agree with you - i had a number of submissions that ran fine over open test set yet failed on hidden test set. In my opinion, this should NEVER be allowed to happen - open test set should be representative of the hidden test set, so that it can actually be used to debug your model. Otherwise, what is the point of it?\n\nI think it is the responsibility of competition organizers to select open test set that fully represents the features of the hidden test set.\n\nHopefully in future competitions organizers will do a better job selecting open test set.", "votes": null }, { "id": "2315828", "postDate": "06/24/2023 13:00:39", "content": "I totally agree with you, unfortunately we can't afford to fully focus on our models only. On my end, although I made quiet a good model with great benchmarks I was never able to submit it. The only submission I was able to do was a copy paste from a public notebook. Hopefully this never happens again.
", "rawMarkdown": "I totally agree with you, unfortunately we can't afford to fully focus on our models only. On my end, although I made quiet a good model with great benchmarks I was never able to submit it. The only submission I was able to do was a copy paste from a public notebook. Hopefully this never happens again.", "votes": null }, { "id": "2315831", "postDate": "06/24/2023 13:02:39", "content": "
If you don't mind me asking, how do you submit successfully ? What is the trick ?
", "rawMarkdown": "If you don't mind me asking, how do you submit successfully ? What is the trick ?", "votes": null }, { "id": "2315891", "postDate": "06/24/2023 13:53:30", "content": "Hi @qurious:
\ndoes it fail quickly or after a few hours?
\ndo you have the same python version in your training and your submission notebook?
\ndo you sort test by index or elapsed_time?
\nis your mask carefully selecting the right question?
I agree that it is frustrating and I spent many, many hours guess-debugging the submission…
", "rawMarkdown": "Hi @qurious:\ndoes it fail quickly or after a few hours?\ndo you have the same python version in your training and your submission notebook?\ndo you sort test by index or elapsed_time?\nis your mask carefully selecting the right question?\n\nI agree that it is frustrating and I spent many, many hours guess-debugging the submission...", "votes": null }, { "id": "2315898", "postDate": "06/24/2023 14:01:24", "content": "Hi Elias, it depends on each notebook since I tried various iterations. I applied my mask to select the right questions to answer for on some submits, I attempted to assemble my own answer dataframe to populate the submission variable. I've sorted by question, by session_id, unsorted, simple boolean mask. The only submission that worked was made using a public notebook, I attempted to adapt the notebook to my model but it still never worked. and yes I used the same python with the notebook I am import from.
", "rawMarkdown": "Hi Elias, it depends on each notebook since I tried various iterations. I applied my mask to select the right questions to answer for on some submits, I attempted to assemble my own answer dataframe to populate the submission variable. I've sorted by question, by session_id, unsorted, simple boolean mask. The only submission that worked was made using a public notebook, I attempted to adapt the notebook to my model but it still never worked. and yes I used the same python with the notebook I am import from.", "votes": null }, { "id": "2316368", "postDate": "06/24/2023 20:38:42", "content": "If you click on the failed notebook and then go to \"Logs\", it may show the reason.
", "rawMarkdown": "If you click on the failed notebook and then go to \"Logs\", it may show the reason.", "votes": null }, { "id": "2316376", "postDate": "06/24/2023 20:47:28", "content": "Scorring Error which is the vaguest error ever, print debug statements did not help either
", "rawMarkdown": "Scorring Error which is the vaguest error ever, print debug statements did not help either", "votes": null }, { "id": "2316994", "postDate": "06/25/2023 11:27:50", "content": "i simplified my submission code until it worked, then rebuilt it back one step at a time.
\nI still have no idea what was causing the problem - final working submission code looks equivalent to non-working submission code.
I'm also stuck with those submission errors, I just lost 5 other submissions today.
\nEven with a code like this, I still get submission scoring error
\n (test, sam_sub) iter_test:\n sam_sub[] = \n sam_sub[] = sam_sub.session_id.apply( x : (x.split()[][:]) )\n sam_sub = sam_sub.sort_values()\n :\n do some FE\n do predictions\n :\n \n env.predict(sam_sub[[, ]])\nI don't understand what's going on, if some error happens, my code should work and submits only 0s.
\nBtw, running same code on the full training data without try/except doesn't throw any error
\nmy simulation on training code looks like this:
\n session_id, user_df tqdm( train_df.groupby(, sort=), total = train_df.session_id.nunique() ): \n level_group [, , ]:\n test = user_df.loc[user_df.level_group==level_group] \n test = test.sample(frac=).reset_index(drop=) \n do some FE\n do predictions\nHave you tried to avoid sorting sample submission by question number? This is just my guess, but may be the original order is somehow enforced during evaluation?
\nI’m using selection by mask to update the correct column and never had any issues with my submissions.
the sorting is part of my previous notebooks that did not throw the submission scoring error. So, the problem is not there.
\nThe error happens after 3 min of running the submission. Knowing that my notebook takes 8-9 min to finish, this means that 30% of the data has been predicted without errors
", "rawMarkdown": "the sorting is part of my previous notebooks that did not throw the submission scoring error. So, the problem is not there.\n\nThe error happens after 3 min of running the submission. Knowing that my notebook takes 8-9 min to finish, this means that 30% of the data has been predicted without errors", "votes": null }, { "id": "2317595", "postDate": "06/25/2023 19:43:36", "content": "are you saying that the submission process results in a error the moment the running time goes over 3 minutes ?
", "rawMarkdown": "are you saying that the submission process results in a error the moment the running time goes over 3 minutes ?", "votes": null }, { "id": "2317602", "postDate": "06/25/2023 19:49:10", "content": "yes, after 3 mins of execution, the error happens
", "rawMarkdown": "yes, after 3 mins of execution, the error happens", "votes": null }, { "id": "2317683", "postDate": "06/25/2023 21:32:35", "content": "This might be the issue, it takes a few minutes to process my features, was this announced anywhere ?
", "rawMarkdown": "This might be the issue, it takes a few minutes to process my features, was this announced anywhere ?", "votes": null }, { "id": "2318866", "postDate": "06/26/2023 16:23:34", "content": "@mchahhou i faced that too before the last annoying updates(in private testset and API), the problem was that there are missing levels in group 13_22, my code was assuming that all levels are available but they weren't which lead to the error after 3~4 min. now after the last rerun of LB all my submissions are turned to be erronous, still trying to figure it out.
", "rawMarkdown": "mchahhou i faced that too before the last annoying updates(in private testset and API), the problem was that there are missing levels in group 13_22, my code was assuming that all levels are available but they weren't which lead to the error after 3~4 min. now after the last rerun of LB all my submissions are turned to be erronous, still trying to figure it out.", "votes": null }, { "id": "2318876", "postDate": "06/26/2023 16:31:47", "content": "I had too, I ended up correcting my features engineering process to add up the missing columns as dummy columns based on a list of columns that looks like what I want.
", "rawMarkdown": "I had too, I ended up correcting my features engineering process to add up the missing columns as dummy columns based on a list of columns that looks like what I want.", "votes": null }, { "id": "2319036", "postDate": "06/26/2023 19:14:44", "content": "any error should be caught with the try/except block and my code should work, that's what I don't understand
", "rawMarkdown": "any error should be caught with the try/except block and my code should work, that's what I don't understand", "votes": null }, { "id": "2319141", "postDate": "06/26/2023 23:06:39", "content": "Exactly, it makes it hard to debug !
", "rawMarkdown": "Exactly, it makes it hard to debug !", "votes": null }, { "id": "2320161", "postDate": "06/27/2023 14:33:32", "content": "@qurious Hi! It looks like your submissions are running out of memory. Just a note: this competition only provides 8GB of RAM, so there isn't a lot of extra room to work with.
", "rawMarkdown": "qurious Hi! It looks like your submissions are running out of memory. Just a note: this competition [only provides 8GB of RAM](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/overview/code-requirements), so there isn't a lot of extra room to work with.", "votes": null }, { "id": "2320173", "postDate": "06/27/2023 14:42:36", "content": "Hi @philculliton could you pleease tell me what's happening with my notebooks , they're all down after the update and can't find the error, i know nthe error is in the features engineering function but can't think of what could be the cause
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4860409%2F2c76c71638da859fb16ff0ac4ca6981c%2Fsubmission.png?generation=1687877167521382&alt=media\")
@philculliton Yes some iterations of my notebooks tend to go over memory, but the real black box issue is Scoring Error. Thank you for your time, I chose not to invest any other efforts in this competition despite the loads of time I already invested.
", "rawMarkdown": "philculliton Yes some iterations of my notebooks tend to go over memory, but the real black box issue is Scoring Error. Thank you for your time, I chose not to invest any other efforts in this competition despite the loads of time I already invested.", "votes": null }, { "id": "2320550", "postDate": "06/27/2023 21:40:35", "content": "@mchahhou Does a try except block always catch OOM errors? I think some OOM errors cannot be caught by Python try except block.
", "rawMarkdown": "mchahhou Does a try except block always catch OOM errors? I think some OOM errors cannot be caught by Python try except block.", "votes": null } ], "comments": [ { "id": 2315525, "author_name": "ravi20076", "author_url": "", "post_date": "06/24/2023 07:51:08", "content": "I suggest you could escalate these issues to Kaggle by writing this on the product feedback page. I am sure this will be addressed better there.
", "votes": null, "replies": [ { "id": 2315563, "author_name": "qurious", "author_url": "", "post_date": "06/24/2023 08:25:31", "content": "I appreciate your suggestion, I went ahead and contacted the hosts and a person from Kaggle stuff. I received an answer from the hosts, although they have dedicated some time to answer me, the proposals they gave me didn't really work sadly.
", "votes": null, "replies": [] } ] }, { "id": 2315821, "author_name": "ymatioun", "author_url": "", "post_date": "06/24/2023 12:57:01", "content": "i agree with you - i had a number of submissions that ran fine over open test set yet failed on hidden test set. In my opinion, this should NEVER be allowed to happen - open test set should be representative of the hidden test set, so that it can actually be used to debug your model. Otherwise, what is the point of it?
\nI think it is the responsibility of competition organizers to select open test set that fully represents the features of the hidden test set.
\nHopefully in future competitions organizers will do a better job selecting open test set.
", "votes": null, "replies": [ { "id": 2315828, "author_name": "qurious", "author_url": "", "post_date": "06/24/2023 13:00:39", "content": "I totally agree with you, unfortunately we can't afford to fully focus on our models only. On my end, although I made quiet a good model with great benchmarks I was never able to submit it. The only submission I was able to do was a copy paste from a public notebook. Hopefully this never happens again.
", "votes": null, "replies": [] }, { "id": 2315833, "author_name": "qurious", "author_url": "", "post_date": "06/24/2023 13:04:23", "content": "If you don't mind me asking, how do you submit successfully ? What is the trick ?
", "votes": null, "replies": [ { "id": 2316994, "author_name": "ymatioun", "author_url": "", "post_date": "06/25/2023 11:27:50", "content": "i simplified my submission code until it worked, then rebuilt it back one step at a time.
\nI still have no idea what was causing the problem - final working submission code looks equivalent to non-working submission code.
Hi @qurious:
\ndoes it fail quickly or after a few hours?
\ndo you have the same python version in your training and your submission notebook?
\ndo you sort test by index or elapsed_time?
\nis your mask carefully selecting the right question?
I agree that it is frustrating and I spent many, many hours guess-debugging the submission…
", "votes": null, "replies": [ { "id": 2315898, "author_name": "qurious", "author_url": "", "post_date": "06/24/2023 14:01:24", "content": "Hi Elias, it depends on each notebook since I tried various iterations. I applied my mask to select the right questions to answer for on some submits, I attempted to assemble my own answer dataframe to populate the submission variable. I've sorted by question, by session_id, unsorted, simple boolean mask. The only submission that worked was made using a public notebook, I attempted to adapt the notebook to my model but it still never worked. and yes I used the same python with the notebook I am import from.
", "votes": null, "replies": [] } ] }, { "id": 2316368, "author_name": "kononenko", "author_url": "", "post_date": "06/24/2023 20:38:42", "content": "If you click on the failed notebook and then go to \"Logs\", it may show the reason.
", "votes": null, "replies": [ { "id": 2316376, "author_name": "qurious", "author_url": "", "post_date": "06/24/2023 20:47:28", "content": "Scorring Error which is the vaguest error ever, print debug statements did not help either
", "votes": null, "replies": [] } ] } ] }, { "id": 2317445, "author_name": "mchahhou", "author_url": "", "post_date": "06/25/2023 17:58:48", "content": "I'm also stuck with those submission errors, I just lost 5 other submissions today.
\nEven with a code like this, I still get submission scoring error
\n (test, sam_sub) iter_test:\n sam_sub[] = \n sam_sub[] = sam_sub.session_id.apply( x : (x.split()[][:]) )\n sam_sub = sam_sub.sort_values()\n :\n do some FE\n do predictions\n :\n \n env.predict(sam_sub[[, ]])\nI don't understand what's going on, if some error happens, my code should work and submits only 0s.
\nBtw, running same code on the full training data without try/except doesn't throw any error
\nmy simulation on training code looks like this:
\n session_id, user_df tqdm( train_df.groupby(, sort=), total = train_df.session_id.nunique() ): \n level_group [, , ]:\n test = user_df.loc[user_df.level_group==level_group] \n test = test.sample(frac=).reset_index(drop=) \n do some FE\n do predictions\nHave you tried to avoid sorting sample submission by question number? This is just my guess, but may be the original order is somehow enforced during evaluation?
\nI’m using selection by mask to update the correct column and never had any issues with my submissions.
the sorting is part of my previous notebooks that did not throw the submission scoring error. So, the problem is not there.
\nThe error happens after 3 min of running the submission. Knowing that my notebook takes 8-9 min to finish, this means that 30% of the data has been predicted without errors
", "votes": null, "replies": [ { "id": 2317595, "author_name": "qurious", "author_url": "", "post_date": "06/25/2023 19:43:36", "content": "are you saying that the submission process results in a error the moment the running time goes over 3 minutes ?
", "votes": null, "replies": [ { "id": 2317602, "author_name": "mchahhou", "author_url": "", "post_date": "06/25/2023 19:49:10", "content": "yes, after 3 mins of execution, the error happens
", "votes": null, "replies": [ { "id": 2317683, "author_name": "qurious", "author_url": "", "post_date": "06/25/2023 21:32:35", "content": "This might be the issue, it takes a few minutes to process my features, was this announced anywhere ?
", "votes": null, "replies": [] } ] } ] }, { "id": 2318866, "author_name": "wouldyoujustfocus", "author_url": "", "post_date": "06/26/2023 16:23:34", "content": "@mchahhou i faced that too before the last annoying updates(in private testset and API), the problem was that there are missing levels in group 13_22, my code was assuming that all levels are available but they weren't which lead to the error after 3~4 min. now after the last rerun of LB all my submissions are turned to be erronous, still trying to figure it out.
", "votes": null, "replies": [ { "id": 2318876, "author_name": "qurious", "author_url": "", "post_date": "06/26/2023 16:31:47", "content": "I had too, I ended up correcting my features engineering process to add up the missing columns as dummy columns based on a list of columns that looks like what I want.
", "votes": null, "replies": [] }, { "id": 2319036, "author_name": "mchahhou", "author_url": "", "post_date": "06/26/2023 19:14:44", "content": "any error should be caught with the try/except block and my code should work, that's what I don't understand
", "votes": null, "replies": [ { "id": 2319141, "author_name": "qurious", "author_url": "", "post_date": "06/26/2023 23:06:39", "content": "Exactly, it makes it hard to debug !
", "votes": null, "replies": [] }, { "id": 2320550, "author_name": "cdeotte", "author_url": "", "post_date": "06/27/2023 21:40:35", "content": "@mchahhou Does a try except block always catch OOM errors? I think some OOM errors cannot be caught by Python try except block.
", "votes": null, "replies": [] } ] } ] } ] } ] } ] }, { "id": 2320161, "author_name": "philculliton", "author_url": "", "post_date": "06/27/2023 14:33:32", "content": "@qurious Hi! It looks like your submissions are running out of memory. Just a note: this competition only provides 8GB of RAM, so there isn't a lot of extra room to work with.
", "votes": null, "replies": [ { "id": 2320173, "author_name": "wouldyoujustfocus", "author_url": "", "post_date": "06/27/2023 14:42:36", "content": "Hi @philculliton could you pleease tell me what's happening with my notebooks , they're all down after the update and can't find the error, i know nthe error is in the features engineering function but can't think of what could be the cause
\n
@philculliton Yes some iterations of my notebooks tend to go over memory, but the real black box issue is Scoring Error. Thank you for your time, I chose not to invest any other efforts in this competition despite the loads of time I already invested.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2315331": "I am an active participant in this competition. I am writing to express my deep concern and dissatisfaction regarding the persistent issues we, as participants, are experiencing during the submission process.\n\nWe are dedicating substantial time and effort in crafting advanced, innovative models. However, the roadblock of a difficult and error-prone submission process has been a consistent pain point, creating an unnecessarily stressful and disheartening competition experience.\n\nA competition participant should, in principle, not have to invest an inordinate amount of effort and time into just being able to submit their work correctly. The time we invest in this competition is valuable, and it is deeply disconcerting to see it wasted on battling submission issues rather than improving our models and fostering learning.\n\nThe gravity of this issue is intensified considering the limited number of submission attempts we have each day. Every erroneous submission squanders a precious attempt and deprives us of the opportunity to test, learn, and enhance our models.\n\nThe cornerstone of such competitions should be the pursuit of innovation and improvement, not the navigation of submission intricacies. When we have to divert our energy away from model development and towards solving submission issues, the learning objective of this competition is not being met.\n\nWhile I do recognize and appreciate the hard work that goes into organizing a competition of this scale, it's essential that these long-standing issues are addressed with urgency. The experience and motivation of numerous participants, including myself, are being adversely affected.\n\nI kindly ask the organizing team to address these problems promptly, or at least provide clear communication about the actions being taken towards resolving these issues. Ensuring a smooth, user-friendly submission process is vital to let us focus on what truly matters - developing the best possible models and learning in the process. \n\nThank you for your immediate attention to this matter. I trust in your commitment to improve our experience and look forward to a swift and effective resolution of these issues.", "2315525": "I suggest you could escalate these issues to Kaggle by writing this on the **product feedback** page. I am sure this will be addressed better there.", "2315563": "I appreciate your suggestion, I went ahead and contacted the hosts and a person from Kaggle stuff. I received an answer from the hosts, although they have dedicated some time to answer me, the proposals they gave me didn't really work sadly.", "2315821": "i agree with you - i had a number of submissions that ran fine over open test set yet failed on hidden test set. In my opinion, this should NEVER be allowed to happen - open test set should be representative of the hidden test set, so that it can actually be used to debug your model. Otherwise, what is the point of it?\n\nI think it is the responsibility of competition organizers to select open test set that fully represents the features of the hidden test set.\n\nHopefully in future competitions organizers will do a better job selecting open test set.", "2315828": "I totally agree with you, unfortunately we can't afford to fully focus on our models only. On my end, although I made quiet a good model with great benchmarks I was never able to submit it. The only submission I was able to do was a copy paste from a public notebook. Hopefully this never happens again.", "2315831": "", "2315833": "If you don't mind me asking, how do you submit successfully ? What is the trick ?", "2315891": "Hi @qurious:\ndoes it fail quickly or after a few hours?\ndo you have the same python version in your training and your submission notebook?\ndo you sort test by index or elapsed_time?\nis your mask carefully selecting the right question?\n\nI agree that it is frustrating and I spent many, many hours guess-debugging the submission...", "2315898": "Hi Elias, it depends on each notebook since I tried various iterations. I applied my mask to select the right questions to answer for on some submits, I attempted to assemble my own answer dataframe to populate the submission variable. I've sorted by question, by session_id, unsorted, simple boolean mask. The only submission that worked was made using a public notebook, I attempted to adapt the notebook to my model but it still never worked. and yes I used the same python with the notebook I am import from.", "2316368": "If you click on the failed notebook and then go to \"Logs\", it may show the reason.", "2316376": "Scorring Error which is the vaguest error ever, print debug statements did not help either", "2316994": "i simplified my submission code until it worked, then rebuilt it back one step at a time.\nI still have no idea what was causing the problem - final working submission code looks equivalent to non-working submission code.", "2317445": "I'm also stuck with those submission errors, I just lost 5 other submissions today.\n\nEven with a code like this, I still get submission scoring error\n\n```python\nfor (test, sam_sub) in iter_test:\n sam_sub['correct'] = 0\n sam_sub['q'] = sam_sub.session_id.apply(lambda x : int(x.split('_')[1][1:]) )\n sam_sub = sam_sub.sort_values('q')\n try:\n do some FE\n do predictions\n except:\n pass\n env.predict(sam_sub[['session_id', 'correct']])\n```\n \nI don't understand what's going on, if some error happens, my code should work and submits only 0s.\n\nBtw, running same code on the full training data without try/except doesn't throw any error \n\nmy simulation on training code looks like this:\n\n```python\nfor session_id, user_df in tqdm( train_df.groupby('session_id', sort=False), total = train_df.session_id.nunique() ): \n for level_group in ['0-4', '5-12', '13-22']:\n test = user_df.loc[user_df.level_group==level_group] \n test = test.sample(frac=1.).reset_index(drop=True) # simulating the shuffle on LB\n do some FE\n do predictions\n```", "2317467": "Have you tried to avoid sorting sample submission by question number? This is just my guess, but may be the original order is somehow enforced during evaluation?\n\nI’m using [selection by mask](https://www.kaggle.com/code/kononenko/datatable-linearmodel-0-676lb-in-6-seconds#Predicting-and-submitting) to update the `correct` column and never had any issues with my submissions.", "2317497": "the sorting is part of my previous notebooks that did not throw the submission scoring error. So, the problem is not there.\n\nThe error happens after 3 min of running the submission. Knowing that my notebook takes 8-9 min to finish, this means that 30% of the data has been predicted without errors", "2317595": "are you saying that the submission process results in a error the moment the running time goes over 3 minutes ?", "2317602": "yes, after 3 mins of execution, the error happens", "2317683": "This might be the issue, it takes a few minutes to process my features, was this announced anywhere ?", "2318866": "mchahhou i faced that too before the last annoying updates(in private testset and API), the problem was that there are missing levels in group 13_22, my code was assuming that all levels are available but they weren't which lead to the error after 3~4 min. now after the last rerun of LB all my submissions are turned to be erronous, still trying to figure it out.", "2318876": "I had too, I ended up correcting my features engineering process to add up the missing columns as dummy columns based on a list of columns that looks like what I want.", "2319036": "any error should be caught with the try/except block and my code should work, that's what I don't understand", "2319141": "Exactly, it makes it hard to debug !", "2320161": "qurious Hi! It looks like your submissions are running out of memory. Just a note: this competition [only provides 8GB of RAM](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/overview/code-requirements), so there isn't a lot of extra room to work with.", "2320173": "Hi @philculliton could you pleease tell me what's happening with my notebooks , they're all down after the update and can't find the error, i know nthe error is in the features engineering function but can't think of what could be the cause\n\n", "2320482": "philculliton Yes some iterations of my notebooks tend to go over memory, but the real black box issue is Scoring Error. Thank you for your time, I chose not to invest any other efforts in this competition despite the loads of time I already invested.", "2320550": "mchahhou Does a try except block always catch OOM errors? I think some OOM errors cannot be caught by Python try except block." }, "source": "meta" }