After a few days of competition I identified the open data of the game. After confirming by a submission that these data composed the LB I immediately informed the host and Kaggle. As I have already experimented such a situation (https://www.kaggle.com/competitions/ashrae-energy-prediction/discussion/118047) I judged that it was fairer and less disruptive to not make that leak public to let the host find a solution.
\nI must say the host is kind, want to do well and had tried his best to find a good solution to ensure fairness (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/396202).
\nMeanwhile @cdeotte found that the level_group were not in the right order (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/388479). The resolution of that bug led to a breaking change (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/400313) that is a mysterious requirement. If anyone understand what technically justifies an order change please explain.
\nAfter the 1st data update I found that all the leaked data had not been removed from the LB. This is the 2nd data leak that explains why there had been another change of the LB scores (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/407887).
\nThen for no reason it has been chosen to introduce a new version of the API for Python 3.10 (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/409898) with a wrong explanation of how it works. This is a risky choice to change the version of an API in production without testing. We can thank @rsakata for his brilliant debug (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/412512). The order of the data and of the questions in the sample_submission are shuffled compared to the dataset and the previous version of the API that was inline with the dataset.
\nIt has then been chosen that this bugged API had to be repercuted to the Python 3.7 version (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/414109). The workaround for me consists of 2 new code blocks that may be duplicated in our solutions instead of centralized in the API, which is inefficient and less performant since it requires useless instructions and to change the original order of the data.
\n test, sample_submission iter_test:\n test = test.sort_values().reset_index(drop=)\n\n sample_submission[] = [(label.split()[][:]) label sample_submission[]]\n sample_submission = sample_submission.sort_values().reset_index(drop=)\n\n preds = [...] \n\n sample_submission[] = * (np.array(preds) > THRESHOLD)\n\n env.predict(sample_submission[[, ]])\nAll this shows that Kaggle did not test the API neither automatically nor manually. If (automatic) tests exist, @philculliton, you likely just have to publish them so that the API can be understood.
\nSo Kagglers, you can admit that this is perfectly clear that, as they can't (do not want to?) resolve the bugs they introduced successively nor rollback to the original Python 3.7 API, they \"need to provide the time (we) need to be successful\"…
", "messages": [ { "id": 2292320, "postDate": "2023-06-08T08:08:14.607Z", "content": "After a few days of competition I identified the open data of the game. After confirming by a submission that these data composed the LB I immediately informed the host and Kaggle. As I have already experimented such a situation (https://www.kaggle.com/competitions/ashrae-energy-prediction/discussion/118047) I judged that it was fairer and less disruptive to not make that leak public to let the host find a solution.
\nI must say the host is kind, want to do well and had tried his best to find a good solution to ensure fairness (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/396202).
\nMeanwhile @cdeotte found that the level_group were not in the right order (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/388479). The resolution of that bug led to a breaking change (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/400313) that is a mysterious requirement. If anyone understand what technically justifies an order change please explain.
\nAfter the 1st data update I found that all the leaked data had not been removed from the LB. This is the 2nd data leak that explains why there had been another change of the LB scores (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/407887).
\nThen for no reason it has been chosen to introduce a new version of the API for Python 3.10 (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/409898) with a wrong explanation of how it works. This is a risky choice to change the version of an API in production without testing. We can thank @rsakata for his brilliant debug (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/412512). The order of the data and of the questions in the sample_submission are shuffled compared to the dataset and the previous version of the API that was inline with the dataset.
\nIt has then been chosen that this bugged API had to be repercuted to the Python 3.7 version (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/414109). The workaround for me consists of 2 new code blocks that may be duplicated in our solutions instead of centralized in the API, which is inefficient and less performant since it requires useless instructions and to change the original order of the data.
\n test, sample_submission iter_test:\n test = test.sort_values().reset_index(drop=)\n\n sample_submission[] = [(label.split()[][:]) label sample_submission[]]\n sample_submission = sample_submission.sort_values().reset_index(drop=)\n\n preds = [...] \n\n sample_submission[] = * (np.array(preds) > THRESHOLD)\n\n env.predict(sample_submission[[, ]])\nAll this shows that Kaggle did not test the API neither automatically nor manually. If (automatic) tests exist, @philculliton, you likely just have to publish them so that the API can be understood.
\nSo Kagglers, you can admit that this is perfectly clear that, as they can't (do not want to?) resolve the bugs they introduced successively nor rollback to the original Python 3.7 API, they \"need to provide the time (we) need to be successful\"…
", "rawMarkdown": "After a few days of competition I identified the open data of the game. After confirming by a submission that these data composed the LB I immediately informed the host and Kaggle. As I have already experimented such a situation (https://www.kaggle.com/competitions/ashrae-energy-prediction/discussion/118047) I judged that it was fairer and less disruptive to not make that leak public to let the host find a solution.\n\nI must say the host is kind, want to do well and had tried his best to find a good solution to ensure fairness (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/396202).\n\nMeanwhile @cdeotte found that the level_group were not in the right order (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/388479). The resolution of that bug led to a breaking change (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/400313) that is a mysterious requirement. If anyone understand what technically justifies an order change please explain.\n\nAfter the 1st data update I found that all the leaked data had not been removed from the LB. This is the 2nd data leak that explains why there had been another change of the LB scores (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/407887).\n\nThen for no reason it has been chosen to introduce a new version of the API for Python 3.10 (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/409898) with a wrong explanation of how it works. This is a risky choice to change the version of an API in production without testing. We can thank @rsakata for his brilliant debug (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/412512). The order of the data and of the questions in the sample_submission are shuffled compared to the dataset and the previous version of the API that was inline with the dataset.\n\nIt has then been chosen that this bugged API had to be repercuted to the Python 3.7 version (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/414109). The workaround for me consists of 2 new code blocks that may be duplicated in our solutions instead of centralized in the API, which is inefficient and less performant since it requires useless instructions and to change the original order of the data.\n\n```python\nfor test, sample_submission in iter_test:\n test = test.sort_values('index').reset_index(drop=True)\n\n sample_submission['question'] = [int(label.split('_')[1][1:]) for label in sample_submission['session_id']]\n sample_submission = sample_submission.sort_values('question').reset_index(drop=True)\n \n preds = [...] # a list of the predictions for the ordered questions of the level_group\n \n sample_submission['correct'] = 1 * (np.array(preds) > THRESHOLD)\n \n env.predict(sample_submission[['session_id', 'correct']])\n```\n\nAll this shows that Kaggle did not test the API neither automatically nor manually. If (automatic) tests exist, @philculliton, you likely just have to publish them so that the API can be understood.\n\nSo Kagglers, you can admit that this is perfectly clear that, as they can't (do not want to?) resolve the bugs they introduced successively nor rollback to the original Python 3.7 API, they \"need to provide the time (we) need to be successful\"...", "votes": 27 }, { "id": 2293192, "postDate": "2023-06-09T02:41:54.223Z", "content": "Thanks @pdnartreb , this is a far more clear explanation.
\nFrom the explanation of @philculliton , maybe the key point is this:
\n\n\nWe resolved this difference when we updated both APIs for the new private LB, but this required competitors to figure out how to move from the old API to the new one.
\n
The hosts think that figuring out the difference between 2 APIs is part of the competition. So they keep the changes of API as a secret.
\nBut I guess most of the competitors (including me), will consider this as a waste of time.
\nWe hope focus on the model, not guessing what has been changed in the API.
Hi Bertrand!
\nThanks, and thank you especially for your help in identifying the leaks earlier in the competition. It was much appreciated by the Kaggle team and the host.
\nThe 3.10 API was added because Kaggle as a whole upgraded all VMs to run Python 3.10. Quite simply, without the 3.10 API no new notebooks would have been able to submit to the competition. We were required to build and add the 3.10 API. We cannot roll back to the old 3.7 API, as it would no longer work for most notebooks.
\nThe timeseries API includes extensive automatic unit tests. We do not share the underlying code of the API. As with all unit tests, a) they don't cover everything and b) we add unit tests when new issues are discovered. New fixes, and new unit tests, were added between the start of the competition and the introduction of the 3.10 API. The difference between the two APIs is the result of one of the changes made during that time.
\nWhen we added the 3.10 API, we decided not to update the 3.7 API for stability reasons. This was, I think, a mistake, as that would have resolved the differences, and both APIs would have behaved the same. I apologize; we were attempting to optimize for stability, but instead introduced a difference. We resolved this difference when we updated both APIs for the new private LB, but this required competitors to figure out how to move from the old API to the new one.
\nWe appreciate people sharing their solutions to this, but based on private messaging, the forums, and other sources of feedback, we saw that a large number of people still needed time to sort this out, which is why we extended the competition. As mentioned in the explanation we know this is disruptive and we do apologize for that, but as the same post mentioned, this is in an effort to ensure a level playing field for all competitors.
", "rawMarkdown": "Hi Bertrand!\n\nThanks, and thank you especially for your help in identifying the leaks earlier in the competition. It was much appreciated by the Kaggle team and the host.\n\nThe 3.10 API was added because Kaggle as a whole upgraded all VMs to run Python 3.10. Quite simply, without the 3.10 API **no new notebooks would have been able to submit to the competition**. We were required to build and add the 3.10 API. We cannot roll back to the old 3.7 API, as it would no longer work for most notebooks.\n\nThe timeseries API includes extensive automatic unit tests. We do not share the underlying code of the API. As with all unit tests, a) they don't cover everything and b) we add unit tests when new issues are discovered. New fixes, and new unit tests, were added between the start of the competition and the introduction of the 3.10 API. The difference between the two APIs is the result of one of the changes made during that time.\n\nWhen we added the 3.10 API, we decided not to update the 3.7 API for stability reasons. This was, I think, a mistake, as that would have resolved the differences, and both APIs would have behaved the same. I apologize; we were attempting to optimize for stability, but instead introduced a difference. We resolved this difference when we updated both APIs for the new private LB, but this required competitors to figure out how to move from the old API to the new one.\n\nWe appreciate people sharing their solutions to this, but based on private messaging, the forums, and other sources of feedback, we saw that a large number of people still needed time to sort this out, which is why we extended the competition. As mentioned in the [explanation](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/415737) we know this is disruptive and we do apologize for that, but as the same post mentioned, this is in an effort to ensure a level playing field for all competitors.", "votes": 1, "replies": [ { "id": 2292783, "postDate": "2023-06-08T16:12:41.730Z", "content": "Hi @philculliton!
\nThanks for your reply.
\nI, probably like many others, just struggle to understand why this is difficult to sort the data or maybe to not shuffle it. You could/should have fixed the bug instead of extending this hard experience.
\nI am sure that the host and you tried to do well but you have made mistakes. Extension does not ensure fairness. It advantages the ones that will be able to work beyond June 14th. This won't be my case. I am deeply disappointed because I think to the sacrificies I made: hundreds of hours (I do not want to count), inform you of a major leak that I could have used for me (I would be free since nearly a month), the time I did not give to my children, …
\nBeyond your words I understand that you are sorry and as I feel for you I won't say or think more.
\nYou are forgiven.
", "rawMarkdown": "Hi @philculliton!\n\nThanks for your reply.\n\nI, probably like many others, just struggle to understand why this is difficult to sort the data or maybe to not shuffle it. You could/should have fixed the bug instead of extending this hard experience.\n\nI am sure that the host and you tried to do well but you have made mistakes. Extension does not ensure fairness. It advantages the ones that will be able to work beyond June 14th. This won't be my case. I am deeply disappointed because I think to the sacrificies I made: hundreds of hours (I do not want to count), inform you of a major leak that I could have used for me (I would be free since nearly a month), the time I did not give to my children, ...\n\nBeyond your words I understand that you are sorry and as I feel for you I won't say or think more.\n\nYou are forgiven.", "votes": 12, "replies": [ { "id": 2293176, "postDate": "2023-06-09T02:09:47.827Z", "content": "Thanks @pdnartreb for the summary!
\nI also understand that @philculliton and the Kaggle team are doing the best.
\nHowever, I feel uncomfortable about this sentence:
\n\n\nwe saw that a large number of people still needed time to sort this out
\n
Why the participants are the one who need to \"sort this out\" because of a bug in the API?
\nAnd even now, we actually still don't see a confirmation that the sorting idea was the correct fix.
", "rawMarkdown": "Thanks @pdnartreb for the summary!\n\nI also understand that @philculliton and the Kaggle team are doing the best.\n\nHowever, I feel uncomfortable about this sentence:\n> we saw that a large number of people still needed time to sort this out\n\nWhy the participants are the one who need to \"sort this out\" because of a bug in the API?\n\nAnd even now, we actually still don't see a **confirmation** that the sorting idea was the correct fix.", "votes": 3 }, { "id": 2293406, "postDate": "2023-06-09T06:44:47.223Z", "rawMarkdown": "", "isDeleted": true } ] } ] } ], "comments": [ { "id": 2293192, "author_name": "huoyeqianxun", "author_url": "", "post_date": "2023-06-09T02:41:54.223000", "content": "Thanks @pdnartreb , this is a far more clear explanation.
\nFrom the explanation of @philculliton , maybe the key point is this:
\n\n\nWe resolved this difference when we updated both APIs for the new private LB, but this required competitors to figure out how to move from the old API to the new one.
\n
The hosts think that figuring out the difference between 2 APIs is part of the competition. So they keep the changes of API as a secret.
\nBut I guess most of the competitors (including me), will consider this as a waste of time.
\nWe hope focus on the model, not guessing what has been changed in the API.
Hi Bertrand!
\nThanks, and thank you especially for your help in identifying the leaks earlier in the competition. It was much appreciated by the Kaggle team and the host.
\nThe 3.10 API was added because Kaggle as a whole upgraded all VMs to run Python 3.10. Quite simply, without the 3.10 API no new notebooks would have been able to submit to the competition. We were required to build and add the 3.10 API. We cannot roll back to the old 3.7 API, as it would no longer work for most notebooks.
\nThe timeseries API includes extensive automatic unit tests. We do not share the underlying code of the API. As with all unit tests, a) they don't cover everything and b) we add unit tests when new issues are discovered. New fixes, and new unit tests, were added between the start of the competition and the introduction of the 3.10 API. The difference between the two APIs is the result of one of the changes made during that time.
\nWhen we added the 3.10 API, we decided not to update the 3.7 API for stability reasons. This was, I think, a mistake, as that would have resolved the differences, and both APIs would have behaved the same. I apologize; we were attempting to optimize for stability, but instead introduced a difference. We resolved this difference when we updated both APIs for the new private LB, but this required competitors to figure out how to move from the old API to the new one.
\nWe appreciate people sharing their solutions to this, but based on private messaging, the forums, and other sources of feedback, we saw that a large number of people still needed time to sort this out, which is why we extended the competition. As mentioned in the explanation we know this is disruptive and we do apologize for that, but as the same post mentioned, this is in an effort to ensure a level playing field for all competitors.
", "votes": 1, "replies": [ { "id": 2292783, "author_name": "Bertrand P", "author_url": "", "post_date": "2023-06-08T16:12:41.730000", "content": "Hi @philculliton!
\nThanks for your reply.
\nI, probably like many others, just struggle to understand why this is difficult to sort the data or maybe to not shuffle it. You could/should have fixed the bug instead of extending this hard experience.
\nI am sure that the host and you tried to do well but you have made mistakes. Extension does not ensure fairness. It advantages the ones that will be able to work beyond June 14th. This won't be my case. I am deeply disappointed because I think to the sacrificies I made: hundreds of hours (I do not want to count), inform you of a major leak that I could have used for me (I would be free since nearly a month), the time I did not give to my children, …
\nBeyond your words I understand that you are sorry and as I feel for you I won't say or think more.
\nYou are forgiven.
", "votes": 12, "replies": [ { "id": 2293176, "author_name": "Bosco Yung", "author_url": "", "post_date": "2023-06-09T02:09:47.827000", "content": "Thanks @pdnartreb for the summary!
\nI also understand that @philculliton and the Kaggle team are doing the best.
\nHowever, I feel uncomfortable about this sentence:
\n\n\nwe saw that a large number of people still needed time to sort this out
\n
Why the participants are the one who need to \"sort this out\" because of a bug in the API?
\nAnd even now, we actually still don't see a confirmation that the sorting idea was the correct fix.
", "votes": 3, "replies": [] }, { "id": 2293406, "author_name": "", "author_url": "", "post_date": "2023-06-09T06:44:47.223000", "content": "", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2292320": "After a few days of competition I identified the open data of the game. After confirming by a submission that these data composed the LB I immediately informed the host and Kaggle. As I have already experimented such a situation (https://www.kaggle.com/competitions/ashrae-energy-prediction/discussion/118047) I judged that it was fairer and less disruptive to not make that leak public to let the host find a solution.\n\nI must say the host is kind, want to do well and had tried his best to find a good solution to ensure fairness (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/396202).\n\nMeanwhile @cdeotte found that the level_group were not in the right order (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/388479). The resolution of that bug led to a breaking change (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/400313) that is a mysterious requirement. If anyone understand what technically justifies an order change please explain.\n\nAfter the 1st data update I found that all the leaked data had not been removed from the LB. This is the 2nd data leak that explains why there had been another change of the LB scores (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/407887).\n\nThen for no reason it has been chosen to introduce a new version of the API for Python 3.10 (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/409898) with a wrong explanation of how it works. This is a risky choice to change the version of an API in production without testing. We can thank @rsakata for his brilliant debug (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/412512). The order of the data and of the questions in the sample_submission are shuffled compared to the dataset and the previous version of the API that was inline with the dataset.\n\nIt has then been chosen that this bugged API had to be repercuted to the Python 3.7 version (https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/414109). The workaround for me consists of 2 new code blocks that may be duplicated in our solutions instead of centralized in the API, which is inefficient and less performant since it requires useless instructions and to change the original order of the data.\n\n```python\nfor test, sample_submission in iter_test:\n test = test.sort_values('index').reset_index(drop=True)\n\n sample_submission['question'] = [int(label.split('_')[1][1:]) for label in sample_submission['session_id']]\n sample_submission = sample_submission.sort_values('question').reset_index(drop=True)\n \n preds = [...] # a list of the predictions for the ordered questions of the level_group\n \n sample_submission['correct'] = 1 * (np.array(preds) > THRESHOLD)\n \n env.predict(sample_submission[['session_id', 'correct']])\n```\n\nAll this shows that Kaggle did not test the API neither automatically nor manually. If (automatic) tests exist, @philculliton, you likely just have to publish them so that the API can be understood.\n\nSo Kagglers, you can admit that this is perfectly clear that, as they can't (do not want to?) resolve the bugs they introduced successively nor rollback to the original Python 3.7 API, they \"need to provide the time (we) need to be successful\"...", "2293192": "Thanks @pdnartreb , this is a far more clear explanation.\n\nFrom the explanation of @philculliton , maybe the key point is this:\n>We resolved this difference when we updated both APIs for the new private LB, but this required competitors to figure out how to move from the old API to the new one.\n \nThe hosts think that figuring out the difference between 2 APIs is part of the competition. So they keep the changes of API as a secret.\n\nBut I guess most of the competitors (including me), will consider this as a waste of time.\nWe hope focus on the model, not guessing what has been changed in the API.\n", "2292536": "Hi Bertrand!\n\nThanks, and thank you especially for your help in identifying the leaks earlier in the competition. It was much appreciated by the Kaggle team and the host.\n\nThe 3.10 API was added because Kaggle as a whole upgraded all VMs to run Python 3.10. Quite simply, without the 3.10 API **no new notebooks would have been able to submit to the competition**. We were required to build and add the 3.10 API. We cannot roll back to the old 3.7 API, as it would no longer work for most notebooks.\n\nThe timeseries API includes extensive automatic unit tests. We do not share the underlying code of the API. As with all unit tests, a) they don't cover everything and b) we add unit tests when new issues are discovered. New fixes, and new unit tests, were added between the start of the competition and the introduction of the 3.10 API. The difference between the two APIs is the result of one of the changes made during that time.\n\nWhen we added the 3.10 API, we decided not to update the 3.7 API for stability reasons. This was, I think, a mistake, as that would have resolved the differences, and both APIs would have behaved the same. I apologize; we were attempting to optimize for stability, but instead introduced a difference. We resolved this difference when we updated both APIs for the new private LB, but this required competitors to figure out how to move from the old API to the new one.\n\nWe appreciate people sharing their solutions to this, but based on private messaging, the forums, and other sources of feedback, we saw that a large number of people still needed time to sort this out, which is why we extended the competition. As mentioned in the [explanation](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/415737) we know this is disruptive and we do apologize for that, but as the same post mentioned, this is in an effort to ensure a level playing field for all competitors." } }