Hi Kagglers,
\nWhen we create competitions for hosts, we try to ensure that competitors get to explore interesting problems. We perform due diligence checks to the extent feasible to create a competition with a secure format and data integrity. Unfortunately, in this case we had several things go wrong. Please know that we, along with the host, have invested a significant amount of time and thought in trying to find the right way forward at every turn, and we apologize for the issues that have emerged.
\nAs we first announced on March 20th, the data for this competition was made public. New data was collected, and we shared the full leaked data after we became aware of the original issue.
\nWhile we never want to extend a competition deadline, when we release new data, or when issues arise close to the deadline, we find it necessary for equity and fairness across all competitors to provide time to adjust to those changes. The new test data was updated on May 31st, 2023 and the reruns are complete for nearly everyone's notebooks. There are still a small number of notebooks with errors that we are continuing to investigate; those investigations may result in a (probably very small) number of score updates for the impacted notebook.
\nNot only did we make updates to the public and private data, but behind the scenes, we regularly make updates to the systems that are used to evaluate your submissions. This competition uses our time-series API, and over the last several months we made updates to the API that evaluates submissions. Several fixes were included in each update that was performed - most significantly, allowing the new 3.10 Kaggle VMs to work with the API, along with ordering and dataframe return issues that were reported during the course of this competition. These API bug fixes caused some notebooks to error out - specifically those with code that does not conform to the API updates. Thank you for the reports letting us know that you were seeing different scores, this led to further updates to the API.
\nTo account for the updates to the API, we’ve added 2 additional weeks to the competition. We know that Kagglers each have plans for the different stages of a competition, and we know that updates to the end date of a competition can be disruptive. We apologize for this disruption. We tried to balance the importance of maintaining stability in a competition with the need to provide the time you need to be successful.
\nOnce again, we’d like to reiterate that we hear your concerns and frustrations. The decisions we made in this competition were made in an effort to ensure an equal playing field for all competitors. We appreciate your patience and dedication in the face of these challenges, and we wish everyone luck as the competition concludes.
\nPhil
", "messages": [ { "id": "2291781", "postDate": "06/07/2023 20:13:07", "content": "Hi Kagglers,
\nWhen we create competitions for hosts, we try to ensure that competitors get to explore interesting problems. We perform due diligence checks to the extent feasible to create a competition with a secure format and data integrity. Unfortunately, in this case we had several things go wrong. Please know that we, along with the host, have invested a significant amount of time and thought in trying to find the right way forward at every turn, and we apologize for the issues that have emerged.
\nAs we first announced on March 20th, the data for this competition was made public. New data was collected, and we shared the full leaked data after we became aware of the original issue.
\nWhile we never want to extend a competition deadline, when we release new data, or when issues arise close to the deadline, we find it necessary for equity and fairness across all competitors to provide time to adjust to those changes. The new test data was updated on May 31st, 2023 and the reruns are complete for nearly everyone's notebooks. There are still a small number of notebooks with errors that we are continuing to investigate; those investigations may result in a (probably very small) number of score updates for the impacted notebook.
\nNot only did we make updates to the public and private data, but behind the scenes, we regularly make updates to the systems that are used to evaluate your submissions. This competition uses our time-series API, and over the last several months we made updates to the API that evaluates submissions. Several fixes were included in each update that was performed - most significantly, allowing the new 3.10 Kaggle VMs to work with the API, along with ordering and dataframe return issues that were reported during the course of this competition. These API bug fixes caused some notebooks to error out - specifically those with code that does not conform to the API updates. Thank you for the reports letting us know that you were seeing different scores, this led to further updates to the API.
\nTo account for the updates to the API, we’ve added 2 additional weeks to the competition. We know that Kagglers each have plans for the different stages of a competition, and we know that updates to the end date of a competition can be disruptive. We apologize for this disruption. We tried to balance the importance of maintaining stability in a competition with the need to provide the time you need to be successful.
\nOnce again, we’d like to reiterate that we hear your concerns and frustrations. The decisions we made in this competition were made in an effort to ensure an equal playing field for all competitors. We appreciate your patience and dedication in the face of these challenges, and we wish everyone luck as the competition concludes.
\nPhil
", "rawMarkdown": "Hi Kagglers,\n\nWhen we create competitions for hosts, we try to ensure that competitors get to explore interesting problems. We perform due diligence checks to the extent feasible to create a competition with a secure format and data integrity. Unfortunately, in this case we had several things go wrong. Please know that we, along with the host, have invested a significant amount of time and thought in trying to find the right way forward at every turn, and we apologize for the issues that have emerged. \n\nAs we first [announced on March 20th](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/396202), the data for this competition was made public. New data was collected, and we shared the full leaked data after we became aware of the original issue.\n\nWhile we never want to extend a competition deadline, when we release new data, or when issues arise close to the deadline, we find it necessary for equity and fairness across all competitors to provide time to adjust to those changes. The new test data was updated on May 31st, 2023 and the reruns are complete for nearly everyone's notebooks. There are still a small number of notebooks with errors that we are continuing to investigate; those investigations may result in a (probably very small) number of score updates for the impacted notebook.\n\nNot only did we make updates to the public and private data, but behind the scenes, we regularly make updates to the systems that are used to evaluate your submissions. This competition uses our time-series API, and over the last several months we made updates to the API that evaluates submissions. Several fixes were included in each update that was performed - most significantly, allowing the new 3.10 Kaggle VMs to work with the API, along with ordering and dataframe return issues that were reported during the course of this competition. These API bug fixes caused [some notebooks to error out](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/402608) - specifically those with code that does not conform to the API updates. Thank you for the reports letting us know that you were seeing different scores, this led to further updates to the API.\n\nTo account for the updates to the API, we’ve added 2 additional weeks to the competition. We know that Kagglers each have plans for the different stages of a competition, and we know that updates to the end date of a competition can be disruptive. We apologize for this disruption. We tried to balance the importance of maintaining stability in a competition with the need to provide the time you need to be successful. \n\nOnce again, we’d like to reiterate that we hear your concerns and frustrations. The decisions we made in this competition were made in an effort to ensure an equal playing field for all competitors. We appreciate your patience and dedication in the face of these challenges, and we wish everyone luck as the competition concludes.\n\nPhil", "votes": null }, { "id": "2291792", "postDate": "06/07/2023 20:20:22", "content": "As a tl:dr question, can the 3.10 API be used the same way we used 3.7 API earlier, without any changes?
\nIt'd be great if you could release another sample submission notebook from your end to keep things clear, as we don't want to waste more submissions just to test and debug the API
It would be really helpful if you would explain in detail what you changed in each API (3.7 and 3.10) with the latest updates and how it will look to our code. I went from 25th to 1000th in a day because of the changes.
", "rawMarkdown": "It would be really helpful if you would explain in detail what you changed in each API (3.7 and 3.10) with the latest updates and how it will look to our code. I went from 25th to 1000th in a day because of the changes.", "votes": null }, { "id": "2291877", "postDate": "06/07/2023 22:31:37", "content": "Also, you should increase the prizes
", "rawMarkdown": "Also, you should increase the prizes", "votes": null }, { "id": "2291918", "postDate": "06/08/2023 00:10:30", "content": "Hi @philculliton , thanks for the explanation and all the effort!
\nCould you please answer one question: what has been changed to the API in the latest update and rerun?
\nMany (if not all) of us saw a significant decrease in our public LB scores, and we guessed it's related to some sorting but could never confirm it. Could you please advice?
\nThanks again!
", "rawMarkdown": "Hi @philculliton , thanks for the explanation and all the effort!\n\nCould you please answer one question: **what has been changed to the API in the latest update and rerun?**\n\nMany (if not all) of us saw a significant decrease in our public LB scores, and we guessed it's related to some sorting but could never confirm it. Could you please advice?\n\nThanks again!", "votes": null }, { "id": "2291957", "postDate": "06/08/2023 01:30:41", "content": "With all due respect, this is not an explanation, this is a retelling of a story that we have all watched anyway
", "rawMarkdown": "With all due respect, this is not an explanation, this is a retelling of a story that we have all watched anyway", "votes": null }, { "id": "2292145", "postDate": "06/08/2023 04:46:00", "content": "@philculliton This is indeed the true concern. All explanations reported here are good-to-know, however, not useful. The true issue is still there. Many (including me) still don't understand in what the API has changed that make notebooks, which worked well before, now are doomed all.
", "rawMarkdown": "philculliton This is indeed the true concern. All explanations reported here are good-to-know, however, not useful. The true issue is still there. Many (including me) still don't understand in what the API has changed that make notebooks, which worked well before, now are doomed all.", "votes": null }, { "id": "2292810", "postDate": "06/08/2023 16:26:40", "content": "At the end it feels like none of the concerns, expressed by competitors within the last several weeks, were ever addressed and none of the questions were ever answered. Correct me if I’m wrong.
", "rawMarkdown": "At the end it feels like none of the concerns, expressed by competitors within the last several weeks, were ever addressed and none of the questions were ever answered. Correct me if I’m wrong.", "votes": null }, { "id": "2292891", "postDate": "06/08/2023 18:10:05", "content": "Can you also please tell us the reason, why the APIs are not passing the sequentially correct data, I believe most of the teams have been impacted by this update, so we deserve clarification regarding this. Also, I think there must be some way so that you correct this issue and provide us sequential data Otherwise hosts will get suboptimal solution at the end of the competition.
\nRegards
", "rawMarkdown": "Can you also please tell us the reason, why the APIs are not passing the sequentially correct data, I believe most of the teams have been impacted by this update, so we deserve clarification regarding this. Also, I think there must be some way so that you correct this issue and provide us sequential data Otherwise hosts will get **suboptimal solution** at the end of the competition.\n\nRegards", "votes": null }, { "id": "2293131", "postDate": "06/09/2023 01:05:18", "content": "Can you please explain why it constantly shows \"submission scoring error\"? This is happening to a lot of submissions.
", "rawMarkdown": "Can you please explain why it constantly shows \"submission scoring error\"? This is happening to a lot of submissions.", "votes": null }, { "id": "2300048", "postDate": "06/13/2023 02:28:35", "content": "Are we allowed to use a locally trained model ?
", "rawMarkdown": "Are we allowed to use a locally trained model ?", "votes": null } ], "comments": [ { "id": 2291792, "author_name": "pjmathematician", "author_url": "", "post_date": "06/07/2023 20:20:22", "content": "As a tl:dr question, can the 3.10 API be used the same way we used 3.7 API earlier, without any changes?
\nIt'd be great if you could release another sample submission notebook from your end to keep things clear, as we don't want to waste more submissions just to test and debug the API
It would be really helpful if you would explain in detail what you changed in each API (3.7 and 3.10) with the latest updates and how it will look to our code. I went from 25th to 1000th in a day because of the changes.
", "votes": null, "replies": [] }, { "id": 2291877, "author_name": "danielphalen", "author_url": "", "post_date": "06/07/2023 22:31:37", "content": "Also, you should increase the prizes
", "votes": null, "replies": [] }, { "id": 2291918, "author_name": "boscoyung", "author_url": "", "post_date": "06/08/2023 00:10:30", "content": "Hi @philculliton , thanks for the explanation and all the effort!
\nCould you please answer one question: what has been changed to the API in the latest update and rerun?
\nMany (if not all) of us saw a significant decrease in our public LB scores, and we guessed it's related to some sorting but could never confirm it. Could you please advice?
\nThanks again!
", "votes": null, "replies": [ { "id": 2292145, "author_name": "shinomoriaoshi", "author_url": "", "post_date": "06/08/2023 04:46:00", "content": "@philculliton This is indeed the true concern. All explanations reported here are good-to-know, however, not useful. The true issue is still there. Many (including me) still don't understand in what the API has changed that make notebooks, which worked well before, now are doomed all.
", "votes": null, "replies": [] } ] }, { "id": 2291957, "author_name": "kvlmll", "author_url": "", "post_date": "06/08/2023 01:30:41", "content": "With all due respect, this is not an explanation, this is a retelling of a story that we have all watched anyway
", "votes": null, "replies": [] }, { "id": 2292810, "author_name": "kononenko", "author_url": "", "post_date": "06/08/2023 16:26:40", "content": "At the end it feels like none of the concerns, expressed by competitors within the last several weeks, were ever addressed and none of the questions were ever answered. Correct me if I’m wrong.
", "votes": null, "replies": [] }, { "id": 2292891, "author_name": "chaudharypriyanshu", "author_url": "", "post_date": "06/08/2023 18:10:05", "content": "Can you also please tell us the reason, why the APIs are not passing the sequentially correct data, I believe most of the teams have been impacted by this update, so we deserve clarification regarding this. Also, I think there must be some way so that you correct this issue and provide us sequential data Otherwise hosts will get suboptimal solution at the end of the competition.
\nRegards
", "votes": null, "replies": [] }, { "id": 2293131, "author_name": "mstrmnd1", "author_url": "", "post_date": "06/09/2023 01:05:18", "content": "Can you please explain why it constantly shows \"submission scoring error\"? This is happening to a lot of submissions.
", "votes": null, "replies": [] }, { "id": 2300048, "author_name": "qurious", "author_url": "", "post_date": "06/13/2023 02:28:35", "content": "Are we allowed to use a locally trained model ?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2291781": "Hi Kagglers,\n\nWhen we create competitions for hosts, we try to ensure that competitors get to explore interesting problems. We perform due diligence checks to the extent feasible to create a competition with a secure format and data integrity. Unfortunately, in this case we had several things go wrong. Please know that we, along with the host, have invested a significant amount of time and thought in trying to find the right way forward at every turn, and we apologize for the issues that have emerged. \n\nAs we first [announced on March 20th](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/396202), the data for this competition was made public. New data was collected, and we shared the full leaked data after we became aware of the original issue.\n\nWhile we never want to extend a competition deadline, when we release new data, or when issues arise close to the deadline, we find it necessary for equity and fairness across all competitors to provide time to adjust to those changes. The new test data was updated on May 31st, 2023 and the reruns are complete for nearly everyone's notebooks. There are still a small number of notebooks with errors that we are continuing to investigate; those investigations may result in a (probably very small) number of score updates for the impacted notebook.\n\nNot only did we make updates to the public and private data, but behind the scenes, we regularly make updates to the systems that are used to evaluate your submissions. This competition uses our time-series API, and over the last several months we made updates to the API that evaluates submissions. Several fixes were included in each update that was performed - most significantly, allowing the new 3.10 Kaggle VMs to work with the API, along with ordering and dataframe return issues that were reported during the course of this competition. These API bug fixes caused [some notebooks to error out](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/402608) - specifically those with code that does not conform to the API updates. Thank you for the reports letting us know that you were seeing different scores, this led to further updates to the API.\n\nTo account for the updates to the API, we’ve added 2 additional weeks to the competition. We know that Kagglers each have plans for the different stages of a competition, and we know that updates to the end date of a competition can be disruptive. We apologize for this disruption. We tried to balance the importance of maintaining stability in a competition with the need to provide the time you need to be successful. \n\nOnce again, we’d like to reiterate that we hear your concerns and frustrations. The decisions we made in this competition were made in an effort to ensure an equal playing field for all competitors. We appreciate your patience and dedication in the face of these challenges, and we wish everyone luck as the competition concludes.\n\nPhil", "2291792": "As a tl:dr question, can the 3.10 API be used the same way we used 3.7 API earlier, without any changes?\nIt'd be great if you could release another sample submission notebook from your end to keep things clear, as we don't want to waste more submissions just to test and debug the API", "2291872": "It would be really helpful if you would explain in detail what you changed in each API (3.7 and 3.10) with the latest updates and how it will look to our code. I went from 25th to 1000th in a day because of the changes.", "2291877": "Also, you should increase the prizes", "2291918": "Hi @philculliton , thanks for the explanation and all the effort!\n\nCould you please answer one question: **what has been changed to the API in the latest update and rerun?**\n\nMany (if not all) of us saw a significant decrease in our public LB scores, and we guessed it's related to some sorting but could never confirm it. Could you please advice?\n\nThanks again!", "2291957": "With all due respect, this is not an explanation, this is a retelling of a story that we have all watched anyway", "2292145": "philculliton This is indeed the true concern. All explanations reported here are good-to-know, however, not useful. The true issue is still there. Many (including me) still don't understand in what the API has changed that make notebooks, which worked well before, now are doomed all.", "2292810": "At the end it feels like none of the concerns, expressed by competitors within the last several weeks, were ever addressed and none of the questions were ever answered. Correct me if I’m wrong.", "2292891": "Can you also please tell us the reason, why the APIs are not passing the sequentially correct data, I believe most of the teams have been impacted by this update, so we deserve clarification regarding this. Also, I think there must be some way so that you correct this issue and provide us sequential data Otherwise hosts will get **suboptimal solution** at the end of the competition.\n\nRegards", "2293131": "Can you please explain why it constantly shows \"submission scoring error\"? This is happening to a lot of submissions.", "2300048": "Are we allowed to use a locally trained model ?" }, "source": "meta" }