\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14268627%2F28631f6758837a3fdb32e7a04e6c1ebc%2Fsssss.png?generation=1686712351053280&alt=media\")
Hi all -
\nAs the host mentioned in this post made when the original leak was acknowledged, we are going to incorporate a new private dataset into the competition that has been gathered over the months since the leak. This will involve rerunning your notebooks on the new private data.
\nAs with last time, this could take quite some time, but we expect far fewer errors this time around, which should hopefully lead to a much shorter turnaround (roughly, this week if possible) as there should be much less to check and investigate.
\nWe have also investigated recent reports from the forums about differences between the original and newer 3.10 versions of the API. This data update required a complete rebuild and replacement of both APIs and should eliminate any differences between the two.
\nShortly, we will freeze submissions while we update the APIs, private data, and scoring mechanisms. We will resume submissions once the update is complete, and then begin re-running notebooks.
", "messages": [ { "id": 2286587, "postDate": "2023-06-03T15:11:50.127Z", "content": "Hi @philculliton, I made some submissions today including some to check the validity of the new API, but all of them failed (some notebooks which were successfully run before, now returns the submission scoring error). Then, could we have an official announcement from Kaggle about,
\nHow did the API change in detail, then how to make a proper submission?
Whether or not the deadline is extended, not a month but only to compensate for the time that the API has been invalid (as we couldn't submit our work), maybe a week or several days?
Thank you, @philculliton
\nP/s: I think we don't get frustrated because of the change regarding some technical issues, what makes people confused and maybe angry is why Kaggle keeps silent when we only need some simple clarifications.
\nUPDATE: While I am posting this post, my rank drops from 28th to 95th as my 0.703 notebooks become invalid.
", "rawMarkdown": "Hi @philculliton, I made some submissions today including some to check the validity of the new API, but **all of them failed** (some notebooks which **were successfully run before, now returns the submission scoring error**). Then, could we have an official announcement from Kaggle about,\n\n1. How did the API change in detail, then how to make a proper submission?\n\n2. Whether or not the deadline is extended, not a month but only to compensate for the time that the API has been invalid (as we couldn't submit our work), maybe a week or several days?\n\nThank you, @philculliton \n\nP/s: I think **we don't get frustrated because of the change regarding some technical issues, what makes people confused and maybe angry is why Kaggle keeps silent** when we only need some simple clarifications.\n\nUPDATE: While I am posting this post, my rank drops from 28th to 95th as my 0.703 notebooks become invalid.", "votes": 24, "replies": [ { "id": 2286613, "postDate": "2023-06-03T15:25:54.883Z", "content": "\"what makes people confused and maybe angry is why Kaggle keeps silent when we only need some simple clarifications\"👍👍👍
", "rawMarkdown": "\"what makes people confused and maybe angry is why Kaggle keeps silent when we only need some simple clarifications\"👍👍👍", "votes": 5 }, { "id": 2286644, "postDate": "2023-06-03T15:56:25.583Z", "content": "I completely agree with the statement about keeping silence. The fact that rerun is started means (for me at least) that @philculliton just does not care about this competition. People have been complaining about new api for several days. We explained several times why rerun is an awful idea due to incompatibility of API versions. But Kaggle host just does not care lol. Seems like they haven't even read this topic.
\nif i see Phil hosting a new competition, it is unlikely that i will participate in it.
@philculliton, thank you for your action (extending the deadline), could you please provide more info about how the API works? Whether it gives 1 level 1 session at a time, or many sessions, many levels, etc.? It would be even better if you could check my notebooks, they constantly get the scoring error. Thank you!
", "rawMarkdown": "@philculliton, thank you for your action (extending the deadline), could you please provide more info about how the API works? Whether it gives 1 level 1 session at a time, or many sessions, many levels, etc.? It would be even better if you could check my notebooks, they constantly get the scoring error. Thank you!" } ] }, { "id": 2283127, "postDate": "2023-06-01T04:43:06.537Z", "content": "I would like to request that the size of the test data not be changed significantly in a code competition where there is an upper limit to the inference time.
\nIf the test data is suddenly changed without prior announcement, as in this case, the work done up to this point may have been in vain.
Did the inference time change after updating the test data?
", "rawMarkdown": "Did the inference time change after updating the test data?", "votes": 1, "replies": [ { "id": 2283253, "postDate": "2023-06-01T06:38:52.710Z", "content": "As for what I submitted today, the inference time did not change.
\nHowever, I don't know if the test data has been updated at this time.
Thank you for your response. I understand.
", "rawMarkdown": "Thank you for your response. I understand.", "votes": 1 } ] } ] }, { "id": 2284890, "postDate": "2023-06-02T10:29:29.483Z", "content": "we cannot estimate inference time for the new dataset. In notebook competition, That factor is very important.
", "rawMarkdown": "1. Could you tell us how much data will give us new and now?\n2. When will you update to new dataset?\n\nwe cannot estimate inference time for the new dataset. In notebook competition, That factor is very important.", "votes": 15 }, { "id": 2282932, "postDate": "2023-05-31T23:50:48.683Z", "content": "I submitted and got LB 0.699 yesterday,
\nAfter 15 hours, I submitted same code, I got LB 0.556…
\nProbably many of you who submit with the old API are facing this problem…
\nIt is an unstable competition.😓
Yes mine also from 0.704 came down to 0.665 😑
", "rawMarkdown": "Yes mine also from 0.704 came down to 0.665 😑", "votes": 1, "replies": [ { "id": 2282954, "postDate": "2023-06-01T01:03:53.833Z", "content": "If they rerun the LB in this state, definitely chaos ensues.
", "rawMarkdown": "If they rerun the LB in this state, definitely chaos ensues." }, { "id": 2282963, "postDate": "2023-06-01T01:16:39.197Z", "content": "I was still using pre 3.10 api , hopefully all balances out if everyone seeing the same scale down.
", "rawMarkdown": "I was still using pre 3.10 api , hopefully all balances out if everyone seeing the same scale down." }, { "id": 2282973, "postDate": "2023-06-01T01:28:37.747Z", "content": "From Phil’s reply, it seems to be that only API have changed, nothing have happened to the public LB dataset.
", "rawMarkdown": "From Phil’s reply, it seems to be that only API have changed, nothing have happened to the public LB dataset. " }, { "id": 2282999, "postDate": "2023-06-01T02:25:36.747Z", "content": "Wait score drops now even with 3.10 api?
", "rawMarkdown": "Wait score drops now even with 3.10 api? " } ] }, { "id": 2282976, "postDate": "2023-06-01T01:38:04.627Z", "content": "I am really used to it now😂. Last rerun, the score of exactly same codes and same models changes from 704 to 702. They told me dataset and api are not changed. We got no explanation even many other people have reported similar cases.
\nThis time my scores change from 702 to 676 with only api changes.
", "rawMarkdown": "I am really used to it now😂. Last rerun, the score of exactly same codes and same models changes from 704 to 702. They told me dataset and api are not changed. We got no explanation even many other people have reported similar cases. \n\nThis time my scores change from 702 to 676 with only api changes.", "votes": 4, "replies": [ { "id": 2283176, "postDate": "2023-06-01T05:31:20.267Z", "content": "Updates: sorting by elapsed_time, give me the same 702. But sort by index, give me the exception. quite weird!!
", "rawMarkdown": "Updates: sorting by elapsed_time, give me the same 702. But sort by index, give me the exception. quite weird!!", "votes": 1 } ] }, { "id": 2285265, "postDate": "2023-06-02T15:19:46.083Z", "content": "\n\nI submitted and got LB 0.699 yesterday,
\n
\n After 15 hours, I submitted same code, I got LB 0.556…
\n Probably many of you who submit with the old API are facing this problem…
\n It is an unstable competition.😓
I was wondering why my same code got different results until I saw this post….🙄
", "rawMarkdown": "> I submitted and got LB 0.699 yesterday, \n> After 15 hours, I submitted **same code**, I got LB 0.556…\n> Probably many of you who submit with the old API are facing this problem...\n> It is an unstable competition.😓\n\nI was wondering why my same code got different results until I saw this post....🙄", "votes": 3 }, { "id": 2285779, "postDate": "2023-06-03T02:01:08.777Z", "content": "Have you solved this problem? If it's resolved, please teach me
", "rawMarkdown": "Have you solved this problem? If it's resolved, please teach me" } ] }, { "id": 2291348, "postDate": "2023-06-07T13:57:05.513Z", "content": "This is not data centric or model centric competition.
\nThis is an API centric competition 🤣🤣
As the host mentioned in this post made when the original leak was acknowledged, we are going to incorporate a new private dataset into the competition that has been gathered over the months since the leak.
I would say most people only expected the first rerun, not the second one, especially not when there are only 14 days left before the deadline(also it seems the rerun is not going to happen immediately but 'roughly, this week if possible' ).
\nThis data update required a complete rebuild and replacement of both APIs and should eliminate any differences between the two.
If the newer new API work as intended, all things are good. But it took several days for people to realize that the 3.10 API has a different mechanic than the old API, and there was never any official explanation before this one.
\nAlso, correct me if I remember it wrong. Last time when the API was updated at the beginning of May, the 1400 team was eliminated to 800 due to the older API no longer working. Should we expect the same and manually rerun our notebook this time?
\nWe will resume submissions once the update is complete, and then begin re-running notebooks.
We are already seeing different voices about whether there should be another ddl extension. On one side people are worrying about the compatibility of their model with new datasets and API, on the other side it's already quite an exhausting experience for 4 months of a single competition yet now I can't help to think if it's ever going to end.
", "rawMarkdown": "`As the host mentioned in this post made when the original leak was acknowledged, we are going to incorporate a new private dataset into the competition that has been gathered over the months since the leak. `\n\nI would say most people only expected the first rerun, not the second one, especially not when there are only 14 days left before the deadline(also it seems the rerun is not going to happen immediately but 'roughly, this week if possible' ).\n\n`This data update required a complete rebuild and replacement of both APIs and should eliminate any differences between the two.`\n \nIf the newer new API work as intended, all things are good. But it took several days for people to realize that the 3.10 API has a different mechanic than the old API, and there was never any official explanation before this one.\n\nAlso, correct me if I remember it wrong. Last time when the API was updated at the beginning of May, the 1400 team was eliminated to 800 due to the older API no longer working. Should we expect the same and manually rerun our notebook this time? \n\n`We will resume submissions once the update is complete, and then begin re-running notebooks.`\n\nWe are already seeing different voices about whether there should be another ddl extension. On one side people are worrying about the compatibility of their model with new datasets and API, on the other side it's already quite an exhausting experience for 4 months of a single competition yet now I can't help to think if it's ever going to end.", "votes": 11 }, { "id": 2284220, "postDate": "2023-06-01T20:08:18.817Z", "content": "\"This competition will run indefinitely with a rolling API which invalidates entries after each month\" 😄
", "rawMarkdown": "\"This competition will run indefinitely with a rolling API which invalidates entries after each month\" 😄 ", "votes": 12 }, { "id": 2283860, "postDate": "2023-06-01T14:44:54.220Z", "content": "To make the situation a little bit more clear, kaggle could post a schedule (date/what exactly is going to be changed) for the planned updates.
\nOtherwise, people have to waste their time by trying to distinguish changes in the computational environment from effects caused by their model enhancements.
", "rawMarkdown": "To make the situation a little bit more clear, kaggle could post a schedule (date/what exactly is going to be changed) for the planned updates. \n\nOtherwise, people have to waste their time by trying to distinguish changes in the computational environment from effects caused by their model enhancements.", "votes": 9 }, { "id": 2284833, "postDate": "2023-06-02T09:52:31.530Z", "content": "So, i have one more question for organizers. What is the point of changing old api?
\nFirstly, in the new api test set is not sorted by indexes (it seems so, but no information has been received from organizers), and we have to waste time on a useless operation. Secondly, changing the api will lead to loss of many submissions, since api versions are not compatible
", "rawMarkdown": "So, i have one more question for organizers. What is the point of changing old api? \n\nFirstly, in the new api test set is not sorted by indexes (it seems so, but no information has been received from organizers), and we have to waste time on a useless operation. Secondly, changing the api will lead to loss of many submissions, since api versions are not compatible", "votes": 7, "replies": [ { "id": 2284887, "postDate": "2023-06-02T10:26:02.170Z", "content": "Because the new API does not have a built-in sort mechanic, so instead of giving the new API function it should have, they decide to delete the function from the old API to make it as broken as the new one.
\nIt's OK if they decide to do it in the middle of the competition, as they already did earlier in April since people have enough time to rerun their older submission with an additional dl extension. But it's NOT OK to do it at the end of the competition with both the new API and dataset. There is no way I manually rerun my over 200 submissions with the old API and dataset within 13 days left.
", "rawMarkdown": "Because the new API does not have a built-in sort mechanic, so instead of giving the new API function it should have, they decide to delete the function from the old API to make it as broken as the new one.\n\nIt's OK if they decide to do it in the middle of the competition, as they already did earlier in April since people have enough time to rerun their older submission with an additional dl extension. But it's NOT OK to do it at the end of the competition with both the new API and dataset. There is no way I manually rerun my over 200 submissions with the old API and dataset within 13 days left.", "votes": 4 } ] }, { "id": 2284393, "postDate": "2023-06-02T02:11:56.820Z", "content": "So i am still not able to reproduce my results with the same models even taking into account index sorting and the (dis)order of qids. Any tips @philculliton ? Or explanations why should we rebuild our pipelines spending subs at the end of competition?
", "rawMarkdown": "So i am still not able to reproduce my results with the same models even taking into account index sorting and the (dis)order of qids. Any tips @philculliton ? Or explanations why should we rebuild our pipelines spending subs at the end of competition?", "votes": 7, "replies": [ { "id": 2286066, "postDate": "2023-06-03T07:42:18.713Z", "content": "I have also encountered this situation.
", "rawMarkdown": "I have also encountered this situation." }, { "id": 2286194, "postDate": "2023-06-03T09:42:43.277Z", "content": "Same here, for me LB drops by 0.001-0.004 in different subs
", "rawMarkdown": "Same here, for me LB drops by 0.001-0.004 in different subs" } ] }, { "id": 2286850, "postDate": "2023-06-03T20:50:54.723Z", "content": "@philculliton \"Don't panic! I'm taking a break because it's the weekend. 😊👍\"
", "rawMarkdown": "@philculliton \"Don't panic! I'm taking a break because it's the weekend. 😊👍\"", "votes": 8, "replies": [ { "id": 2287025, "postDate": "2023-06-04T04:16:53.580Z", "content": "laughing 😃
", "rawMarkdown": "laughing 😃", "votes": 1, "replies": [ { "id": 2287048, "postDate": "2023-06-04T05:05:03.700Z", "content": "Need Help
\nhttps://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/414914
\nI got an issue when trying to submit my code to Kaggle with an error message stating \"Notebook Threw Exception.\"
", "rawMarkdown": "Need Help\n\nhttps://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/414914\n\nI got an issue when trying to submit my code to Kaggle with an error message stating \"Notebook Threw Exception.\" ", "votes": -3 } ] } ] }, { "id": 2283678, "postDate": "2023-06-01T12:35:30.700Z", "content": "I have never entered a competition that had confusing changes for users 14 days before it ended.
\nMost of what I've been trying to improve my LB score has been wasted.
\nAs other users have pointed out, I think it should be extended.
\nHowever, repeated extensions are discouraging me😩
No one more extension PLZ😷
", "rawMarkdown": "No one more extension PLZ😷", "votes": 6 }, { "id": 2308429, "postDate": "2023-06-18T23:26:40.333Z", "content": "Hi @philculliton ,
\nthe LB update be implemented after all? Or is it already done? It's already 10 days ago. Please let me know the situation.
I select one of mine old submission as final submission, however, with the new API running, it went wrong with the hint \"Submission Scoring Error\". But I can not deselect it. It has never happened in a normal competition!
\nHope Kaggle team can fix this problem!
\n
Why is the order of the events per session not being maintained since it plays a crucial role? sorting it by index can lead to errors since there are some ~1% sessions(mentioned in this post: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/395250) where the index is not increasing by one sequentially.
", "rawMarkdown": "Why is the order of the events per session not being maintained since it plays a crucial role? sorting it by index can lead to errors since there are some ~1% sessions(mentioned in this post: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/395250) where the index is not increasing by one sequentially.\n\n", "votes": 5, "replies": [ { "id": 2286436, "postDate": "2023-06-03T13:04:12.123Z", "content": "Oh, so this is why I cant replicate my old result exactly!
", "rawMarkdown": "Oh, so this is why I cant replicate my old result exactly!", "votes": 1 } ] }, { "id": 2283076, "postDate": "2023-06-01T03:46:23.810Z", "content": "Why do things always go wrong. Pheasant platform?
", "rawMarkdown": "Why do things always go wrong. Pheasant platform?", "votes": 4 }, { "id": 2285042, "postDate": "2023-06-02T12:16:52.650Z", "content": "no more extension please…
", "rawMarkdown": "no more extension please...", "votes": 3 }, { "id": 2282393, "postDate": "2023-05-31T14:53:58.903Z", "content": "\n\nThis data update required a complete rebuild and replacement of both APIs and should eliminate any differences between the two.
\n
Does it mean there are changes to the old API as well? I guess we should also expect a difference in the public lb score as well.
\nIs it possible to release the API in Python code format (apart from the .so file), so that we know exactly what data (format, order) will be in the inference loop? I think it is still very confusing to most of us how does the test loop work.
Thank you!
", "rawMarkdown": "> This data update required a complete rebuild and replacement of both APIs and should eliminate any differences between the two.\n\nDoes it mean there are changes to the old API as well? I guess we should also expect a difference in the public lb score as well. \n\nIs it possible to release the API in Python code format (apart from the `.so` file), so that we know exactly what data (format, order) will be in the inference loop? I think it is still very confusing to most of us how does the test loop work.\n\nThank you!", "votes": 3 }, { "id": 2291229, "postDate": "2023-06-07T12:25:58.317Z", "content": "For the trouble you should increase the prizes.
", "rawMarkdown": "For the trouble you should increase the prizes.", "votes": 2 }, { "id": 2286710, "postDate": "2023-06-03T17:21:48.607Z", "content": "Due to some problems, please extend the competition
", "rawMarkdown": "Due to some problems, please extend the competition", "votes": 2 }, { "id": 2283421, "postDate": "2023-06-01T08:39:15.263Z", "content": "Instead of spending submissions to test hypotheses, i have to spend submissions to replicate my best result 😈
\nI think many people use old api for submitting and after changing it and restarting notebooks, we will simply lose our old submissions. Because restarting my old notebooks without any change gives me something like 0.65.
\nAnd judging by the experience of last rerun and number of angry comments in this topic, I think that LB rerun will definitely not take 1-2 days
\nI don’t understand what is the difficulty in presenting the code or a description of the api so that we don’t spend submissions on debugging code…
\nMeanwhile the competition should end in 14 days…
", "rawMarkdown": "Instead of spending submissions to test hypotheses, i have to spend submissions to replicate my best result 😈 \n\nI think many people use old api for submitting and after changing it and restarting notebooks, we will simply lose our old submissions. Because restarting my old notebooks without any change gives me something like 0.65. \n\nAnd judging by the experience of last rerun and number of angry comments in this topic, I think that LB rerun will definitely not take 1-2 days\n\nI don’t understand what is the difficulty in presenting the code or a description of the api so that we don’t spend submissions on debugging code...\n\nMeanwhile the competition should end in 14 days...", "votes": 2 }, { "id": 2283190, "postDate": "2023-06-01T05:44:34.813Z", "content": "You wasted a day of my time and five submissions
", "rawMarkdown": "You wasted a day of my time and five submissions", "votes": 3 }, { "id": 2284336, "postDate": "2023-06-02T00:22:37.423Z", "content": "Is the API still being updated? The three consecutive submissions just now reported errorsIs the API still being updated? The three consecutive submissions just now reported errors？
", "rawMarkdown": "Is the API still being updated? The three consecutive submissions just now reported errorsIs the API still being updated? The three consecutive submissions just now reported errors？", "votes": 1, "replies": [ { "id": 2284436, "postDate": "2023-06-02T03:12:54.673Z", "content": "I submitted it two hours ago and still haven't reported an errorI submitted it two hours ago and still haven't reported an error
", "rawMarkdown": "I submitted it two hours ago and still haven't reported an errorI submitted it two hours ago and still haven't reported an error", "replies": [ { "id": 2284493, "postDate": "2023-06-02T05:23:42.257Z", "content": "it reported error when the inference is finished
", "rawMarkdown": "it reported error when the inference is finished" }, { "id": 2284547, "postDate": "2023-06-02T06:23:40.100Z", "content": "I have a similar case here. Yesterday I submitted 2 notebooks at the same time. One succeed, the other failed. Today I resubmit the failed one again, it scores successfully. So I guess the api is somewhat unstable.
", "rawMarkdown": "I have a similar case here. Yesterday I submitted 2 notebooks at the same time. One succeed, the other failed. Today I resubmit the failed one again, it scores successfully. So I guess the api is somewhat unstable.", "votes": 1 } ] } ] }, { "id": 2283043, "postDate": "2023-06-01T03:13:49.750Z", "content": "After API changed, I submitted the same notebook. LB decreased from 0.705 to 0.662.
", "rawMarkdown": "After API changed, I submitted the same notebook. LB decreased from 0.705 to 0.662.", "votes": 1, "replies": [ { "id": 2283052, "postDate": "2023-06-01T03:21:42.340Z", "content": "Sorting test dataframe by elapsed_time, LB returned from 0.662 to 0.704.
", "rawMarkdown": "Sorting test dataframe by elapsed_time, LB returned from 0.662 to 0.704.", "votes": 3, "replies": [ { "id": 2283061, "postDate": "2023-06-01T03:25:08.383Z", "content": "Does sorting by index return to your 0.705?
", "rawMarkdown": "Does sorting by index return to your 0.705?" }, { "id": 2283138, "postDate": "2023-06-01T04:54:54.263Z", "content": "Yes. Sorting by index, LB returned to 0.705.
", "rawMarkdown": "Yes. Sorting by index, LB returned to 0.705." }, { "id": 2283166, "postDate": "2023-06-01T05:17:43.793Z", "content": "sorting by elapsed_time, give me the same 702. But sort by index, give me the exception. WTF😑
", "rawMarkdown": "sorting by elapsed_time, give me the same 702. But sort by index, give me the exception. WTF😑" }, { "id": 2283175, "postDate": "2023-06-01T05:30:25.590Z", "content": "sorted by index, still got low score
\nread here about the question numbers order, took into account but still got -0.001 from the original result 👀
Thx for this.. confused sorting by index helping or elapsed_time, might as well as experiment with both .
", "rawMarkdown": "Thx for this.. confused sorting by index helping or elapsed_time, might as well as experiment with both ." } ] } ] }, { "id": 2282593, "postDate": "2023-05-31T17:10:57.127Z", "content": "Does this mean only the private dataset will be changed, and public LB will stay the same?
", "rawMarkdown": "Does this mean only the private dataset will be changed, and public LB will stay the same?", "votes": 1 }, { "id": 2282228, "postDate": "2023-05-31T13:16:56.210Z", "content": "I failed to submit this afternoon (submission score error). I suspect that changes are made to the test set during my submission process, and then I submitted the same version of the notebook tonight. I found that the score had changed, and the presenter's statement now confirms this.
", "rawMarkdown": "I failed to submit this afternoon (submission score error). I suspect that changes are made to the test set during my submission process, and then I submitted the same version of the notebook tonight. I found that the score had changed, and the presenter's statement now confirms this.", "votes": 1, "replies": [ { "id": 2282596, "postDate": "2023-05-31T17:12:30.803Z", "content": "Wait, so the public LB dataset has changed AGAIN??
", "rawMarkdown": "Wait, so the public LB dataset has changed AGAIN??", "replies": [ { "id": 2282911, "postDate": "2023-05-31T22:33:34.387Z", "content": "Yes，It has changed.
", "rawMarkdown": "Yes，It has changed." }, { "id": 2282917, "postDate": "2023-05-31T22:53:52.600Z", "content": "The public API has been updated. The test set itself still has all the same samples. If your submission was impacted by the difference between the different Python version APIs then the update may change your score.
", "rawMarkdown": "The public API has been updated. The test set itself still has all the same samples. If your submission was impacted by the difference between the different Python version APIs then the update may change your score.", "votes": -5 }, { "id": 2282947, "postDate": "2023-06-01T00:45:08.100Z", "content": "So, let me understand the situation. You have just changed old API too (without any announcement about how it works now, or I just haven't seen it? doubt).
\nNew version of old API is not compatible with the previous version, and I have spent 4 submissions and a day searching bug in my code to realize it.
\nNow you will rerun all submissions which used old API with the new one. All scores will drop dramatically then. It equals removing competitors from LB.
\nMoreover, you are doing it two weeks before the end.
\nFor me it looks like a fatal combo of strange decisions
", "rawMarkdown": "So, let me understand the situation. You have just changed old API too (without any announcement about how it works now, or I just haven't seen it? doubt).\n\nNew version of old API is not compatible with the previous version, and I have spent 4 submissions and a day searching bug in my code to realize it.\n\nNow you will rerun all submissions which used old API with the new one. All scores will drop dramatically then. It equals removing competitors from LB.\n\nMoreover, you are doing it two weeks before the end.\n\nFor me it looks like a fatal combo of *strange* decisions", "votes": 3 }, { "id": 2282951, "postDate": "2023-06-01T00:51:55.807Z", "content": "I just don't get it. Is it a problem to make new version compatible with the previous one? As I understand there is some problem with sorting. Why can't you do it built-in?
", "rawMarkdown": "I just don't get it. Is it a problem to make new version compatible with the previous one? As I understand there is some problem with sorting. Why can't you do it built-in?", "votes": 3 }, { "id": 2282953, "postDate": "2023-06-01T00:59:03.763Z", "content": "Soooo, instead of making new API works, they made the old API broken? This is really mind boogling, what have lead to this decision???
", "rawMarkdown": "Soooo, instead of making new API works, they made the old API broken? This is really mind boogling, what have lead to this decision???", "votes": 4 }, { "id": 2283902, "postDate": "2023-06-01T15:11:25.347Z", "content": "@philculliton this would be nice to have an *official *statement on whether there is a bug or not with the old/new API version. There's large differences between scores with the new version of the API, and many people points towards a potential issue with a sort() somewhere in the code.
", "rawMarkdown": "@philculliton this would be nice to have an *official *statement on whether there is a bug or not with the old/new API version. There's large differences between scores with the new version of the API, and many people points towards a potential issue with a **sort()** somewhere in the code." } ] } ] }, { "id": 2282230, "postDate": "2023-05-31T13:18:23.790Z", "content": "any ddl extension？
", "rawMarkdown": "any ddl extension？", "votes": 2 }, { "id": 2282930, "postDate": "2023-05-31T23:46:33.887Z", "content": "can we extend one more month? It takes time to get adapted to the new api.
", "rawMarkdown": "can we extend one more month? It takes time to get adapted to the new api." }, { "id": 2282183, "postDate": "2023-05-31T12:51:06.890Z", "content": "Hi all -
\nAs the host mentioned in this post made when the original leak was acknowledged, we are going to incorporate a new private dataset into the competition that has been gathered over the months since the leak. This will involve rerunning your notebooks on the new private data.
\nAs with last time, this could take quite some time, but we expect far fewer errors this time around, which should hopefully lead to a much shorter turnaround (roughly, this week if possible) as there should be much less to check and investigate.
\nWe have also investigated recent reports from the forums about differences between the original and newer 3.10 versions of the API. This data update required a complete rebuild and replacement of both APIs and should eliminate any differences between the two.
\nShortly, we will freeze submissions while we update the APIs, private data, and scoring mechanisms. We will resume submissions once the update is complete, and then begin re-running notebooks.
", "rawMarkdown": "Hi all -\n\nAs the host mentioned in [this post](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/396202) made when the original leak was acknowledged, we are going to incorporate a new private dataset into the competition that has been gathered over the months since the leak. This will involve rerunning your notebooks on the new private data.\n\nAs with last time, this could take quite some time, but we expect far fewer errors this time around, which should hopefully lead to a much shorter turnaround (roughly, this week if possible) as there should be much less to check and investigate.\n\nWe have also investigated recent reports from the forums about differences between the original and newer 3.10 versions of the API. This data update required a complete rebuild and replacement of both APIs and should eliminate any differences between the two.\n\nShortly, we will freeze submissions while we update the APIs, private data, and scoring mechanisms. We will resume submissions once the update is complete, and then begin re-running notebooks.", "votes": -88 }, { "id": 2285865, "postDate": "2023-06-03T04:09:09.830Z", "content": "HI post the rescoring and LB update will the competition time be extended again ?
\nOut of 5 submissions today, one was 'submission scoring error'
“What makes people confused and maybe angry is why Kaggle keeps silent when we only need some simple clarifications”. What a thoughtful sentence.
", "rawMarkdown": "“What makes people confused and maybe angry is why Kaggle keeps silent when we only need some simple clarifications”. What a thoughtful sentence." }, { "id": 2312518, "postDate": "2023-06-22T01:39:21.460Z", "content": "I just don't get it, why is it so hard to submit in this competition ? I have already burned 20 submissions just trying to submit my own predictions, printing the submission.csv all looks fine, this is so frustrating after spending 60 hours in the last 12 days setting up a model. Is the submission system down for now ? am I the only one experience this ?
", "rawMarkdown": "I just don't get it, why is it so hard to submit in this competition ? I have already burned 20 submissions just trying to submit my own predictions, printing the submission.csv all looks fine, this is so frustrating after spending 60 hours in the last 12 days setting up a model. Is the submission system down for now ? am I the only one experience this ?" }, { "id": 2292261, "postDate": "2023-06-08T06:53:49.880Z", "content": "There is one select submission which is wrong after new run, but I cannot cancel it. Therefore I waste one submission chance.
", "rawMarkdown": "There is one select submission which is wrong after new run, but I cannot cancel it. Therefore I waste one submission chance. " }, { "id": 2289283, "postDate": "2023-06-06T04:10:33.703Z", "content": "I hadn't read this discussion thread, now I understand why I was having lower LB, now I don't even know which of my submissions is performing better, I'm a newbie but I can't believe it
", "rawMarkdown": "I hadn't read this discussion thread, now I understand why I was having lower LB, now I don't even know which of my submissions is performing better, I'm a newbie but I can't believe it " }, { "id": 2286974, "postDate": "2023-06-04T03:00:58.243Z", "content": "All my submission become \"Submissions Scoring Error\", is it normal?
", "rawMarkdown": "All my submission become \"Submissions Scoring Error\", is it normal?", "replies": [ { "id": 2286980, "postDate": "2023-06-04T03:07:14.373Z", "content": "Me too.
\nWe all have been experiencing such phenomenon since yesterday.
Thanks for your sharing.
", "rawMarkdown": "Thanks for your sharing.", "votes": 1 } ] } ] }, { "id": 2283546, "postDate": "2023-06-01T10:38:16.390Z", "content": "Sorting by index doesn't really exactly replicate my result 0.701 -> 0.701, however sorting by public score gives different order. What is going on exactly…
", "rawMarkdown": "Sorting by index doesn't really exactly replicate my result 0.701 -> 0.701, however sorting by public score gives different order. What is going on exactly..." }, { "id": 2283373, "postDate": "2023-06-01T08:06:42.423Z", "content": "My score is also decreasing after re-run without any change in code and models and using old API, from 0.698 to 0.571…
\nBut high scores in old LB are still high scores in new test dataset.
Wasn't the LB update previously completed?
\nDid anyone know that there are two levels of updates?
It was.
\nNobody knew 🙃
14 days to go btw :)
", "rawMarkdown": "14 days to go btw :) " }, { "id": 2282494, "postDate": "2023-05-31T15:58:35.217Z", "content": "Will it fix 'Notebook threw exception' error ?
", "rawMarkdown": "Will it fix 'Notebook threw exception' error ?\n" }, { "id": 2282364, "postDate": "2023-05-31T14:39:54.377Z", "content": "HI post the rescoring and LB update will the competition time be extended again ?
", "rawMarkdown": "HI post the rescoring and LB update will the competition time be extended again ?" }, { "id": 2282206, "postDate": "2023-05-31T13:04:10.320Z", "content": "Hi! PHIL CULLITON.
\nSo when will you resume submissions and will this competition extend again? I'm not sure that I have enough time to update my models.
Hi @philculliton, I made some submissions today including some to check the validity of the new API, but all of them failed (some notebooks which were successfully run before, now returns the submission scoring error). Then, could we have an official announcement from Kaggle about,
\nHow did the API change in detail, then how to make a proper submission?
Whether or not the deadline is extended, not a month but only to compensate for the time that the API has been invalid (as we couldn't submit our work), maybe a week or several days?
Thank you, @philculliton
\nP/s: I think we don't get frustrated because of the change regarding some technical issues, what makes people confused and maybe angry is why Kaggle keeps silent when we only need some simple clarifications.
\nUPDATE: While I am posting this post, my rank drops from 28th to 95th as my 0.703 notebooks become invalid.
", "votes": 24, "replies": [ { "id": 2286613, "author_name": "老肥", "author_url": "", "post_date": "2023-06-03T15:25:54.883000", "content": "\"what makes people confused and maybe angry is why Kaggle keeps silent when we only need some simple clarifications\"👍👍👍
", "votes": 5, "replies": [] }, { "id": 2286644, "author_name": "empty", "author_url": "", "post_date": "2023-06-03T15:56:25.583000", "content": "I completely agree with the statement about keeping silence. The fact that rerun is started means (for me at least) that @philculliton just does not care about this competition. People have been complaining about new api for several days. We explained several times why rerun is an awful idea due to incompatibility of API versions. But Kaggle host just does not care lol. Seems like they haven't even read this topic.
\nif i see Phil hosting a new competition, it is unlikely that i will participate in it.
@philculliton, thank you for your action (extending the deadline), could you please provide more info about how the API works? Whether it gives 1 level 1 session at a time, or many sessions, many levels, etc.? It would be even better if you could check my notebooks, they constantly get the scoring error. Thank you!
", "votes": 0, "replies": [] } ] }, { "id": 2283127, "author_name": "Takoi", "author_url": "", "post_date": "2023-06-01T04:43:06.537000", "content": "I would like to request that the size of the test data not be changed significantly in a code competition where there is an upper limit to the inference time.
\nIf the test data is suddenly changed without prior announcement, as in this case, the work done up to this point may have been in vain.
Did the inference time change after updating the test data?
", "votes": 1, "replies": [ { "id": 2283253, "author_name": "Takoi", "author_url": "", "post_date": "2023-06-01T06:38:52.710000", "content": "As for what I submitted today, the inference time did not change.
\nHowever, I don't know if the test data has been updated at this time.
Thank you for your response. I understand.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2284890, "author_name": "tereka", "author_url": "", "post_date": "2023-06-02T10:29:29.483000", "content": "we cannot estimate inference time for the new dataset. In notebook competition, That factor is very important.
", "votes": 15, "replies": [] }, { "id": 2282932, "author_name": "Makotu", "author_url": "", "post_date": "2023-05-31T23:50:48.683000", "content": "I submitted and got LB 0.699 yesterday,
\nAfter 15 hours, I submitted same code, I got LB 0.556…
\nProbably many of you who submit with the old API are facing this problem…
\nIt is an unstable competition.😓
Yes mine also from 0.704 came down to 0.665 😑
", "votes": 1, "replies": [ { "id": 2282954, "author_name": "Makotu", "author_url": "", "post_date": "2023-06-01T01:03:53.833000", "content": "If they rerun the LB in this state, definitely chaos ensues.
", "votes": 0, "replies": [] }, { "id": 2282963, "author_name": "Gaurav Rawat", "author_url": "", "post_date": "2023-06-01T01:16:39.197000", "content": "I was still using pre 3.10 api , hopefully all balances out if everyone seeing the same scale down.
", "votes": 0, "replies": [] }, { "id": 2282973, "author_name": "Woprime", "author_url": "", "post_date": "2023-06-01T01:28:37.747000", "content": "From Phil’s reply, it seems to be that only API have changed, nothing have happened to the public LB dataset.
", "votes": 0, "replies": [] }, { "id": 2282999, "author_name": "empty", "author_url": "", "post_date": "2023-06-01T02:25:36.747000", "content": "Wait score drops now even with 3.10 api?
", "votes": 0, "replies": [] } ] }, { "id": 2282976, "author_name": "ADAM.", "author_url": "", "post_date": "2023-06-01T01:38:04.627000", "content": "I am really used to it now😂. Last rerun, the score of exactly same codes and same models changes from 704 to 702. They told me dataset and api are not changed. We got no explanation even many other people have reported similar cases.
\nThis time my scores change from 702 to 676 with only api changes.
", "votes": 4, "replies": [ { "id": 2283176, "author_name": "ADAM.", "author_url": "", "post_date": "2023-06-01T05:31:20.267000", "content": "Updates: sorting by elapsed_time, give me the same 702. But sort by index, give me the exception. quite weird!!
", "votes": 1, "replies": [] } ] }, { "id": 2285265, "author_name": "Xiao-Su (Frank) Hu", "author_url": "", "post_date": "2023-06-02T15:19:46.083000", "content": "\n\nI submitted and got LB 0.699 yesterday,
\n
\n After 15 hours, I submitted same code, I got LB 0.556…
\n Probably many of you who submit with the old API are facing this problem…
\n It is an unstable competition.😓
I was wondering why my same code got different results until I saw this post….🙄
", "votes": 3, "replies": [] }, { "id": 2285779, "author_name": "Pusheen5314", "author_url": "", "post_date": "2023-06-03T02:01:08.777000", "content": "Have you solved this problem? If it's resolved, please teach me
", "votes": 0, "replies": [] } ] }, { "id": 2291348, "author_name": "Data Science Dances", "author_url": "", "post_date": "2023-06-07T13:57:05.513000", "content": "This is not data centric or model centric competition.
\nThis is an API centric competition 🤣🤣
As the host mentioned in this post made when the original leak was acknowledged, we are going to incorporate a new private dataset into the competition that has been gathered over the months since the leak.
I would say most people only expected the first rerun, not the second one, especially not when there are only 14 days left before the deadline(also it seems the rerun is not going to happen immediately but 'roughly, this week if possible' ).
\nThis data update required a complete rebuild and replacement of both APIs and should eliminate any differences between the two.
If the newer new API work as intended, all things are good. But it took several days for people to realize that the 3.10 API has a different mechanic than the old API, and there was never any official explanation before this one.
\nAlso, correct me if I remember it wrong. Last time when the API was updated at the beginning of May, the 1400 team was eliminated to 800 due to the older API no longer working. Should we expect the same and manually rerun our notebook this time?
\nWe will resume submissions once the update is complete, and then begin re-running notebooks.
We are already seeing different voices about whether there should be another ddl extension. On one side people are worrying about the compatibility of their model with new datasets and API, on the other side it's already quite an exhausting experience for 4 months of a single competition yet now I can't help to think if it's ever going to end.
", "votes": 11, "replies": [] }, { "id": 2284220, "author_name": "steubk", "author_url": "", "post_date": "2023-06-01T20:08:18.817000", "content": "\"This competition will run indefinitely with a rolling API which invalidates entries after each month\" 😄
", "votes": 12, "replies": [] }, { "id": 2283860, "author_name": "Oleksiy Kononenko", "author_url": "", "post_date": "2023-06-01T14:44:54.220000", "content": "To make the situation a little bit more clear, kaggle could post a schedule (date/what exactly is going to be changed) for the planned updates.
\nOtherwise, people have to waste their time by trying to distinguish changes in the computational environment from effects caused by their model enhancements.
", "votes": 9, "replies": [] }, { "id": 2284833, "author_name": "superslon", "author_url": "", "post_date": "2023-06-02T09:52:31.530000", "content": "So, i have one more question for organizers. What is the point of changing old api?
\nFirstly, in the new api test set is not sorted by indexes (it seems so, but no information has been received from organizers), and we have to waste time on a useless operation. Secondly, changing the api will lead to loss of many submissions, since api versions are not compatible
", "votes": 7, "replies": [ { "id": 2284887, "author_name": "Ya Xu", "author_url": "", "post_date": "2023-06-02T10:26:02.170000", "content": "Because the new API does not have a built-in sort mechanic, so instead of giving the new API function it should have, they decide to delete the function from the old API to make it as broken as the new one.
\nIt's OK if they decide to do it in the middle of the competition, as they already did earlier in April since people have enough time to rerun their older submission with an additional dl extension. But it's NOT OK to do it at the end of the competition with both the new API and dataset. There is no way I manually rerun my over 200 submissions with the old API and dataset within 13 days left.
", "votes": 4, "replies": [] } ] }, { "id": 2284393, "author_name": "empty", "author_url": "", "post_date": "2023-06-02T02:11:56.820000", "content": "So i am still not able to reproduce my results with the same models even taking into account index sorting and the (dis)order of qids. Any tips @philculliton ? Or explanations why should we rebuild our pipelines spending subs at the end of competition?
", "votes": 7, "replies": [ { "id": 2286066, "author_name": "cjz", "author_url": "", "post_date": "2023-06-03T07:42:18.713000", "content": "I have also encountered this situation.
", "votes": 0, "replies": [] }, { "id": 2286194, "author_name": "Joel Erikanders", "author_url": "", "post_date": "2023-06-03T09:42:43.277000", "content": "Same here, for me LB drops by 0.001-0.004 in different subs
", "votes": 0, "replies": [] } ] }, { "id": 2286850, "author_name": "zakopuro", "author_url": "", "post_date": "2023-06-03T20:50:54.723000", "content": "@philculliton \"Don't panic! I'm taking a break because it's the weekend. 😊👍\"
", "votes": 8, "replies": [ { "id": 2287025, "author_name": "empty", "author_url": "", "post_date": "2023-06-04T04:16:53.580000", "content": "laughing 😃
", "votes": 1, "replies": [ { "id": 2287048, "author_name": "Lintang Gilang Pratama", "author_url": "", "post_date": "2023-06-04T05:05:03.700000", "content": "Need Help
\nhttps://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/414914
\nI got an issue when trying to submit my code to Kaggle with an error message stating \"Notebook Threw Exception.\"
", "votes": -3, "replies": [] } ] } ] }, { "id": 2283678, "author_name": "M. Ikeda", "author_url": "", "post_date": "2023-06-01T12:35:30.700000", "content": "I have never entered a competition that had confusing changes for users 14 days before it ended.
\nMost of what I've been trying to improve my LB score has been wasted.
\nAs other users have pointed out, I think it should be extended.
\nHowever, repeated extensions are discouraging me😩
No one more extension PLZ😷
", "votes": 6, "replies": [] }, { "id": 2308429, "author_name": "Makotu", "author_url": "", "post_date": "2023-06-18T23:26:40.333000", "content": "Hi @philculliton ,
\nthe LB update be implemented after all? Or is it already done? It's already 10 days ago. Please let me know the situation.
I select one of mine old submission as final submission, however, with the new API running, it went wrong with the hint \"Submission Scoring Error\". But I can not deselect it. It has never happened in a normal competition!
\nHope Kaggle team can fix this problem!
\n
Why is the order of the events per session not being maintained since it plays a crucial role? sorting it by index can lead to errors since there are some ~1% sessions(mentioned in this post: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/395250) where the index is not increasing by one sequentially.
", "votes": 5, "replies": [ { "id": 2286436, "author_name": "Woprime", "author_url": "", "post_date": "2023-06-03T13:04:12.123000", "content": "Oh, so this is why I cant replicate my old result exactly!
", "votes": 1, "replies": [] } ] }, { "id": 2283076, "author_name": "Rib~", "author_url": "", "post_date": "2023-06-01T03:46:23.810000", "content": "Why do things always go wrong. Pheasant platform?
", "votes": 4, "replies": [] }, { "id": 2285042, "author_name": "wakaka", "author_url": "", "post_date": "2023-06-02T12:16:52.650000", "content": "no more extension please…
", "votes": 3, "replies": [] }, { "id": 2282393, "author_name": "Anthony Chiu", "author_url": "", "post_date": "2023-05-31T14:53:58.903000", "content": "\n\nThis data update required a complete rebuild and replacement of both APIs and should eliminate any differences between the two.
\n
Does it mean there are changes to the old API as well? I guess we should also expect a difference in the public lb score as well.
\nIs it possible to release the API in Python code format (apart from the .so file), so that we know exactly what data (format, order) will be in the inference loop? I think it is still very confusing to most of us how does the test loop work.
Thank you!
", "votes": 3, "replies": [] }, { "id": 2291229, "author_name": "Daniel Phalen", "author_url": "", "post_date": "2023-06-07T12:25:58.317000", "content": "For the trouble you should increase the prizes.
", "votes": 2, "replies": [] }, { "id": 2286710, "author_name": "Vadim Kamaev", "author_url": "", "post_date": "2023-06-03T17:21:48.607000", "content": "Due to some problems, please extend the competition
", "votes": 2, "replies": [] }, { "id": 2283421, "author_name": "superslon", "author_url": "", "post_date": "2023-06-01T08:39:15.263000", "content": "Instead of spending submissions to test hypotheses, i have to spend submissions to replicate my best result 😈
\nI think many people use old api for submitting and after changing it and restarting notebooks, we will simply lose our old submissions. Because restarting my old notebooks without any change gives me something like 0.65.
\nAnd judging by the experience of last rerun and number of angry comments in this topic, I think that LB rerun will definitely not take 1-2 days
\nI don’t understand what is the difficulty in presenting the code or a description of the api so that we don’t spend submissions on debugging code…
\nMeanwhile the competition should end in 14 days…
", "votes": 2, "replies": [] }, { "id": 2283190, "author_name": "Sihan Lv", "author_url": "", "post_date": "2023-06-01T05:44:34.813000", "content": "You wasted a day of my time and five submissions
", "votes": 3, "replies": [] }, { "id": 2284336, "author_name": "老肥", "author_url": "", "post_date": "2023-06-02T00:22:37.423000", "content": "Is the API still being updated? The three consecutive submissions just now reported errorsIs the API still being updated? The three consecutive submissions just now reported errors？
", "votes": 1, "replies": [ { "id": 2284436, "author_name": "xiaobenla", "author_url": "", "post_date": "2023-06-02T03:12:54.673000", "content": "I submitted it two hours ago and still haven't reported an errorI submitted it two hours ago and still haven't reported an error
", "votes": 0, "replies": [ { "id": 2284493, "author_name": "老肥", "author_url": "", "post_date": "2023-06-02T05:23:42.257000", "content": "it reported error when the inference is finished
", "votes": 0, "replies": [] }, { "id": 2284547, "author_name": "ADAM.", "author_url": "", "post_date": "2023-06-02T06:23:40.100000", "content": "I have a similar case here. Yesterday I submitted 2 notebooks at the same time. One succeed, the other failed. Today I resubmit the failed one again, it scores successfully. So I guess the api is somewhat unstable.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2283043, "author_name": "theta", "author_url": "", "post_date": "2023-06-01T03:13:49.750000", "content": "After API changed, I submitted the same notebook. LB decreased from 0.705 to 0.662.
", "votes": 1, "replies": [ { "id": 2283052, "author_name": "theta", "author_url": "", "post_date": "2023-06-01T03:21:42.340000", "content": "Sorting test dataframe by elapsed_time, LB returned from 0.662 to 0.704.
", "votes": 3, "replies": [ { "id": 2283061, "author_name": "Woprime", "author_url": "", "post_date": "2023-06-01T03:25:08.383000", "content": "Does sorting by index return to your 0.705?
", "votes": 0, "replies": [] }, { "id": 2283138, "author_name": "theta", "author_url": "", "post_date": "2023-06-01T04:54:54.263000", "content": "Yes. Sorting by index, LB returned to 0.705.
", "votes": 0, "replies": [] }, { "id": 2283166, "author_name": "ADAM.", "author_url": "", "post_date": "2023-06-01T05:17:43.793000", "content": "sorting by elapsed_time, give me the same 702. But sort by index, give me the exception. WTF😑
", "votes": 0, "replies": [] }, { "id": 2283175, "author_name": "empty", "author_url": "", "post_date": "2023-06-01T05:30:25.590000", "content": "sorted by index, still got low score
\nread here about the question numbers order, took into account but still got -0.001 from the original result 👀
Thx for this.. confused sorting by index helping or elapsed_time, might as well as experiment with both .
", "votes": 0, "replies": [] } ] } ] }, { "id": 2282593, "author_name": "Woprime", "author_url": "", "post_date": "2023-05-31T17:10:57.127000", "content": "Does this mean only the private dataset will be changed, and public LB will stay the same?
", "votes": 1, "replies": [] }, { "id": 2282228, "author_name": "老肥", "author_url": "", "post_date": "2023-05-31T13:16:56.210000", "content": "I failed to submit this afternoon (submission score error). I suspect that changes are made to the test set during my submission process, and then I submitted the same version of the notebook tonight. I found that the score had changed, and the presenter's statement now confirms this.
", "votes": 1, "replies": [ { "id": 2282596, "author_name": "Woprime", "author_url": "", "post_date": "2023-05-31T17:12:30.803000", "content": "Wait, so the public LB dataset has changed AGAIN??
", "votes": 0, "replies": [ { "id": 2282911, "author_name": "老肥", "author_url": "", "post_date": "2023-05-31T22:33:34.387000", "content": "Yes，It has changed.
", "votes": 0, "replies": [] }, { "id": 2282917, "author_name": "Phil Culliton", "author_url": "", "post_date": "2023-05-31T22:53:52.600000", "content": "The public API has been updated. The test set itself still has all the same samples. If your submission was impacted by the difference between the different Python version APIs then the update may change your score.
", "votes": -5, "replies": [] }, { "id": 2282947, "author_name": "empty", "author_url": "", "post_date": "2023-06-01T00:45:08.100000", "content": "So, let me understand the situation. You have just changed old API too (without any announcement about how it works now, or I just haven't seen it? doubt).
\nNew version of old API is not compatible with the previous version, and I have spent 4 submissions and a day searching bug in my code to realize it.
\nNow you will rerun all submissions which used old API with the new one. All scores will drop dramatically then. It equals removing competitors from LB.
\nMoreover, you are doing it two weeks before the end.
\nFor me it looks like a fatal combo of strange decisions
", "votes": 3, "replies": [] }, { "id": 2282951, "author_name": "empty", "author_url": "", "post_date": "2023-06-01T00:51:55.807000", "content": "I just don't get it. Is it a problem to make new version compatible with the previous one? As I understand there is some problem with sorting. Why can't you do it built-in?
", "votes": 3, "replies": [] }, { "id": 2282953, "author_name": "Woprime", "author_url": "", "post_date": "2023-06-01T00:59:03.763000", "content": "Soooo, instead of making new API works, they made the old API broken? This is really mind boogling, what have lead to this decision???
", "votes": 4, "replies": [] }, { "id": 2283902, "author_name": "FabienDaniel", "author_url": "", "post_date": "2023-06-01T15:11:25.347000", "content": "@philculliton this would be nice to have an *official *statement on whether there is a bug or not with the old/new API version. There's large differences between scores with the new version of the API, and many people points towards a potential issue with a sort() somewhere in the code.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2282230, "author_name": "ADAM.", "author_url": "", "post_date": "2023-05-31T13:18:23.790000", "content": "any ddl extension？
", "votes": 2, "replies": [] }, { "id": 2282930, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2023-05-31T23:46:33.887000", "content": "can we extend one more month? It takes time to get adapted to the new api.
", "votes": 0, "replies": [] }, { "id": 2285865, "author_name": "", "author_url": "", "post_date": "2023-06-03T04:09:09.830000", "content": "", "votes": -1, "replies": [] }, { "id": 2369798, "author_name": "", "author_url": "", "post_date": "2023-08-02T03:14:54.313000", "content": "", "votes": 0, "replies": [] }, { "id": 2312518, "author_name": "", "author_url": "", "post_date": "2023-06-22T01:39:21.460000", "content": "", "votes": 0, "replies": [] }, { "id": 2292261, "author_name": "", "author_url": "", "post_date": "2023-06-08T06:53:49.880000", "content": "", "votes": 0, "replies": [] }, { "id": 2289283, "author_name": "", "author_url": "", "post_date": "2023-06-06T04:10:33.703000", "content": "", "votes": 0, "replies": [] }, { "id": 2286974, "author_name": "", "author_url": "", "post_date": "2023-06-04T03:00:58.243000", "content": "", "votes": 0, "replies": [ { "id": 2286980, "author_name": "", "author_url": "", "post_date": "2023-06-04T03:07:14.373000", "content": "", "votes": 1, "replies": [ { "id": 2286984, "author_name": "", "author_url": "", "post_date": "2023-06-04T03:14:47.883000", "content": "", "votes": 1, "replies": [] } ] } ] }, { "id": 2283546, "author_name": "", "author_url": "", "post_date": "2023-06-01T10:38:16.390000", "content": "", "votes": 0, "replies": [] }, { "id": 2283373, "author_name": "", "author_url": "", "post_date": "2023-06-01T08:06:42.423000", "content": "", "votes": 0, "replies": [] }, { "id": 2283161, "author_name": "", "author_url": "", "post_date": "2023-06-01T05:14:53.967000", "content": "", "votes": 0, "replies": [ { "id": 2283167, "author_name": "", "author_url": "", "post_date": "2023-06-01T05:20:36.147000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 2282564, "author_name": "", "author_url": "", "post_date": "2023-05-31T16:40:55.647000", "content": "", "votes": 0, "replies": [] }, { "id": 2282494, "author_name": "", "author_url": "", "post_date": "2023-05-31T15:58:35.217000", "content": "", "votes": 0, "replies": [] }, { "id": 2282364, "author_name": "", "author_url": "", "post_date": "2023-05-31T14:39:54.377000", "content": "", "votes": 0, "replies": [] }, { "id": 2282206, "author_name": "", "author_url": "", "post_date": "2023-05-31T13:04:10.320000", "content": "", "votes": 0, "replies": [] }, { "id": 2311142, "author_name": "", "author_url": "", "post_date": "2023-06-21T00:12:12.163000", "content": "", "votes": 0, "replies": [] }, { "id": 2286983, "author_name": "", "author_url": "", "post_date": "2023-06-04T03:11:54.777000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2286587": "Hi @philculliton, I made some submissions today including some to check the validity of the new API, but **all of them failed** (some notebooks which **were successfully run before, now returns the submission scoring error**). Then, could we have an official announcement from Kaggle about,\n\n1. How did the API change in detail, then how to make a proper submission?\n\n2. Whether or not the deadline is extended, not a month but only to compensate for the time that the API has been invalid (as we couldn't submit our work), maybe a week or several days?\n\nThank you, @philculliton \n\nP/s: I think **we don't get frustrated because of the change regarding some technical issues, what makes people confused and maybe angry is why Kaggle keeps silent** when we only need some simple clarifications.\n\nUPDATE: While I am posting this post, my rank drops from 28th to 95th as my 0.703 notebooks become invalid.", "2283127": "I would like to request that the size of the test data not be changed significantly in a code competition where there is an upper limit to the inference time.\nIf the test data is suddenly changed without prior announcement, as in this case, the work done up to this point may have been in vain.", "2284890": "1. Could you tell us how much data will give us new and now?\n2. When will you update to new dataset?\n\nwe cannot estimate inference time for the new dataset. In notebook competition, That factor is very important.", "2282932": "I submitted and got LB 0.699 yesterday, \nAfter 15 hours, I submitted **same code**, I got LB 0.556…\nProbably many of you who submit with the old API are facing this problem...\nIt is an unstable competition.😓", "2291348": "This is not data centric or model centric competition.\nThis is an API centric competition 🤣🤣", "2282520": "`As the host mentioned in this post made when the original leak was acknowledged, we are going to incorporate a new private dataset into the competition that has been gathered over the months since the leak. `\n\nI would say most people only expected the first rerun, not the second one, especially not when there are only 14 days left before the deadline(also it seems the rerun is not going to happen immediately but 'roughly, this week if possible' ).\n\n`This data update required a complete rebuild and replacement of both APIs and should eliminate any differences between the two.`\n \nIf the newer new API work as intended, all things are good. But it took several days for people to realize that the 3.10 API has a different mechanic than the old API, and there was never any official explanation before this one.\n\nAlso, correct me if I remember it wrong. Last time when the API was updated at the beginning of May, the 1400 team was eliminated to 800 due to the older API no longer working. Should we expect the same and manually rerun our notebook this time? \n\n`We will resume submissions once the update is complete, and then begin re-running notebooks.`\n\nWe are already seeing different voices about whether there should be another ddl extension. On one side people are worrying about the compatibility of their model with new datasets and API, on the other side it's already quite an exhausting experience for 4 months of a single competition yet now I can't help to think if it's ever going to end.", "2284220": "\"This competition will run indefinitely with a rolling API which invalidates entries after each month\" 😄 ", "2283860": "To make the situation a little bit more clear, kaggle could post a schedule (date/what exactly is going to be changed) for the planned updates. \n\nOtherwise, people have to waste their time by trying to distinguish changes in the computational environment from effects caused by their model enhancements.", "2284833": "So, i have one more question for organizers. What is the point of changing old api? \n\nFirstly, in the new api test set is not sorted by indexes (it seems so, but no information has been received from organizers), and we have to waste time on a useless operation. Secondly, changing the api will lead to loss of many submissions, since api versions are not compatible", "2284393": "So i am still not able to reproduce my results with the same models even taking into account index sorting and the (dis)order of qids. Any tips @philculliton ? Or explanations why should we rebuild our pipelines spending subs at the end of competition?", "2286850": "@philculliton \"Don't panic! I'm taking a break because it's the weekend. 😊👍\"", "2283678": "I have never entered a competition that had confusing changes for users 14 days before it ended. \nMost of what I've been trying to improve my LB score has been wasted.\nAs other users have pointed out, I think it should be extended. \nHowever, repeated extensions are discouraging me😩", "2282377": "No one more extension PLZ😷", "2308429": "Hi @philculliton ,\nthe LB update be implemented after all? Or is it already done? It's already 10 days ago. Please let me know the situation.", "2301565": "I select one of mine old submission as final submission, however, with the new API running, it went wrong with the hint \"Submission Scoring Error\". But I can not deselect it. It has never happened in a normal competition! \n\nHope Kaggle team can fix this problem! \n", "2286103": "Why is the order of the events per session not being maintained since it plays a crucial role? sorting it by index can lead to errors since there are some ~1% sessions(mentioned in this post: https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/395250) where the index is not increasing by one sequentially.\n\n", "2283076": "Why do things always go wrong. Pheasant platform?", "2285042": "no more extension please...", "2282393": "> This data update required a complete rebuild and replacement of both APIs and should eliminate any differences between the two.\n\nDoes it mean there are changes to the old API as well? I guess we should also expect a difference in the public lb score as well. \n\nIs it possible to release the API in Python code format (apart from the `.so` file), so that we know exactly what data (format, order) will be in the inference loop? I think it is still very confusing to most of us how does the test loop work.\n\nThank you!", "2291229": "For the trouble you should increase the prizes.", "2286710": "Due to some problems, please extend the competition", "2283421": "Instead of spending submissions to test hypotheses, i have to spend submissions to replicate my best result 😈 \n\nI think many people use old api for submitting and after changing it and restarting notebooks, we will simply lose our old submissions. Because restarting my old notebooks without any change gives me something like 0.65. \n\nAnd judging by the experience of last rerun and number of angry comments in this topic, I think that LB rerun will definitely not take 1-2 days\n\nI don’t understand what is the difficulty in presenting the code or a description of the api so that we don’t spend submissions on debugging code...\n\nMeanwhile the competition should end in 14 days...", "2283190": "You wasted a day of my time and five submissions", "2284336": "Is the API still being updated? The three consecutive submissions just now reported errorsIs the API still being updated? The three consecutive submissions just now reported errors？", "2283043": "After API changed, I submitted the same notebook. LB decreased from 0.705 to 0.662.", "2282593": "Does this mean only the private dataset will be changed, and public LB will stay the same?", "2282228": "I failed to submit this afternoon (submission score error). I suspect that changes are made to the test set during my submission process, and then I submitted the same version of the notebook tonight. I found that the score had changed, and the presenter's statement now confirms this.", "2282230": "any ddl extension？", "2282930": "can we extend one more month? It takes time to get adapted to the new api.", "2282183": "Hi all -\n\nAs the host mentioned in [this post](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/396202) made when the original leak was acknowledged, we are going to incorporate a new private dataset into the competition that has been gathered over the months since the leak. This will involve rerunning your notebooks on the new private data.\n\nAs with last time, this could take quite some time, but we expect far fewer errors this time around, which should hopefully lead to a much shorter turnaround (roughly, this week if possible) as there should be much less to check and investigate.\n\nWe have also investigated recent reports from the forums about differences between the original and newer 3.10 versions of the API. This data update required a complete rebuild and replacement of both APIs and should eliminate any differences between the two.\n\nShortly, we will freeze submissions while we update the APIs, private data, and scoring mechanisms. We will resume submissions once the update is complete, and then begin re-running notebooks.", "2285865": "HI post the rescoring and LB update will the competition time be extended again ?\nOut of 5 submissions today, one was 'submission scoring error'", "2369798": "“What makes people confused and maybe angry is why Kaggle keeps silent when we only need some simple clarifications”. What a thoughtful sentence.", "2312518": "I just don't get it, why is it so hard to submit in this competition ? I have already burned 20 submissions just trying to submit my own predictions, printing the submission.csv all looks fine, this is so frustrating after spending 60 hours in the last 12 days setting up a model. Is the submission system down for now ? am I the only one experience this ?", "2292261": "There is one select submission which is wrong after new run, but I cannot cancel it. Therefore I waste one submission chance. ", "2289283": "I hadn't read this discussion thread, now I understand why I was having lower LB, now I don't even know which of my submissions is performing better, I'm a newbie but I can't believe it ", "2286974": "All my submission become \"Submissions Scoring Error\", is it normal?", "2283546": "Sorting by index doesn't really exactly replicate my result 0.701 -> 0.701, however sorting by public score gives different order. What is going on exactly...", "2283373": "My score is also decreasing after re-run without any change in code and models and using old API, from 0.698 to 0.571...\nBut high scores in old LB are still high scores in new test dataset.", "2283161": "Wasn't the LB update previously completed?\nDid anyone know that there are two levels of updates?", "2282564": "14 days to go btw :) ", "2282494": "Will it fix 'Notebook threw exception' error ?\n", "2282364": "HI post the rescoring and LB update will the competition time be extended again ?", "2282206": "Hi! PHIL CULLITON.\nSo when will you resume submissions and will this competition extend again? I'm not sure that I have enough time to update my models.", "2311142": "", "2286983": "" } }