![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F911838%2Fb9d9a93cbefe5963a83e5d029bf5e743%2FScreenshot%20from%202023-05-04%2000-12-16.png?generation=1683148363932424&alt=media\")
\n@philculliton this saying it's not 50/50
This competition is in the following situation:
\nI will provide a timeline of the situation as follows:
\nFebruary 7th: The competition starts.
\nFebruary 18th: A report is made that a leak has occurred due to a bug in the API. Kaggle staff says they will fix it but doesn't give a schedule.
\nMarch 21st: The train and test data are updated, but the API bug is not fixed and there is no report on it.
\nApril 8th: Kaggle staff reports that the API bug has been fixed.
\nSame day: There is a report pointing out a bug in the API, but there is no response to it.
\nApril 19th: A report is made that the API has been fixed and the LB will be updated within about a day.
\n
\n
\n(Update)
\nApril 24th: There were no bugs reported on 21 April; the API appears to have been fixed correctly.
\nApril 28th: The LB has been partially updated, but the updates appear to have been suspended. No reports from staff so far.
I don't think continuing this competition as it is will lead to success.
\nTo make the competition successful, I believe the best approach is to close the competition once and have a discussion between the staff and host to restart it in the proper state.
@philculliton @sohier @maggiemd
", "messages": [ { "id": "2230483", "postDate": "04/22/2023 11:58:48", "content": "This competition is in the following situation:
\nI will provide a timeline of the situation as follows:
\nFebruary 7th: The competition starts.
\nFebruary 18th: A report is made that a leak has occurred due to a bug in the API. Kaggle staff says they will fix it but doesn't give a schedule.
\nMarch 21st: The train and test data are updated, but the API bug is not fixed and there is no report on it.
\nApril 8th: Kaggle staff reports that the API bug has been fixed.
\nSame day: There is a report pointing out a bug in the API, but there is no response to it.
\nApril 19th: A report is made that the API has been fixed and the LB will be updated within about a day.
\n
\n
\n(Update)
\nApril 24th: There were no bugs reported on 21 April; the API appears to have been fixed correctly.
\nApril 28th: The LB has been partially updated, but the updates appear to have been suspended. No reports from staff so far.
I don't think continuing this competition as it is will lead to success.
\nTo make the competition successful, I believe the best approach is to close the competition once and have a discussion between the staff and host to restart it in the proper state.
@philculliton @sohier @maggiemd
", "rawMarkdown": "This competition is in the following situation:\n\n- There are bugs in the API\n- The leaderboard is not up to date\n- There is no response from the host or Kaggle staff\n\nwow that's great @zakopur0
", "rawMarkdown": "***wow that's great*** @zakopur0", "votes": null }, { "id": "2230541", "postDate": "04/22/2023 13:21:48", "content": "Hi! Any reports of API bugs (understandably, I hope) pause leaderboard updates. We check them out, see if we can reproduce them, and fix them if there's an issue. This sometimes takes quite a bit of time. We then resume the leaderboard update. We also have received some rather large and significant bugs reports outside of the forums which have required a great deal of due diligence.
\nRe: the most recent report you linked: I was working on that issue today and can't reproduce it. The issue that Chris Deotte and others reported appears to have been fixed, and all non-forum bug reports have been run down and checked out. The private API has not needed any fixes since the very early report at the start of the competition, and no new issues have been reported there, so we will resume the LB update when we get confirmation from the commenter you linked that this isn't some broader problem.
\nAlso, as noted in the update post, the LB update will take a considerable amount of time. There are tens of thousands of submissions with a maximum runtime of 9 hours. We do parallelize the process a great deal, but it may be a day or more from resumption before new scores are reflected.
", "rawMarkdown": "Hi! Any reports of API bugs (understandably, I hope) pause leaderboard updates. We check them out, see if we can reproduce them, and fix them if there's an issue. This sometimes takes quite a bit of time. We then resume the leaderboard update. We also have received some rather large and significant bugs reports outside of the forums which have required a great deal of due diligence.\n\nRe: the most recent report you linked: I was working on that issue today and can't reproduce it. The issue that Chris Deotte and others reported appears to have been fixed, and all non-forum bug reports have been run down and checked out. The private API has not needed any fixes since the very early report at the start of the competition, and no new issues have been reported there, so we will resume the LB update when we get confirmation from the commenter you linked that this isn't some broader problem.\n\nAlso, as noted in the update post, the LB update will take a considerable amount of time. There are tens of thousands of submissions with a maximum runtime of 9 hours. We do parallelize the process a great deal, but it may be a day or more from resumption before new scores are reflected.", "votes": null }, { "id": "2231439", "postDate": "04/23/2023 10:04:09", "content": "Thank you for your reply.
\nI understand that APIs can be complex and debugging can be difficult.
\nHowever, the participants just want to know how much progress has been made and when the debugging will be completed.
\nParticipants would be pleased to receive regular reports on the current situation. (We cannot wait for 1~2 months without any report.)
That's the point, the worst thing is waiting without reply.
", "rawMarkdown": "That's the point, the worst thing is waiting without reply.", "votes": null }, { "id": "2231698", "postDate": "04/23/2023 15:06:03", "content": "Thanks for your feedback. I will attempt to provide more regular updates. We have, as noted, received a number of potentially very impactful bug reports outside of the forums. They have turned out to be non-issues but they have consumed a great deal of work and made it difficult to provide useful updates.
\nI do not know precisely when the debugging will be completed. I just finished helping the commenter you linked. In the process I received a new bug report. I will kick off the LB update when the process is done.
", "rawMarkdown": "Thanks for your feedback. I will attempt to provide more regular updates. We have, as noted, received a number of potentially very impactful bug reports outside of the forums. They have turned out to be non-issues but they have consumed a great deal of work and made it difficult to provide useful updates.\n\nI do not know precisely when the debugging will be completed. I just finished helping the commenter you linked. In the process I received a new bug report. I will kick off the LB update when the process is done.", "votes": null }, { "id": "2231715", "postDate": "04/23/2023 15:13:51", "content": "Thank you!😊
\nI know debugging is a lot of work, but good luck. I hope the competition is a success.
Hi! The error from April 21th was actually caused by my code and was not an API bug. Wanted to let you know so you can update the status.
", "rawMarkdown": "Hi! The error from April 21th was actually caused by my code and was not an API bug. Wanted to let you know so you can update the status.", "votes": null }, { "id": "2232051", "postDate": "04/24/2023 00:43:38", "content": "Thank you!! updated.
", "rawMarkdown": "Thank you!! updated.", "votes": null }, { "id": "2232797", "postDate": "04/24/2023 16:16:33", "content": "Agreed that this competition has had a lot of issues. However, I think I can say for a lot of participants that discontinuing the competition is NOT a good idea. Many people, from competitors to host and Kaggle staffs, have been and are putting months of work into this event and - for what it's worth - would love to see its conclusion. Personally, as much as I am frustrated about all the bugs and issues, I also appreciate the fact that @philculliton @sohier and @maggiemd are trying their best to read all issue comments, respond to every reporters and follow through with all complaints. There are a lot to learn from this competitions, for both the host and the attendees.
", "rawMarkdown": "Agreed that this competition has had a lot of issues. However, I think I can say for a lot of participants that discontinuing the competition is **NOT** a good idea. Many people, from competitors to host and Kaggle staffs, have been and are putting months of work into this event and - for what it's worth - would love to see its conclusion. Personally, as much as I am frustrated about all the bugs and issues, I also appreciate the fact that @philculliton @sohier and @maggiemd are trying their best to read all issue comments, respond to every reporters and follow through with all complaints. There are a lot to learn from this competitions, for both the host and the attendees.", "votes": null }, { "id": "2234140", "postDate": "04/24/2023 23:54:00", "content": "I'm NOT proposing to discontinuing it. I'm proposing to \"restart\" it to make it a better competition.
", "rawMarkdown": "I'm **NOT** proposing to discontinuing it. I'm proposing to \"restart\" it to make it a better competition.", "votes": null }, { "id": "2239919", "postDate": "04/30/2023 03:00:49", "content": "So now is a good time to join the competition?
", "rawMarkdown": "So now is a good time to join the competition?", "votes": null }, { "id": "2240258", "postDate": "04/30/2023 11:33:02", "content": "I think it's still early to join since the calculations for LB are not yet complete.
\nAnd the Kaggle staff has stopped contacting us again.
I am not sure the status of this competition and I think many teams are puzzled about this. This competion has more than 1400+ teams in the past time, but for now this is only 900+ teams. Maybe it is because Kaggle team has rerun all the submissions, but I still doubt this competion should still have more than 1400 teams….
", "rawMarkdown": "I am not sure the status of this competition and I think many teams are puzzled about this. This competion has more than 1400+ teams in the past time, but for now this is only 900+ teams. Maybe it is because Kaggle team has rerun all the submissions, but I still doubt this competion should still have more than 1400 teams....", "votes": null }, { "id": "2242467", "postDate": "05/02/2023 09:56:21", "content": "It is very informative. I will not give up either Thank you for providing this information @zakopur0
", "rawMarkdown": "It is very informative. I will not give up either Thank you for providing this information @zakopur0", "votes": null }, { "id": "2242695", "postDate": "05/02/2023 13:02:27", "content": "Thanks for the information!!
", "rawMarkdown": "Thanks for the information!!", "votes": null }, { "id": "2243188", "postDate": "05/02/2023 18:14:34", "content": "Also, I noticed the test data for leaderboard have changed without any explanation(because I rerun the exactly same models and get different LB and the percentage of public Lb data has also changed from 50% to 36% as far as I remembered).
", "rawMarkdown": "Also, I noticed the test data for leaderboard have changed without any explanation(because I rerun the exactly same models and get different LB and the percentage of public Lb data has also changed from 50% to 36% as far as I remembered).", "votes": null }, { "id": "2243305", "postDate": "05/02/2023 19:45:42", "content": "Hahaha😂 Thank you for the information you provided.
\nThis is my first experience with test data being changed without any reporting.😂
can I ask what is the CV/LB currently in your case? I have a single model with 0.698 CV but the LB is only 0.690, I checked everything, and they are all correct, I don't know what really happened… My guess is my understanding of how API works is wrong, but I raised a discussion and Kaggle staff don't answer yet… It is a shame because the problem of this competition itself is very interesting, but the competition set-up is just too bad.
", "rawMarkdown": "can I ask what is the CV/LB currently in your case? I have a single model with 0.698 CV but the LB is only 0.690, I checked everything, and they are all correct, I don't know what really happened... My guess is my understanding of how API works is wrong, but I raised a discussion and Kaggle staff don't answer yet... It is a shame because the problem of this competition itself is very interesting, but the competition set-up is just too bad.", "votes": null }, { "id": "2243413", "postDate": "05/02/2023 22:16:33", "content": "Is this real?? When did you notice this happend?
", "rawMarkdown": "Is this real?? When did you notice this happend?", "votes": null }, { "id": "2243470", "postDate": "05/03/2023 00:00:31", "content": "\n\nI will attempt to provide more regular updates
\n
@philculliton
\nI know you have a lot of questions coming in, can you answer them?
My cv/lb correlation is stable, lb is a bit higher than cv.(cv: 0.701, lb: 0.704(0.702 after rerun)).
", "rawMarkdown": "My cv/lb correlation is stable, lb is a bit higher than cv.(cv: 0.701, lb: 0.704(0.702 after rerun)).", "votes": null }, { "id": "2243548", "postDate": "05/03/2023 01:55:40", "content": "About 1 month ago, I ran 3 models after the first update on LB data which is caused by leaked data.
\nRecently, kaggle started to rerun all the submissions and the rerun have been suspended due to unknown reason. So, I manually rerun those 3 submission and found the lb scores of those 3 submissions has dropped a bit. In those 3 submissions, I uploaded model files and predicted, so I guess there is no reason for the lb change except the data change. Also, I noticed the percentage of public lb changed from 0.5 to 0.36 as far as I remembered.
It's really confusing. It's most likely those 500 teams disappear due to the LB rerun mentioned by Kaggle stuff (hopefully they will rerun their submission at least once before the competition ends), but according to other replies on this topic LB score actually doesn't update itself with the hidden test dataset update, when people rerun their submission manually they got a different LB score. It's like a Schrödinger’s rerun at this time.
", "rawMarkdown": "It's really confusing. It's most likely those 500 teams disappear due to the LB rerun mentioned by Kaggle stuff (hopefully they will rerun their submission at least once before the competition ends), but according to other replies on this topic LB score actually doesn't update itself with the hidden test dataset update, when people rerun their submission manually they got a different LB score. It's like a Schrödinger’s rerun at this time.", "votes": null }, { "id": "2243823", "postDate": "05/03/2023 07:38:19", "content": "lol. confirmed
\nthat's so circus
@hookman wait, am I right that your 704 is not reproducible now?
", "rawMarkdown": "hookman wait, am I right that your 704 is not reproducible now?", "votes": null }, { "id": "2243850", "postDate": "05/03/2023 08:19:06", "content": "yes. It's only 702 now. How about your best public lb?
", "rawMarkdown": "yes. It's only 702 now. How about your best public lb?", "votes": null }, { "id": "2243859", "postDate": "05/03/2023 08:27:00", "content": "So in summary, even after the LB update, LB is still pretty much useless. 😑
", "rawMarkdown": "So in summary, even after the LB update, LB is still pretty much useless. 😑", "votes": null }, { "id": "2243861", "postDate": "05/03/2023 08:31:28", "content": "My lb improved 700 -> 701 just due to rerun tonight :)
", "rawMarkdown": "My lb improved 700 -> 701 just due to rerun tonight :)", "votes": null }, { "id": "2243871", "postDate": "05/03/2023 08:39:03", "content": "cool!! you got improved after rerun.
", "rawMarkdown": "cool!! you got improved after rerun.", "votes": null }, { "id": "2244563", "postDate": "05/03/2023 17:57:30", "content": "Hi @hookman, I noticed the same, I can't reproduce my highest score, I get .001 lower with the same code. I spent a long time and many trials to make sure it is all exactly the same on my side. I have the same CV and the same feature importance, so the training data hasn't changed. It may be the public/private split that has changed. My submission scores dropped 18 days ago.
", "rawMarkdown": "Hi @hookman, I noticed the same, I can't reproduce my highest score, I get .001 lower with the same code. I spent a long time and many trials to make sure it is all exactly the same on my side. I have the same CV and the same feature importance, so the training data hasn't changed. It may be the public/private split that has changed. My submission scores dropped 18 days ago.", "votes": null }, { "id": "2244569", "postDate": "05/03/2023 18:03:41", "content": "Hi! The test data hasn't changed since March 20th. The public / private split is still 50/50 - the percentage of scored samples has changed due to an issue with the original leaked data, but the public / private split has not changed.
", "rawMarkdown": "Hi! The test data hasn't changed since March 20th. The public / private split is still 50/50 - the percentage of scored samples has changed due to an issue with the original leaked data, but the public / private split has not changed.", "votes": null }, { "id": "2244571", "postDate": "05/03/2023 18:04:52", "content": "The rerun has not been suspended. It's large and is taking time. I will post a notification when it's complete.
\nThe percentage of the public LB has not changed - the percentage of scored samples has changed due to an issue with the original leaked data. The public / private split is still 50/50.
", "rawMarkdown": "The rerun has not been suspended. It's large and is taking time. I will post a notification when it's complete.\n\nThe percentage of the public LB has not changed - the percentage of scored samples has changed due to an issue with the original leaked data. The public / private split is still 50/50.", "votes": null }, { "id": "2244585", "postDate": "05/03/2023 18:23:45", "content": "Ok, thanks for the response.
", "rawMarkdown": "Ok, thanks for the response.", "votes": null }, { "id": "2244734", "postDate": "05/03/2023 21:13:28", "content": "
\n@philculliton this saying it's not 50/50
@philculliton Hi Phil! As several people have reported they cannot reproduce their public LB and you confirmed that the test data has not changed since March 20th, I guess there could be some potential bugs. Could you please take some time to look into this?
\nAs @gehallak mentioned this happens about 18 days ago(i.e. about April 15th). Are there some version or other updates near this date?
", "rawMarkdown": "philculliton Hi Phil! As several people have reported they cannot reproduce their public LB and you confirmed that the test data has not changed since March 20th, I guess there could be some potential bugs. Could you please take some time to look into this?\n\nAs @gehallak mentioned this happens about 18 days ago(i.e. about April 15th). Are there some version or other updates near this date?", "votes": null }, { "id": "2244897", "postDate": "05/04/2023 01:43:09", "content": "Hi @philculliton, @hookman, If the test data hasn't changed AND the split public/private hasn't changed, then maybe the public and private test data were switched ?
", "rawMarkdown": "Hi @philculliton, @hookman, If the test data hasn't changed AND the split public/private hasn't changed, then maybe the public and private test data were switched ?", "votes": null }, { "id": "2244903", "postDate": "05/04/2023 01:53:47", "content": "@gehallak 😂 I have also considered the possibility of this happening.
", "rawMarkdown": "gehallak 😂 I have also considered the possibility of this happening.", "votes": null }, { "id": "2250202", "postDate": "05/08/2023 12:11:46", "content": "so
\nthere are several threads like this about new Kaggle API bug.
\nthere are several confirmations of the fact that same code returned different lb scores.
but there is only silence from @philculliton and other hosts :(
\nso sad to see how an interesting competition is spoiled by the indifference of the organizers
@philculliton only one month left, hope we can have a REAL LB as soon as possible…
", "rawMarkdown": "philculliton only one month left, hope we can have a REAL LB as soon as possible...", "votes": null }, { "id": "2250292", "postDate": "05/08/2023 13:23:36", "content": "I bet there will be another deadline extension
", "rawMarkdown": "I bet there will be another deadline extension", "votes": null }, { "id": "2250309", "postDate": "05/08/2023 13:38:54", "content": "endless game 😂
", "rawMarkdown": "endless game 😂", "votes": null }, { "id": "2250384", "postDate": "05/08/2023 14:21:31", "content": "He doesn't do what he's supposed to do as a staff, so I hope he will be replaced by another staff.
", "rawMarkdown": "He doesn't do what he's supposed to do as a staff, so I hope he will be replaced by another staff.", "votes": null }, { "id": "2250405", "postDate": "05/08/2023 14:41:16", "content": "No more extension pls :(
", "rawMarkdown": "No more extension pls :(", "votes": null } ], "comments": [ { "id": 2230495, "author_name": "pradeepsapparapu", "author_url": "", "post_date": "04/22/2023 12:24:57", "content": "wow that's great @zakopur0
", "votes": null, "replies": [] }, { "id": 2230541, "author_name": "philculliton", "author_url": "", "post_date": "04/22/2023 13:21:48", "content": "Hi! Any reports of API bugs (understandably, I hope) pause leaderboard updates. We check them out, see if we can reproduce them, and fix them if there's an issue. This sometimes takes quite a bit of time. We then resume the leaderboard update. We also have received some rather large and significant bugs reports outside of the forums which have required a great deal of due diligence.
\nRe: the most recent report you linked: I was working on that issue today and can't reproduce it. The issue that Chris Deotte and others reported appears to have been fixed, and all non-forum bug reports have been run down and checked out. The private API has not needed any fixes since the very early report at the start of the competition, and no new issues have been reported there, so we will resume the LB update when we get confirmation from the commenter you linked that this isn't some broader problem.
\nAlso, as noted in the update post, the LB update will take a considerable amount of time. There are tens of thousands of submissions with a maximum runtime of 9 hours. We do parallelize the process a great deal, but it may be a day or more from resumption before new scores are reflected.
", "votes": null, "replies": [ { "id": 2231439, "author_name": "zakopur0", "author_url": "", "post_date": "04/23/2023 10:04:09", "content": "Thank you for your reply.
\nI understand that APIs can be complex and debugging can be difficult.
\nHowever, the participants just want to know how much progress has been made and when the debugging will be completed.
\nParticipants would be pleased to receive regular reports on the current situation. (We cannot wait for 1~2 months without any report.)
That's the point, the worst thing is waiting without reply.
", "votes": null, "replies": [] }, { "id": 2231698, "author_name": "philculliton", "author_url": "", "post_date": "04/23/2023 15:06:03", "content": "Thanks for your feedback. I will attempt to provide more regular updates. We have, as noted, received a number of potentially very impactful bug reports outside of the forums. They have turned out to be non-issues but they have consumed a great deal of work and made it difficult to provide useful updates.
\nI do not know precisely when the debugging will be completed. I just finished helping the commenter you linked. In the process I received a new bug report. I will kick off the LB update when the process is done.
", "votes": null, "replies": [ { "id": 2231715, "author_name": "zakopur0", "author_url": "", "post_date": "04/23/2023 15:13:51", "content": "Thank you!😊
\nI know debugging is a lot of work, but good luck. I hope the competition is a success.
\n\nI will attempt to provide more regular updates
\n
@philculliton
\nI know you have a lot of questions coming in, can you answer them?
Hi! The error from April 21th was actually caused by my code and was not an API bug. Wanted to let you know so you can update the status.
", "votes": null, "replies": [ { "id": 2232051, "author_name": "zakopur0", "author_url": "", "post_date": "04/24/2023 00:43:38", "content": "Thank you!! updated.
", "votes": null, "replies": [] } ] }, { "id": 2232797, "author_name": "hoangnguyen719", "author_url": "", "post_date": "04/24/2023 16:16:33", "content": "Agreed that this competition has had a lot of issues. However, I think I can say for a lot of participants that discontinuing the competition is NOT a good idea. Many people, from competitors to host and Kaggle staffs, have been and are putting months of work into this event and - for what it's worth - would love to see its conclusion. Personally, as much as I am frustrated about all the bugs and issues, I also appreciate the fact that @philculliton @sohier and @maggiemd are trying their best to read all issue comments, respond to every reporters and follow through with all complaints. There are a lot to learn from this competitions, for both the host and the attendees.
", "votes": null, "replies": [ { "id": 2234140, "author_name": "zakopur0", "author_url": "", "post_date": "04/24/2023 23:54:00", "content": "I'm NOT proposing to discontinuing it. I'm proposing to \"restart\" it to make it a better competition.
", "votes": null, "replies": [] } ] }, { "id": 2239919, "author_name": "xiaowangiiiii", "author_url": "", "post_date": "04/30/2023 03:00:49", "content": "So now is a good time to join the competition?
", "votes": null, "replies": [ { "id": 2240258, "author_name": "zakopur0", "author_url": "", "post_date": "04/30/2023 11:33:02", "content": "I think it's still early to join since the calculations for LB are not yet complete.
\nAnd the Kaggle staff has stopped contacting us again.
I am not sure the status of this competition and I think many teams are puzzled about this. This competion has more than 1400+ teams in the past time, but for now this is only 900+ teams. Maybe it is because Kaggle team has rerun all the submissions, but I still doubt this competion should still have more than 1400 teams….
", "votes": null, "replies": [ { "id": 2243654, "author_name": "ataraxian", "author_url": "", "post_date": "05/03/2023 04:29:01", "content": "It's really confusing. It's most likely those 500 teams disappear due to the LB rerun mentioned by Kaggle stuff (hopefully they will rerun their submission at least once before the competition ends), but according to other replies on this topic LB score actually doesn't update itself with the hidden test dataset update, when people rerun their submission manually they got a different LB score. It's like a Schrödinger’s rerun at this time.
", "votes": null, "replies": [] } ] }, { "id": 2242467, "author_name": "risakashiwabara", "author_url": "", "post_date": "05/02/2023 09:56:21", "content": "It is very informative. I will not give up either Thank you for providing this information @zakopur0
", "votes": null, "replies": [] }, { "id": 2242695, "author_name": "naumanahmed97", "author_url": "", "post_date": "05/02/2023 13:02:27", "content": "Thanks for the information!!
", "votes": null, "replies": [] }, { "id": 2243188, "author_name": "hookman", "author_url": "", "post_date": "05/02/2023 18:14:34", "content": "Also, I noticed the test data for leaderboard have changed without any explanation(because I rerun the exactly same models and get different LB and the percentage of public Lb data has also changed from 50% to 36% as far as I remembered).
", "votes": null, "replies": [ { "id": 2243305, "author_name": "zakopur0", "author_url": "", "post_date": "05/02/2023 19:45:42", "content": "Hahaha😂 Thank you for the information you provided.
\nThis is my first experience with test data being changed without any reporting.😂
can I ask what is the CV/LB currently in your case? I have a single model with 0.698 CV but the LB is only 0.690, I checked everything, and they are all correct, I don't know what really happened… My guess is my understanding of how API works is wrong, but I raised a discussion and Kaggle staff don't answer yet… It is a shame because the problem of this competition itself is very interesting, but the competition set-up is just too bad.
", "votes": null, "replies": [ { "id": 2243543, "author_name": "hookman", "author_url": "", "post_date": "05/03/2023 01:46:07", "content": "My cv/lb correlation is stable, lb is a bit higher than cv.(cv: 0.701, lb: 0.704(0.702 after rerun)).
", "votes": null, "replies": [ { "id": 2243844, "author_name": "kvlmll", "author_url": "", "post_date": "05/03/2023 08:12:22", "content": "@hookman wait, am I right that your 704 is not reproducible now?
", "votes": null, "replies": [ { "id": 2243850, "author_name": "hookman", "author_url": "", "post_date": "05/03/2023 08:19:06", "content": "yes. It's only 702 now. How about your best public lb?
", "votes": null, "replies": [ { "id": 2243859, "author_name": "woprime", "author_url": "", "post_date": "05/03/2023 08:27:00", "content": "So in summary, even after the LB update, LB is still pretty much useless. 😑
", "votes": null, "replies": [] }, { "id": 2243861, "author_name": "kvlmll", "author_url": "", "post_date": "05/03/2023 08:31:28", "content": "My lb improved 700 -> 701 just due to rerun tonight :)
", "votes": null, "replies": [ { "id": 2243871, "author_name": "hookman", "author_url": "", "post_date": "05/03/2023 08:39:03", "content": "cool!! you got improved after rerun.
", "votes": null, "replies": [] } ] } ] } ] } ] } ] }, { "id": 2243413, "author_name": "woprime", "author_url": "", "post_date": "05/02/2023 22:16:33", "content": "Is this real?? When did you notice this happend?
", "votes": null, "replies": [ { "id": 2243548, "author_name": "hookman", "author_url": "", "post_date": "05/03/2023 01:55:40", "content": "About 1 month ago, I ran 3 models after the first update on LB data which is caused by leaked data.
\nRecently, kaggle started to rerun all the submissions and the rerun have been suspended due to unknown reason. So, I manually rerun those 3 submission and found the lb scores of those 3 submissions has dropped a bit. In those 3 submissions, I uploaded model files and predicted, so I guess there is no reason for the lb change except the data change. Also, I noticed the percentage of public lb changed from 0.5 to 0.36 as far as I remembered.
The rerun has not been suspended. It's large and is taking time. I will post a notification when it's complete.
\nThe percentage of the public LB has not changed - the percentage of scored samples has changed due to an issue with the original leaked data. The public / private split is still 50/50.
", "votes": null, "replies": [ { "id": 2244585, "author_name": "woprime", "author_url": "", "post_date": "05/03/2023 18:23:45", "content": "Ok, thanks for the response.
", "votes": null, "replies": [] }, { "id": 2244734, "author_name": "kvlmll", "author_url": "", "post_date": "05/03/2023 21:13:28", "content": "
\n@philculliton this saying it's not 50/50
lol. confirmed
\nthat's so circus
Hi @hookman, I noticed the same, I can't reproduce my highest score, I get .001 lower with the same code. I spent a long time and many trials to make sure it is all exactly the same on my side. I have the same CV and the same feature importance, so the training data hasn't changed. It may be the public/private split that has changed. My submission scores dropped 18 days ago.
", "votes": null, "replies": [] }, { "id": 2244569, "author_name": "philculliton", "author_url": "", "post_date": "05/03/2023 18:03:41", "content": "Hi! The test data hasn't changed since March 20th. The public / private split is still 50/50 - the percentage of scored samples has changed due to an issue with the original leaked data, but the public / private split has not changed.
", "votes": null, "replies": [ { "id": 2244894, "author_name": "hookman", "author_url": "", "post_date": "05/04/2023 01:33:33", "content": "@philculliton Hi Phil! As several people have reported they cannot reproduce their public LB and you confirmed that the test data has not changed since March 20th, I guess there could be some potential bugs. Could you please take some time to look into this?
\nAs @gehallak mentioned this happens about 18 days ago(i.e. about April 15th). Are there some version or other updates near this date?
", "votes": null, "replies": [ { "id": 2244897, "author_name": "gehallak", "author_url": "", "post_date": "05/04/2023 01:43:09", "content": "Hi @philculliton, @hookman, If the test data hasn't changed AND the split public/private hasn't changed, then maybe the public and private test data were switched ?
", "votes": null, "replies": [ { "id": 2244903, "author_name": "hookman", "author_url": "", "post_date": "05/04/2023 01:53:47", "content": "@gehallak 😂 I have also considered the possibility of this happening.
", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 2250202, "author_name": "kvlmll", "author_url": "", "post_date": "05/08/2023 12:11:46", "content": "so
\nthere are several threads like this about new Kaggle API bug.
\nthere are several confirmations of the fact that same code returned different lb scores.
but there is only silence from @philculliton and other hosts :(
\nso sad to see how an interesting competition is spoiled by the indifference of the organizers
@philculliton only one month left, hope we can have a REAL LB as soon as possible…
", "votes": null, "replies": [ { "id": 2250292, "author_name": "kvlmll", "author_url": "", "post_date": "05/08/2023 13:23:36", "content": "I bet there will be another deadline extension
", "votes": null, "replies": [ { "id": 2250309, "author_name": "zui0711", "author_url": "", "post_date": "05/08/2023 13:38:54", "content": "endless game 😂
", "votes": null, "replies": [ { "id": 2250405, "author_name": "ashufet", "author_url": "", "post_date": "05/08/2023 14:41:16", "content": "No more extension pls :(
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2250384, "author_name": "zakopur0", "author_url": "", "post_date": "05/08/2023 14:21:31", "content": "He doesn't do what he's supposed to do as a staff, so I hope he will be replaced by another staff.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2230483": "This competition is in the following situation:\n\n- There are bugs in the API\n- The leaderboard is not up to date\n- There is no response from the host or Kaggle staff\n\n