\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16582438%2F339c17a3bb378028b6377dc453e19dfe%2F.PNG?generation=1740567817799427&alt=media\")
Hello Kaggle Team, Kaggle Community, and Competition hosts,
\nOur team participated in the Jane Street Real-Time Market Data Forecasting competition, and we encountered a critical issue during the forecasting period where none of our submissions were properly scored on the Private Leaderboard.
\nDuring the Public Leaderboard phase, our submissions were successfully evaluated, and we had no issues. However, once the competition transitioned into the forecasting period, every submission we made—whether it was our own developed solution, a publicly available notebook solution, or a combination of both—failed to be scored correctly. This happened regardless of whether we selected those solutions as our final submissions.
\nThe submission logs indicate that all our submissions were marked as \"Succeeded,\" yet they were not evaluated on the Private LB. The attached image provides evidence of this issue.
\n
We want to clarify that our team did not engage in any rule violations or unethical practices. Given that multiple solutions were affected, we believe this could be a technical issue rather than a problem specific to our team.
\nCould the Kaggle team and Community please investigate this matter and provide clarification on why our submissions were not evaluated on the Private LB? We would appreciate any insights or possible resolutions.
", "messages": [ { "id": 3134452, "postDate": "2025-02-26T11:05:21.027Z", "content": "Hello Kaggle Team, Kaggle Community, and Competition hosts,
\nOur team participated in the Jane Street Real-Time Market Data Forecasting competition, and we encountered a critical issue during the forecasting period where none of our submissions were properly scored on the Private Leaderboard.
\nDuring the Public Leaderboard phase, our submissions were successfully evaluated, and we had no issues. However, once the competition transitioned into the forecasting period, every submission we made—whether it was our own developed solution, a publicly available notebook solution, or a combination of both—failed to be scored correctly. This happened regardless of whether we selected those solutions as our final submissions.
\nThe submission logs indicate that all our submissions were marked as \"Succeeded,\" yet they were not evaluated on the Private LB. The attached image provides evidence of this issue.
\n
We want to clarify that our team did not engage in any rule violations or unethical practices. Given that multiple solutions were affected, we believe this could be a technical issue rather than a problem specific to our team.
\nCould the Kaggle team and Community please investigate this matter and provide clarification on why our submissions were not evaluated on the Private LB? We would appreciate any insights or possible resolutions.
", "rawMarkdown": "Hello Kaggle Team, Kaggle Community, and Competition hosts,\n\nOur team participated in the Jane Street Real-Time Market Data Forecasting competition, and we encountered a critical issue during the forecasting period where none of our submissions were properly scored on the Private Leaderboard.\n\nDuring the Public Leaderboard phase, our submissions were successfully evaluated, and we had no issues. However, once the competition transitioned into the forecasting period, every submission we made—whether it was our own developed solution, a publicly available notebook solution, or a combination of both—failed to be scored correctly. This happened regardless of whether we selected those solutions as our final submissions.\n\nThe submission logs indicate that all our submissions were marked as \"Succeeded,\" yet they were not evaluated on the Private LB. The attached image provides evidence of this issue.\n\n\nWe want to clarify that our team did not engage in any rule violations or unethical practices. Given that multiple solutions were affected, we believe this could be a technical issue rather than a problem specific to our team.\n\nCould the Kaggle team and Community please investigate this matter and provide clarification on why our submissions were not evaluated on the Private LB? We would appreciate any insights or possible resolutions.", "votes": 6 }, { "id": 3155737, "postDate": "2025-03-21T11:05:14.403Z", "content": "I have encountered similar issues in other competitions. It’s possible that certain edge cases in the real-time data—whether due to unexpected data format changes, irregular time series intervals, or specific forecasting rules—caused the system to skip final scoring. Even when the submission logs showed “Succeeded,” there may have been hidden validation checks or internal constraints, such as a limit on data availability or acceptable input structures, that resulted in an incomplete evaluation process. If these edge cases weren’t clearly outlined in the competition rules, teams might inadvertently submit solutions that appear valid yet are never fully scored.
", "rawMarkdown": "I have encountered similar issues in other competitions. It’s possible that certain edge cases in the real-time data—whether due to unexpected data format changes, irregular time series intervals, or specific forecasting rules—caused the system to skip final scoring. Even when the submission logs showed “Succeeded,” there may have been hidden validation checks or internal constraints, such as a limit on data availability or acceptable input structures, that resulted in an incomplete evaluation process. If these edge cases weren’t clearly outlined in the competition rules, teams might inadvertently submit solutions that appear valid yet are never fully scored.", "votes": 3 }, { "id": 3136493, "postDate": "2025-02-28T15:32:27.943Z", "content": "There is something same with our team. In the first LB update, our 2 submission were both evaluated. But in the second LB update, one of submissions were not evaluated. So now our team have only 1 submission. :(
", "rawMarkdown": "There is something same with our team. In the first LB update, our 2 submission were both evaluated. But in the second LB update, one of submissions were not evaluated. So now our team have only 1 submission. :(", "votes": 3, "replies": [ { "id": 3136966, "postDate": "2025-03-01T03:48:24.407Z", "content": "Still, it’s fortunate that one of your team’s two submissions survived! Both of our submissions weren’t scored, so we ended up unranked…😭
", "rawMarkdown": "Still, it’s fortunate that one of your team’s two submissions survived! Both of our submissions weren’t scored, so we ended up unranked…😭" } ] }, { "id": 3134535, "postDate": "2025-02-26T13:30:00.270Z", "content": "The \"Succeeded\" ones shwon in your screenshot are the ones that passed the first phase e.g. before the close of public LB. They have nothing to do with the LB update.
\nIn the LB update, the two chosen submission is re-run on the entire dataset and creates two new submissions with comments Notebook Rerun for Submission [] on 02/05/2025 under it. The other unchosen ones remain as it was.
After we entered the forecasting phase, there was one update on 02/05/2025 (the first update). For this update, you should see the two chosen submission, namely \"Lucky Number 456\" and \"lightweighting by removing qxgb n cat fold\", got re-run and have Notebook Rerun for Submission [] on 02/05/2025 comment under it.
In your screenshot, the two chosen submission did not pass the first LB update. This would be the reason why your team did not get evaluated in the second update (which was on 02/11/2025).
", "rawMarkdown": "The \"Succeeded\" ones shwon in your screenshot are the ones that passed the first phase e.g. before the close of public LB. They have nothing to do with the LB update.\n\nIn the LB update, the two chosen submission is re-run on the entire dataset and creates two new submissions with comments `Notebook Rerun for Submission [] on 02/05/2025` under it. The other unchosen ones remain as it was.\n\nAfter we entered the forecasting phase, there was one update on 02/05/2025 (the first update). For this update, you should see the two chosen submission, namely \"Lucky Number 456\" and \"lightweighting by removing qxgb n cat fold\", got re-run and have `Notebook Rerun for Submission [] on 02/05/2025` comment under it. \n\nIn your screenshot, the two chosen submission did not pass the first LB update. This would be the reason why your team did not get evaluated in the second update (which was on 02/11/2025).", "votes": 3, "replies": [ { "id": 3134570, "postDate": "2025-02-26T14:01:37.190Z", "content": "Thank you for your thoughtful explanation. Our team also experienced issues with the rerun not being processed correctly during the February 5th update. Since then, we have been trying to identify the cause, but we were unable to figure it out on our own, which is why we have only now decided to post in the discussion. We are not sure what went wrong, and it is quite frustrating that our submissions were not properly evaluated.🥺
", "rawMarkdown": "Thank you for your thoughtful explanation. Our team also experienced issues with the rerun not being processed correctly during the February 5th update. Since then, we have been trying to identify the cause, but we were unable to figure it out on our own, which is why we have only now decided to post in the discussion. We are not sure what went wrong, and it is quite frustrating that our submissions were not properly evaluated.🥺" } ] }, { "id": 3152648, "postDate": "2025-03-18T03:07:54.623Z", "content": "I also suffered from this problem. My notebook didn't pass the first rerun. It shows a note saying
\nNotebook Rerun for Submission XXX on 02/11/2025, and has an Error sign in the begining. So I remained unrank.
\nIs it because my online retrain used up all the time? Or maybe some error occurred during private evaluated?
I tried to rerun my notebook on Kaggle, and found that one of the models is no longer public, which leads to an Error while running the notebook. I think this maybe the reason why my code can't be evaluated. So sad…😢
", "rawMarkdown": "I tried to rerun my notebook on Kaggle, and found that one of the models is no longer public, which leads to an Error while running the notebook. I think this maybe the reason why my code can't be evaluated. So sad...😢", "votes": 1, "replies": [ { "id": 3152979, "postDate": "2025-03-18T10:18:55.597Z", "content": "Oh, I see… I’m sorry to hear that 🥺 I wonder if my solution also failed for the same reason… It would have been better if there were more conditions regarding the use of public notebooks as the final solution.
", "rawMarkdown": "Oh, I see… I’m sorry to hear that 🥺 I wonder if my solution also failed for the same reason… It would have been better if there were more conditions regarding the use of public notebooks as the final solution." } ] } ] }, { "id": 3145988, "postDate": "2025-03-10T12:53:13.943Z", "content": "I'm wondering whether or not public notebooks that are added to the input are restricted from further updates. In the case of datasets, there is no issue because it was pinned once it was attached.
", "rawMarkdown": "I'm wondering whether or not public notebooks that are added to the input are restricted from further updates. In the case of datasets, there is no issue because it was pinned once it was attached.", "votes": 1, "replies": [ { "id": 3146969, "postDate": "2025-03-11T14:16:40.703Z", "content": "A solution that didn't use any public notebook resources at all also encountered a problem!😭
", "rawMarkdown": "A solution that didn't use any public notebook resources at all also encountered a problem!😭" } ] }, { "id": 3134465, "postDate": "2025-02-26T11:28:59.480Z", "content": "@donghuilim it is possible that your code created an error in the evaluation phase causing this problem
", "rawMarkdown": "@donghuilim it is possible that your code created an error in the evaluation phase causing this problem ", "votes": 1, "replies": [ { "id": 3134475, "postDate": "2025-02-26T11:51:21.190Z", "content": "If there had been a potential issue with the code of one of our team members, it would not have caused problems for all of our team's submissions during the evaluation phase. Some of our submissions were based on different baselines, and some were submitted individually without coordination among team members. We truly do not know what the issue is. From what I have seen in other discussion posts about this competition, it appears that other teams are experiencing similar issues as well. We genuinely have no idea what might be causing this.
", "rawMarkdown": "If there had been a potential issue with the code of one of our team members, it would not have caused problems for all of our team's submissions during the evaluation phase. Some of our submissions were based on different baselines, and some were submitted individually without coordination among team members. We truly do not know what the issue is. From what I have seen in other discussion posts about this competition, it appears that other teams are experiencing similar issues as well. We genuinely have no idea what might be causing this." } ] }, { "id": 3146799, "postDate": "2025-03-11T09:01:07.150Z", "content": "There is something same with me.
\nIn the first LB update, my 2 submission were both evaluated. But in the LB update in March, both of my submissions were not evaluated.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F12797059%2F617413cb756583077082abc4b56104d9%2F2025-03-11%2017.59.31.png?generation=1741683660337525&alt=media\")
Could there be an issue with the code that everyone is sharing..? Could it be a problem with the API provided by Jane Street, or has there been some change in the updated data?
", "rawMarkdown": "Could there be an issue with the code that everyone is sharing..? Could it be a problem with the API provided by Jane Street, or has there been some change in the updated data?" } ] }, { "id": 3145689, "postDate": "2025-03-10T06:37:41.857Z", "content": "Our notebook also appears through the first update of the test set, but not through the second update, whether the second update changes the format of the test set @ryanholbrook ![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F21506685%2Fc2ab150feabc8b9eca04b302604aa701%2FSnipaste_2025-03-10_14-34-42.png?generation=1741588499856585&alt=media\")
I think dataset or model is missing or deleted check your notebook @liangfenglin
", "rawMarkdown": "I think dataset or model is missing or deleted check your notebook @liangfenglin ", "replies": [ { "id": 3146973, "postDate": "2025-03-11T14:19:48.080Z", "content": "I also checked my notebook, and I believe there's no issue with my code.. It’s quite a frustrating situation.. 🙁
", "rawMarkdown": "I also checked my notebook, and I believe there's no issue with my code.. It’s quite a frustrating situation.. 🙁" } ] } ] }, { "id": 3134458, "postDate": "2025-02-26T11:16:14.937Z", "content": "I would appreciate it if you could reply.
", "rawMarkdown": "I would appreciate it if you could reply." }, { "id": 3147562, "postDate": "2025-03-12T06:19:40.067Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 3155737, "author_name": "InkWhisper", "author_url": "", "post_date": "2025-03-21T11:05:14.403000", "content": "I have encountered similar issues in other competitions. It’s possible that certain edge cases in the real-time data—whether due to unexpected data format changes, irregular time series intervals, or specific forecasting rules—caused the system to skip final scoring. Even when the submission logs showed “Succeeded,” there may have been hidden validation checks or internal constraints, such as a limit on data availability or acceptable input structures, that resulted in an incomplete evaluation process. If these edge cases weren’t clearly outlined in the competition rules, teams might inadvertently submit solutions that appear valid yet are never fully scored.
", "votes": 3, "replies": [] }, { "id": 3136493, "author_name": "I2nfinit3y", "author_url": "", "post_date": "2025-02-28T15:32:27.943000", "content": "There is something same with our team. In the first LB update, our 2 submission were both evaluated. But in the second LB update, one of submissions were not evaluated. So now our team have only 1 submission. :(
", "votes": 3, "replies": [ { "id": 3136966, "author_name": "Chan Lee", "author_url": "", "post_date": "2025-03-01T03:48:24.407000", "content": "Still, it’s fortunate that one of your team’s two submissions survived! Both of our submissions weren’t scored, so we ended up unranked…😭
", "votes": 0, "replies": [] } ] }, { "id": 3134535, "author_name": "SLi", "author_url": "", "post_date": "2025-02-26T13:30:00.270000", "content": "The \"Succeeded\" ones shwon in your screenshot are the ones that passed the first phase e.g. before the close of public LB. They have nothing to do with the LB update.
\nIn the LB update, the two chosen submission is re-run on the entire dataset and creates two new submissions with comments Notebook Rerun for Submission [] on 02/05/2025 under it. The other unchosen ones remain as it was.
After we entered the forecasting phase, there was one update on 02/05/2025 (the first update). For this update, you should see the two chosen submission, namely \"Lucky Number 456\" and \"lightweighting by removing qxgb n cat fold\", got re-run and have Notebook Rerun for Submission [] on 02/05/2025 comment under it.
In your screenshot, the two chosen submission did not pass the first LB update. This would be the reason why your team did not get evaluated in the second update (which was on 02/11/2025).
", "votes": 3, "replies": [ { "id": 3134570, "author_name": "Chan Lee", "author_url": "", "post_date": "2025-02-26T14:01:37.190000", "content": "Thank you for your thoughtful explanation. Our team also experienced issues with the rerun not being processed correctly during the February 5th update. Since then, we have been trying to identify the cause, but we were unable to figure it out on our own, which is why we have only now decided to post in the discussion. We are not sure what went wrong, and it is quite frustrating that our submissions were not properly evaluated.🥺
", "votes": 0, "replies": [] } ] }, { "id": 3152648, "author_name": "Rick Zhu", "author_url": "", "post_date": "2025-03-18T03:07:54.623000", "content": "I also suffered from this problem. My notebook didn't pass the first rerun. It shows a note saying
\nNotebook Rerun for Submission XXX on 02/11/2025, and has an Error sign in the begining. So I remained unrank.
\nIs it because my online retrain used up all the time? Or maybe some error occurred during private evaluated?
I tried to rerun my notebook on Kaggle, and found that one of the models is no longer public, which leads to an Error while running the notebook. I think this maybe the reason why my code can't be evaluated. So sad…😢
", "votes": 1, "replies": [ { "id": 3152979, "author_name": "Chan Lee", "author_url": "", "post_date": "2025-03-18T10:18:55.597000", "content": "Oh, I see… I’m sorry to hear that 🥺 I wonder if my solution also failed for the same reason… It would have been better if there were more conditions regarding the use of public notebooks as the final solution.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3145988, "author_name": "Vladimir Demidov", "author_url": "", "post_date": "2025-03-10T12:53:13.943000", "content": "I'm wondering whether or not public notebooks that are added to the input are restricted from further updates. In the case of datasets, there is no issue because it was pinned once it was attached.
", "votes": 1, "replies": [ { "id": 3146969, "author_name": "Chan Lee", "author_url": "", "post_date": "2025-03-11T14:16:40.703000", "content": "A solution that didn't use any public notebook resources at all also encountered a problem!😭
", "votes": 0, "replies": [] } ] }, { "id": 3134465, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2025-02-26T11:28:59.480000", "content": "@donghuilim it is possible that your code created an error in the evaluation phase causing this problem
", "votes": 1, "replies": [ { "id": 3134475, "author_name": "Chan Lee", "author_url": "", "post_date": "2025-02-26T11:51:21.190000", "content": "If there had been a potential issue with the code of one of our team members, it would not have caused problems for all of our team's submissions during the evaluation phase. Some of our submissions were based on different baselines, and some were submitted individually without coordination among team members. We truly do not know what the issue is. From what I have seen in other discussion posts about this competition, it appears that other teams are experiencing similar issues as well. We genuinely have no idea what might be causing this.
", "votes": 0, "replies": [] } ] }, { "id": 3146799, "author_name": "takumi71", "author_url": "", "post_date": "2025-03-11T09:01:07.150000", "content": "There is something same with me.
\nIn the first LB update, my 2 submission were both evaluated. But in the LB update in March, both of my submissions were not evaluated.
\n
Could there be an issue with the code that everyone is sharing..? Could it be a problem with the API provided by Jane Street, or has there been some change in the updated data?
", "votes": 0, "replies": [] } ] }, { "id": 3145689, "author_name": "", "author_url": "", "post_date": "2025-03-10T06:37:41.857000", "content": "Our notebook also appears through the first update of the test set, but not through the second update, whether the second update changes the format of the test set @ryanholbrook
I think dataset or model is missing or deleted check your notebook @liangfenglin
", "votes": 0, "replies": [ { "id": 3146973, "author_name": "Chan Lee", "author_url": "", "post_date": "2025-03-11T14:19:48.080000", "content": "I also checked my notebook, and I believe there's no issue with my code.. It’s quite a frustrating situation.. 🙁
", "votes": 0, "replies": [] } ] } ] }, { "id": 3134458, "author_name": "DONGHUI LIM", "author_url": "", "post_date": "2025-02-26T11:16:14.937000", "content": "I would appreciate it if you could reply.
", "votes": 0, "replies": [] }, { "id": 3147562, "author_name": "", "author_url": "", "post_date": "2025-03-12T06:19:40.067000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3134452": "Hello Kaggle Team, Kaggle Community, and Competition hosts,\n\nOur team participated in the Jane Street Real-Time Market Data Forecasting competition, and we encountered a critical issue during the forecasting period where none of our submissions were properly scored on the Private Leaderboard.\n\nDuring the Public Leaderboard phase, our submissions were successfully evaluated, and we had no issues. However, once the competition transitioned into the forecasting period, every submission we made—whether it was our own developed solution, a publicly available notebook solution, or a combination of both—failed to be scored correctly. This happened regardless of whether we selected those solutions as our final submissions.\n\nThe submission logs indicate that all our submissions were marked as \"Succeeded,\" yet they were not evaluated on the Private LB. The attached image provides evidence of this issue.\n\n\nWe want to clarify that our team did not engage in any rule violations or unethical practices. Given that multiple solutions were affected, we believe this could be a technical issue rather than a problem specific to our team.\n\nCould the Kaggle team and Community please investigate this matter and provide clarification on why our submissions were not evaluated on the Private LB? We would appreciate any insights or possible resolutions.", "3155737": "I have encountered similar issues in other competitions. It’s possible that certain edge cases in the real-time data—whether due to unexpected data format changes, irregular time series intervals, or specific forecasting rules—caused the system to skip final scoring. Even when the submission logs showed “Succeeded,” there may have been hidden validation checks or internal constraints, such as a limit on data availability or acceptable input structures, that resulted in an incomplete evaluation process. If these edge cases weren’t clearly outlined in the competition rules, teams might inadvertently submit solutions that appear valid yet are never fully scored.", "3136493": "There is something same with our team. In the first LB update, our 2 submission were both evaluated. But in the second LB update, one of submissions were not evaluated. So now our team have only 1 submission. :(", "3134535": "The \"Succeeded\" ones shwon in your screenshot are the ones that passed the first phase e.g. before the close of public LB. They have nothing to do with the LB update.\n\nIn the LB update, the two chosen submission is re-run on the entire dataset and creates two new submissions with comments `Notebook Rerun for Submission [] on 02/05/2025` under it. The other unchosen ones remain as it was.\n\nAfter we entered the forecasting phase, there was one update on 02/05/2025 (the first update). For this update, you should see the two chosen submission, namely \"Lucky Number 456\" and \"lightweighting by removing qxgb n cat fold\", got re-run and have `Notebook Rerun for Submission [] on 02/05/2025` comment under it. \n\nIn your screenshot, the two chosen submission did not pass the first LB update. This would be the reason why your team did not get evaluated in the second update (which was on 02/11/2025).", "3152648": "I also suffered from this problem. My notebook didn't pass the first rerun. It shows a note saying\n`Notebook Rerun for Submission XXX on 02/11/2025`, and has an Error sign in the begining. So I remained unrank. \nIs it because my online retrain used up all the time? Or maybe some error occurred during private evaluated?", "3145988": "I'm wondering whether or not public notebooks that are added to the input are restricted from further updates. In the case of datasets, there is no issue because it was pinned once it was attached.", "3134465": "@donghuilim it is possible that your code created an error in the evaluation phase causing this problem ", "3146799": "There is something same with me.\nIn the first LB update, my 2 submission were both evaluated. But in the LB update in March, both of my submissions were not evaluated.\n", "3145689": "Our notebook also appears through the first update of the test set, but not through the second update, whether the second update changes the format of the test set @ryanholbrook  ", "3134458": "I would appreciate it if you could reply.", "3147562": "" } }