Hi everyone,
\nTo help ensure you have the most up-to-date data to test your submissions against, we'll be extending the public test set with an additional ~80 days of data. The extension is continuous with the original set. The symbol ids and the number of time ids per date are both unchanged.
\nWe'll be rerunning all submissions against this updated data. As we currently have thousands of submissions to rerun, expect this to take at least several days. Once the rerun is complete, the leaderboard will show your updated score.
\nI hope to kick off the update later today. Please let us know if you have any questions or comments!
\nUPDATE 12/09 11:15 EST: Submissions temporarily paused while data is updated. Will turn on again shortly.
\nUPDATE 12/09 13:15 EST: The data update is complete. Submissions are live again. New submissions will be scored against the updated data. I will be kicking off the rerun of existing submissions soon. Expect this to take a while.
\nUPDATE 12/12 09:30 EST: I'll be releasing the first set of rescores soon. This set only contains those submissions manually chosen as \"selected\" in the Submissions tab, but not those which had been automatically selected at the time of the data update.
\nI am continuing the rescore with the another set of submissions, which I hope will be completed within a few days. Note that these rescores have to be done in bulk, which is why scores aren't updated incrementally.
", "messages": [ { "id": 3067689, "postDate": "2024-12-09T14:33:36.003Z", "content": "Hi everyone,
\nTo help ensure you have the most up-to-date data to test your submissions against, we'll be extending the public test set with an additional ~80 days of data. The extension is continuous with the original set. The symbol ids and the number of time ids per date are both unchanged.
\nWe'll be rerunning all submissions against this updated data. As we currently have thousands of submissions to rerun, expect this to take at least several days. Once the rerun is complete, the leaderboard will show your updated score.
\nI hope to kick off the update later today. Please let us know if you have any questions or comments!
\nUPDATE 12/09 11:15 EST: Submissions temporarily paused while data is updated. Will turn on again shortly.
\nUPDATE 12/09 13:15 EST: The data update is complete. Submissions are live again. New submissions will be scored against the updated data. I will be kicking off the rerun of existing submissions soon. Expect this to take a while.
\nUPDATE 12/12 09:30 EST: I'll be releasing the first set of rescores soon. This set only contains those submissions manually chosen as \"selected\" in the Submissions tab, but not those which had been automatically selected at the time of the data update.
\nI am continuing the rescore with the another set of submissions, which I hope will be completed within a few days. Note that these rescores have to be done in bulk, which is why scores aren't updated incrementally.
", "rawMarkdown": "Hi everyone,\n\nTo help ensure you have the most up-to-date data to test your submissions against, we'll be extending the public test set with an additional ~80 days of data. The extension is continuous with the original set. The symbol ids and the number of time ids per date are both unchanged.\n\nWe'll be rerunning all submissions against this updated data. As we currently have thousands of submissions to rerun, expect this to take at least several days. Once the rerun is complete, the leaderboard will show your updated score.\n\nI hope to kick off the update later today. Please let us know if you have any questions or comments!\n\n**UPDATE 12/09 11:15 EST:** Submissions temporarily paused while data is updated. Will turn on again shortly.\n\n**UPDATE 12/09 13:15 EST:** The data update is complete. Submissions are live again. New submissions will be scored against the updated data. I will be kicking off the rerun of existing submissions soon. Expect this to take a while.\n\n**UPDATE 12/12 09:30 EST:** I'll be releasing the first set of rescores soon. This set only contains those submissions manually chosen as \"selected\" in the Submissions tab, but not those which had been automatically selected at the time of the data update.\n\nI am continuing the rescore with the another set of submissions, which I hope will be completed within a few days. Note that these rescores have to be done in bulk, which is why scores aren't updated incrementally.", "votes": 54 }, { "id": 3088359, "postDate": "2025-01-04T15:35:45.010Z", "content": "Got longer than 9 hour submission time (which means that the submission was waiting for resource and did not start) and kaggle error, will this happen also in the last days?
\nupdate: facing this issue for all the recent submissions
", "rawMarkdown": "Got longer than 9 hour submission time (which means that the submission was waiting for resource and did not start) and kaggle error, will this happen also in the last days?\n\nupdate: facing this issue for all the recent submissions", "votes": 14, "replies": [ { "id": 3088403, "postDate": "2025-01-04T16:16:59.047Z", "content": "If the reason for this is the rerunning of old submissions on the new data, it would be nice to consider delaying it until after the competition ends. @ryanholbrook
", "rawMarkdown": "If the reason for this is the rerunning of old submissions on the new data, it would be nice to consider delaying it until after the competition ends. @ryanholbrook ", "votes": 2 }, { "id": 3088953, "postDate": "2025-01-05T11:23:27.563Z", "content": "I've encountered into the same issue. Yesterday my code took only 4 hours but today takes 9 hours. What happens to the
\napi server?
I have been experiencing the same phenomenon for the past three days. submit, which takes about 2h to finish, did not finish even after 8h, and I got a Notebook Inference Server Error.
\n@ryanholbrook
I've been experiencing the same issue: submissions vastly exceeding the 9-hour limit (some completing after 16 hours) with sporadic successes
", "rawMarkdown": "I've been experiencing the same issue: submissions vastly exceeding the 9-hour limit (some completing after 16 hours) with sporadic successes" } ] } ] }, { "id": 3089771, "postDate": "2025-01-06T14:10:35.430Z", "content": "@ryanholbrook - Can you please check the high submission running time issue as highlighted by others. All 4 notebooks of mine from last evening has gone much over the estimated time and timed out. Thanks
", "rawMarkdown": "@ryanholbrook - Can you please check the high submission running time issue as highlighted by others. All 4 notebooks of mine from last evening has gone much over the estimated time and timed out. Thanks", "votes": 7 }, { "id": 3068211, "postDate": "2024-12-10T04:51:05.500Z", "content": "My submission, which is a very simple notebook (just for testing the end-to-end submission process), has been taking over 5 hours to score. Could this delay be due to backfilling and rerunning existing submissions?
", "rawMarkdown": "My submission, which is a very simple notebook (just for testing the end-to-end submission process), has been taking over 5 hours to score. Could this delay be due to backfilling and rerunning existing submissions?", "votes": 10, "replies": [ { "id": 3068288, "postDate": "2024-12-10T05:57:07.123Z", "content": "Same here. I thought it was my bug X_X
", "rawMarkdown": "Same here. I thought it was my bug X_X", "votes": 2 }, { "id": 3068346, "postDate": "2024-12-10T07:11:45.343Z", "content": "Same here, stuck in scoring.
", "rawMarkdown": "Same here, stuck in scoring.", "votes": 4 }, { "id": 3068457, "postDate": "2024-12-10T09:34:33.957Z", "content": "Same for me as well. Stuck in scoring for the last 10 hours :(
", "rawMarkdown": "Same for me as well. Stuck in scoring for the last 10 hours :(", "votes": 2, "replies": [ { "id": 3068548, "postDate": "2024-12-10T12:00:54.920Z", "content": "are you guys still facing the problem?
", "rawMarkdown": "are you guys still facing the problem?", "votes": 2 }, { "id": 3068554, "postDate": "2024-12-10T12:07:08.213Z", "content": "My submission has been waiting for a score for 13 hours.
", "rawMarkdown": "My submission has been waiting for a score for 13 hours.", "votes": 1 } ] }, { "id": 3068572, "postDate": "2024-12-10T12:32:14.487Z", "content": "Hi @daebak2021 (and others) I will check up on it. It might be that submissions are getting queued due to compute quotas.
", "rawMarkdown": "Hi @daebak2021 (and others) I will check up on it. It might be that submissions are getting queued due to compute quotas.", "votes": 4, "replies": [ { "id": 3068776, "postDate": "2024-12-10T16:42:24.843Z", "content": "I have a submission pending for 13 hours due to the update, will it get rerun if it fails?
", "rawMarkdown": "I have a submission pending for 13 hours due to the update, will it get rerun if it fails?" }, { "id": 3068833, "postDate": "2024-12-10T17:42:08.340Z", "content": "My submissions ran for ~16 hours and then failed with \"Kaggle error\"
", "rawMarkdown": "My submissions ran for ~16 hours and then failed with \"Kaggle error\"", "votes": 2 }, { "id": 3068966, "postDate": "2024-12-10T22:29:59.523Z", "content": "Same here, still facing the problem. Made 5 submissions yesterday, and all are 9h+ now.
", "rawMarkdown": "Same here, still facing the problem. Made 5 submissions yesterday, and all are 9h+ now.", "votes": 1 }, { "id": 3069057, "postDate": "2024-12-11T02:35:46.670Z", "content": "Same here!
", "rawMarkdown": "Same here!\n" }, { "id": 3069538, "postDate": "2024-12-11T15:43:04.737Z", "content": "Did this work for anyone? I'm still having the same issue as you guys
", "rawMarkdown": "Did this work for anyone? I'm still having the same issue as you guys", "isDeleted": true }, { "id": 3069699, "postDate": "2024-12-11T19:16:59.373Z", "content": "Hasn't worked for me yet, I am still having the same issue.
", "rawMarkdown": "Hasn't worked for me yet, I am still having the same issue. " } ] } ] }, { "id": 3087593, "postDate": "2025-01-03T16:32:51.757Z", "content": "I got a question here. And much thanks to anyone could answer my question:
\nThe rule said that the inference time should be within 1 minute. But since for a successful public leaderboard submission, we got a limit of 8-hours running time. And the validation set in leaderboard has nearly 200 days. So the actual inference time should be completed in 480*60 = 28800s / （900 time id for each date *200） = 0.16s. Am I right? @ryanholbrook @yuanzhezhou
", "rawMarkdown": "I got a question here. And much thanks to anyone could answer my question:\n\nThe rule said that the inference time should be within 1 minute. But since for a successful public leaderboard submission, we got a limit of 8-hours running time. And the validation set in leaderboard has nearly 200 days. So the actual inference time should be completed in 480*60 = 28800s / （900 time id for each date *200） = 0.16s. Am I right? @ryanholbrook @yuanzhezhou ", "votes": 1, "replies": [ { "id": 3087595, "postDate": "2025-01-03T16:36:47.613Z", "content": "0.16s to complete all things including feature engineering and inference
", "rawMarkdown": "0.16s to complete all things including feature engineering and inference", "votes": -1 } ] }, { "id": 3096475, "postDate": "2025-01-14T12:43:28.687Z", "content": "How often can we expect leaderboard updates now during the forecasting phase?
", "rawMarkdown": "How often can we expect leaderboard updates now during the forecasting phase?", "votes": 2 }, { "id": 3095976, "postDate": "2025-01-14T00:41:26.783Z", "content": "How often can we expect leaderboard updates now in forecasting phase?
", "rawMarkdown": "How often can we expect leaderboard updates now in forecasting phase?", "votes": 2, "replies": [ { "id": 3097497, "postDate": "2025-01-15T12:42:59.450Z", "content": "seemingly 15d? But I forget where I saw it.
", "rawMarkdown": "seemingly 15d? But I forget where I saw it.", "replies": [ { "id": 3099182, "postDate": "2025-01-17T12:09:13.620Z", "content": "I dont see it anywhere. But previous Jane Street competition seems to be updated every 2 weeks: https://www.kaggle.com/competitions/jane-street-market-prediction/discussion/226745
", "rawMarkdown": "I dont see it anywhere. But previous Jane Street competition seems to be updated every 2 weeks: https://www.kaggle.com/competitions/jane-street-market-prediction/discussion/226745" }, { "id": 3099184, "postDate": "2025-01-17T12:16:09.097Z", "rawMarkdown": "", "votes": 3, "isDeleted": true }, { "id": 3099800, "postDate": "2025-01-18T09:54:28.747Z", "content": "We are planning to update the private leaderboard roughly every month. The second update of public learderboard should happen before it but won't count into the private score.
", "rawMarkdown": "We are planning to update the private leaderboard roughly every month. The second update of public learderboard should happen before it but won't count into the private score.", "votes": 8 }, { "id": 3100180, "postDate": "2025-01-18T22:16:30.807Z", "content": "\n\nWe are planning to update the private leaderboard roughly every month. The second update of public learderboard should happen before it but won't count into the private score.
\n
So if I'm understanding correctly, the public leaderboard will be updated once more with the rest of the data up until the end date of the first phase, then the private leaderboard will be updated periodically with data collected after the second phase starts and will only score the newly collected data.
\nWill we be able to see the private leaderboard? If so, will that replace the public leaderboard that we see now, or will we be able to see both the public and private leaderboards?
", "rawMarkdown": ">We are planning to update the private leaderboard roughly every month. The second update of public learderboard should happen before it but won't count into the private score.\n\nSo if I'm understanding correctly, the public leaderboard will be updated once more with the rest of the data up until the end date of the first phase, then the private leaderboard will be updated periodically with data collected after the second phase starts and will only score the newly collected data. \n\nWill we be able to see the private leaderboard? If so, will that replace the public leaderboard that we see now, or will we be able to see both the public and private leaderboards?" }, { "id": 3100499, "postDate": "2025-01-19T12:59:56.473Z", "content": "Hopefully, a ‘late submission’ feature or some submission API could be made available for lower-ranked participants like me to track our improvement. Many of us are still working on preparing ourselves for future JS competitions. Of course, these submissions wouldn’t affect the existing leaderboard or rankings. Thanks for sharing your solution 4 years ago that I am still learning from, and thanks for hosting this competition.
", "rawMarkdown": "Hopefully, a ‘late submission’ feature or some submission API could be made available for lower-ranked participants like me to track our improvement. Many of us are still working on preparing ourselves for future JS competitions. Of course, these submissions wouldn’t affect the existing leaderboard or rankings. Thanks for sharing your solution 4 years ago that I am still learning from, and thanks for hosting this competition.", "votes": -2 }, { "id": 3102143, "postDate": "2025-01-21T18:25:50.137Z", "content": "@gogo827jz thanks for the information. when is the public LB update?
", "rawMarkdown": "@gogo827jz thanks for the information. when is the public LB update?", "votes": 1 } ] } ] }, { "id": 3070815, "postDate": "2024-12-13T04:20:25.677Z", "content": "My previous selected submissions have not yet been processed. Will the score remain the same, or are there any issues when rerunning the notebook, such as timeouts or out-of-memory errors?
", "rawMarkdown": "My previous selected submissions have not yet been processed. Will the score remain the same, or are there any issues when rerunning the notebook, such as timeouts or out-of-memory errors? ", "votes": 1, "replies": [ { "id": 3071525, "postDate": "2024-12-13T21:20:04.973Z", "content": "If they haven't been updated, it's likely they failed the rescore. There was nothing to publish in that case.
", "rawMarkdown": "If they haven't been updated, it's likely they failed the rescore. There was nothing to publish in that case.", "votes": 1 } ] }, { "id": 3069024, "postDate": "2024-12-11T01:00:50.890Z", "content": "Will the previous submissions' score be updated? It does not seem to be gradually updating.
\nupdate: now updating
", "rawMarkdown": "Will the previous submissions' score be updated? It does not seem to be gradually updating.\n\nupdate: now updating", "votes": 2, "replies": [ { "id": 3069046, "postDate": "2024-12-11T01:50:37.020Z", "content": "Same question …
", "rawMarkdown": "Same question ... ", "votes": 1 }, { "id": 3069263, "postDate": "2024-12-11T08:57:04.470Z", "content": "you can just resubmit for faster scoring
", "rawMarkdown": "you can just resubmit for faster scoring", "replies": [ { "id": 3069400, "postDate": "2024-12-11T12:55:30.727Z", "content": "Unfortunately, resubmit didn't work for me.
", "rawMarkdown": "Unfortunately, resubmit didn't work for me.", "votes": 1 } ] } ] }, { "id": 3089864, "postDate": "2025-01-06T16:07:23.430Z", "content": "What does Score mean in leaderboard, can someone please clarify what metric is that?
", "rawMarkdown": "What does Score mean in leaderboard, can someone please clarify what metric is that?\n", "votes": -7 }, { "id": 3085630, "postDate": "2025-01-01T11:45:42.070Z", "content": "What is the relevance of tags in the features.csv and responder.csv files?
", "rawMarkdown": "What is the relevance of tags in the features.csv and responder.csv files?", "votes": -1 }, { "id": 3081423, "postDate": "2024-12-26T17:37:14.113Z", "content": "Hi,I am sorry. I am unable to score. The script is paused for several hours. Can you do something about it?
\nThanks.
In the hidden test set will there be responder_6 column also ? We have to only calculate and submit the output.parquet which we get after running the trained model on test.parquet right ?
", "rawMarkdown": "In the hidden test set will there be responder_6 column also ? We have to only calculate and submit the output.parquet which we get after running the trained model on test.parquet right ?", "votes": -6, "replies": [ { "id": 3083283, "postDate": "2024-12-29T08:40:51.670Z", "content": "Yes, in the hidden test set, there should be a responder_6 column, which is typically used to indicate whether a customer responded to a particular offer or not. You are correct that your task is to run your trained model on the test.parquet file, which includes the responder_6 column and other relevant features.
", "rawMarkdown": "Yes, in the hidden test set, there should be a responder_6 column, which is typically used to indicate whether a customer responded to a particular offer or not. You are correct that your task is to run your trained model on the test.parquet file, which includes the responder_6 column and other relevant features.", "replies": [ { "id": 3085626, "postDate": "2025-01-01T11:43:39.203Z", "content": "Thanks bro .
", "rawMarkdown": "Thanks bro ." }, { "id": 3090956, "postDate": "2025-01-07T20:28:16.270Z", "content": "hmm, as far as I understand, test file contains: row_id, date_id, time_id, symbol_id, weight, is_scored and features_xx (00 to 78) columns, while lags file contains: date_id, time_id, symbol_id, responder_x_lag_1 (0 to 8) columns and we should use kaggle_evaluation and predict function …?
", "rawMarkdown": "hmm, as far as I understand, test file contains: row_id, date_id, time_id, symbol_id, weight, is_scored and features_xx (00 to 78) columns, while lags file contains: date_id, time_id, symbol_id, responder_x_lag_1 (0 to 8) columns and we should use kaggle_evaluation and predict function ...?\t" } ] } ] }, { "id": 3079267, "postDate": "2024-12-23T13:17:55.657Z", "content": "Why in the test.parquet file all rows are having only values as zero everywhere ?
", "rawMarkdown": "Why in the test.parquet file all rows are having only values as zero everywhere ?", "votes": -7 }, { "id": 3079209, "postDate": "2024-12-23T11:24:28.567Z", "content": "Why does the test.parquet file does not contain any responder features ?
", "rawMarkdown": "Why does the test.parquet file does not contain any responder features ?", "votes": -7 }, { "id": 3079170, "postDate": "2024-12-23T09:37:14.563Z", "content": "The train.parquet file is solely responsible for building the model ? Like do we have to do the train test split on train.parquet only for building the model and then submit ?
", "rawMarkdown": "The train.parquet file is solely responsible for building the model ? Like do we have to do the train test split on train.parquet only for building the model and then submit ?", "votes": -6 }, { "id": 3077528, "postDate": "2024-12-21T05:03:22.417Z", "content": "What about the hidden test set?
", "rawMarkdown": "What about the hidden test set?", "votes": -2 }, { "id": 3097814, "postDate": "2025-01-15T18:10:44.840Z", "content": "When wil it be updated?
", "rawMarkdown": "When wil it be updated?" }, { "id": 3097119, "postDate": "2025-01-15T04:26:06.843Z", "content": "Will everyone's score be updated in the forecasting phase using their selected notebooks, or just a selected amount of people's notebook can be updated?
", "rawMarkdown": "Will everyone's score be updated in the forecasting phase using their selected notebooks, or just a selected amount of people's notebook can be updated?" }, { "id": 3097068, "postDate": "2025-01-15T02:32:46.757Z", "content": "During the forecast phase, which leaderboard will be updated: the public or the private?
\nI hope the private leaderboard will be updated while the public leaderboard remains unchanged.
The extension of the public test set sounds really helpful. One thing I’m wondering about—is there a chance the additional data will cause any major changes in the nature of the data or the features we’re working with?
", "rawMarkdown": "The extension of the public test set sounds really helpful. One thing I’m wondering about—is there a chance the additional data will cause any major changes in the nature of the data or the features we’re working with? " }, { "id": 3093871, "postDate": "2025-01-11T12:41:38.997Z", "content": "Is there any clarity on the final structure of training data; I assume that the date id's will be updated to the 14th of January with the submission date rolling out from there. This will insure that the date ids are continous? Any thoughts, anyone?
", "rawMarkdown": "Is there any clarity on the final structure of training data; I assume that the date id's will be updated to the 14th of January with the submission date rolling out from there. This will insure that the date ids are continous? Any thoughts, anyone?" }, { "id": 3087816, "postDate": "2025-01-03T23:22:22.013Z", "content": "@ryanholbrook I have a question regarding the 9-hour time limit at the end of the forecasting phase. Is this limit cumulative only for the execution time of each predict() call? I’ve noticed that the time taken between predict() calls (possibly for processing the input data frame) is often longer than the actual execution time of predict() itself. Is anyone clear about how the 9-hour limit is calculated? I believe it would be reasonable to consider only the time spent in predict(). Thanks for any information!
", "rawMarkdown": "@ryanholbrook I have a question regarding the 9-hour time limit at the end of the forecasting phase. Is this limit cumulative only for the execution time of each predict() call? I’ve noticed that the time taken between predict() calls (possibly for processing the input data frame) is often longer than the actual execution time of predict() itself. Is anyone clear about how the 9-hour limit is calculated? I believe it would be reasonable to consider only the time spent in predict(). Thanks for any information!", "replies": [ { "id": 3091973, "postDate": "2025-01-09T03:09:26.007Z", "content": "i 've the same question
", "rawMarkdown": "i 've the same question\n" } ] }, { "id": 3083364, "postDate": "2024-12-29T11:32:19.770Z", "content": "\n\nAdditionally, new symbol_id values may appear in future test sets.
\n
This was stated in the data page. Will there be any new symbol_id added to the test set? This is very concerning. It could potentially cause many submission fail.
", "rawMarkdown": ">Additionally, new symbol_id values may appear in future test sets.\n\nThis was stated in the data page. Will there be any new symbol_id added to the test set? This is very concerning. It could potentially cause many submission fail. ", "replies": [ { "id": 3083553, "postDate": "2024-12-29T16:40:44.277Z", "content": "The official notebook said it will, I think we need concerning it🤣
", "rawMarkdown": "The official notebook said it will, I think we need concerning it🤣" } ] }, { "id": 3080661, "postDate": "2024-12-25T14:33:25.457Z", "content": "Got it but what about the hidden test set?
", "rawMarkdown": "Got it but what about the hidden test set?" }, { "id": 3080031, "postDate": "2024-12-24T14:25:52.080Z", "content": "I have a problem like \"Server did not register a listener for predict\". How could I deal with it ?
\n/kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py in get_all_predictions(self)
\n 51 all_predictions = []
\n 52 for data_batch, validation_batch in self.generate_data_batches():
\n---> 53 predictions = self.predict(*data_batch)
\n 54 self.validate_prediction_batch(predictions, validation_batch)
\n 55 all_predictions.append(predictions)
/kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py in predict(self, *args, **kwargs)
\n 64 return self.client.send('predict', *args, **kwargs)
\n 65 except Exception as e:
\n---> 66 self.handle_server_error(e, 'predict')
\n 67
\n 68 def set_response_timeout_seconds(self, timeout_seconds: float=6_000):
/kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/base_gateway.py in handle_server_error(self, exception, endpoint)
\n 184 raise GatewayRuntimeError(GatewayRuntimeErrorType.SERVER_NEVER_STARTED) from None
\n 185 if f'No listener for {endpoint} was registered' in exception_str:
\n--> 186 raise GatewayRuntimeError(GatewayRuntimeErrorType.SERVER_MISSING_ENDPOINT, f'Server did not register a listener for {endpoint}') from None
\n 187 if 'Exception calling application' in exception_str:
\n 188 # Extract just the exception message raised by the inference server
GatewayRuntimeError: (, 'Server did not register a listener for predict')
", "rawMarkdown": "I have a problem like \"Server did not register a listener for predict\". How could I deal with it ?\n/kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py in get_all_predictions(self)\n 51 all_predictions = []\n 52 for data_batch, validation_batch in self.generate_data_batches():\n---> 53 predictions = self.predict(*data_batch)\n 54 self.validate_prediction_batch(predictions, validation_batch)\n 55 all_predictions.append(predictions)\n\n/kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py in predict(self, *args, **kwargs)\n 64 return self.client.send('predict', *args, **kwargs)\n 65 except Exception as e:\n---> 66 self.handle_server_error(e, 'predict')\n 67 \n 68 def set_response_timeout_seconds(self, timeout_seconds: float=6_000):\n\n/kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/base_gateway.py in handle_server_error(self, exception, endpoint)\n 184 raise GatewayRuntimeError(GatewayRuntimeErrorType.SERVER_NEVER_STARTED) from None\n 185 if f'No listener for {endpoint} was registered' in exception_str:\n--> 186 raise GatewayRuntimeError(GatewayRuntimeErrorType.SERVER_MISSING_ENDPOINT, f'Server did not register a listener for {endpoint}') from None\n 187 if 'Exception calling application' in exception_str:\n 188 # Extract just the exception message raised by the inference server\n\nGatewayRuntimeError: (Do you provide time series data in the test set? The example data (from test.parquet) only has one day and one time_id with 39 different symbols. Does each day contains one time_id or multiple time_ids for that day in both the public and private test sets.
", "rawMarkdown": "Do you provide time series data in the test set? The example data (from test.parquet) only has one day and one time_id with 39 different symbols. Does each day contains one time_id or multiple time_ids for that day in both the public and private test sets. " }, { "id": 3074299, "postDate": "2024-12-17T14:38:33.547Z", "content": "Was 80 actual days or trading days added? / or 80 new date_id added?
", "rawMarkdown": "Was 80 actual days or trading days added? / or 80 new date_id added? ", "replies": [ { "id": 3074640, "postDate": "2024-12-17T21:59:31.283Z", "content": "It's about 80 new date_ids.
", "rawMarkdown": "It's about 80 new date_ids." } ] }, { "id": 3074022, "postDate": "2024-12-17T08:32:10.450Z", "content": "My kaggle kernel crashed from time to time while building training and validation sets, which was a bit distressing, but it was exciting to participate in a competition like this for the first time
", "rawMarkdown": "My kaggle kernel crashed from time to time while building training and validation sets, which was a bit distressing, but it was exciting to participate in a competition like this for the first time" }, { "id": 3071526, "postDate": "2024-12-13T21:22:36.567Z", "content": "Hi, is it possible to use pretrained model for this competition ?
", "rawMarkdown": "Hi, is it possible to use pretrained model for this competition ? ", "replies": [ { "id": 3085609, "postDate": "2025-01-01T11:29:28.967Z", "content": "Yes, just upload your trained model's result as a .pkl document and create a new dataset to upload it
", "rawMarkdown": "Yes, just upload your trained model's result as a .pkl document and create a new dataset to upload it\n", "replies": [ { "id": 3088564, "postDate": "2025-01-04T20:32:28.947Z", "content": "Hey, I am not sure I understand what you mean with \"create a new dataset to upload it\". Doesn't it make more sense to upload the model and load it to make the predictions?
", "rawMarkdown": "Hey, I am not sure I understand what you mean with \"create a new dataset to upload it\". Doesn't it make more sense to upload the model and load it to make the predictions?" } ] } ] }, { "id": 3070504, "postDate": "2024-12-12T18:38:39.143Z", "content": "Can you explain to me the logic of the leaderboard? I used to be #43 with a score of 0.0076, and then people who submitted later with the same score got above me in the leaderboard, so now I'm ~200. Why do people who submit later with the same score get the upper hand? Typically, it should be vice versa. With the same score, the person who submits earlier should be higher in the leaderboard.
", "rawMarkdown": "Can you explain to me the logic of the leaderboard? I used to be #43 with a score of 0.0076, and then people who submitted later with the same score got above me in the leaderboard, so now I'm ~200. Why do people who submit later with the same score get the upper hand? Typically, it should be vice versa. With the same score, the person who submits earlier should be higher in the leaderboard.", "replies": [ { "id": 3070533, "postDate": "2024-12-12T19:55:39.427Z", "content": "The scores on the leaderboard are truncated. So if you actually have a score of 0.00761, someone with an actual score of 0.00769 will have a better ranking even if they submitted after you.
", "rawMarkdown": "The scores on the leaderboard are truncated. So if you actually have a score of 0.00761, someone with an actual score of 0.00769 will have a better ranking even if they submitted after you.", "votes": 6 } ] }, { "id": 3070301, "postDate": "2024-12-12T14:26:19.883Z", "content": "Hi everyone,
\nPlease see the latest update in the post above.
", "rawMarkdown": "Hi everyone,\n\nPlease see the latest update in the post above." }, { "id": 3069872, "postDate": "2024-12-12T01:27:46.703Z", "content": "At first, i m trying to wait re-run and score updating, but nothing happed;
\nThen i m trying to re-submit my best score notebook, with a long long long time left, it fail …;
\nfinal i submit a open source notebook because its simple and fast, then i got difference score
\n……
\ni look like a 🤡
Can someone explain what they mean by the word \"public\"? Is this data visible or downloadable somewhere? If it's being used for the leaderboard scoring, by definition it is not actually public is it?
", "rawMarkdown": "Can someone explain what they mean by the word \"public\"? Is this data visible or downloadable somewhere? If it's being used for the leaderboard scoring, by definition it is not actually public is it?", "replies": [ { "id": 3068682, "postDate": "2024-12-10T14:49:27.473Z", "content": "public refers to the public LB where your LB score is visible after a submission. not that the data is shared public.
", "rawMarkdown": "public refers to the public LB where your LB score is visible after a submission. not that the data is shared public.", "votes": 2 } ] }, { "id": 3067828, "postDate": "2024-12-09T16:38:03.517Z", "content": "So in the final forecasting phase, will this public test set still be part of the full test set just unscored? So we need to account for a lot of extra time compared to current LB run time it seems..?
", "rawMarkdown": "So in the final forecasting phase, will this public test set still be part of the full test set just unscored? So we need to account for a lot of extra time compared to current LB run time it seems..?", "replies": [ { "id": 3067838, "postDate": "2024-12-09T16:52:21.687Z", "content": "Yes, correct. In the final forecasting phase, the test data will comprise all the dates from the start of the public test set to the end of the private test set. After competition close, there will be an additional portion of public data added to the public test set, and then an updating private set. After close, only the private set is scored, but the your submissions are run on the full, combined test set.
", "rawMarkdown": "Yes, correct. In the final forecasting phase, the test data will comprise all the dates from the start of the public test set to the end of the private test set. After competition close, there will be an additional portion of public data added to the public test set, and then an updating private set. After close, only the private set is scored, but the your submissions are run on the full, combined test set.", "votes": 4, "replies": [ { "id": 3067840, "postDate": "2024-12-09T16:57:54.990Z", "content": "Hi Ryan, for the is_score label, it should be expected that all batches within the public date range should have is_score=False, right? It would be weird to have one or a few batches (or even rows within a batch) having is_score=True while the rest to be false.
Thank you!
", "rawMarkdown": "Hi Ryan, for the `is_score` label, it should be expected that all batches within the public date range should have `is_score=False`, right? It would be weird to have one or a few batches (or even rows within a batch) having `is_score=True` while the rest to be false. \n\nThank you!", "votes": 1 }, { "id": 3067849, "postDate": "2024-12-09T17:15:56.677Z", "content": "After the competition closes, the entire public test set will be unscored.
", "rawMarkdown": "After the competition closes, the entire public test set will be unscored.", "votes": 1 }, { "id": 3069227, "postDate": "2024-12-11T07:46:18.297Z", "content": "Hi, how should our inference time be in order to not exceed the time limit during private lb scoring? I assume there are a lot of new data during that phase compared with the current submission
", "rawMarkdown": "Hi, how should our inference time be in order to not exceed the time limit during private lb scoring? I assume there are a lot of new data during that phase compared with the current submission ", "votes": 1 } ] } ] }, { "id": 3067711, "postDate": "2024-12-09T14:44:40.950Z", "content": "Hello, is the new dataset scored or unscored?
", "rawMarkdown": "Hello, is the new dataset scored or unscored?", "replies": [ { "id": 3067761, "postDate": "2024-12-09T15:23:16.543Z", "content": "This extension is scored as part of the public test set. I'll post here to let you know when the data has been updated and also when the rescores of existing submissions are available.
", "rawMarkdown": "This extension is scored as part of the public test set. I'll post here to let you know when the data has been updated and also when the rescores of existing submissions are available.", "votes": 1, "replies": [ { "id": 3067893, "postDate": "2024-12-09T18:12:58.173Z", "content": "Hello, do adding this extension data means the public LB will involve about 120 days from the first phase + 80 days of extension data, totaling around 200 days' data for scoring, while the final forecasting phase will only involve about 120(4.5m rows) days' data?
\nSo, at final forecasting phase, we will actually have more runtime available for each scored prediction?
During the forecasting phase, the test set will be the combined public and private sets, but only the private portion will be scored. This is to ensure the continuity of the dataset. We are going to increase the submission time limit during the forecasting phase, but the size of the test set will also be larger.
", "rawMarkdown": "During the forecasting phase, the test set will be the combined public and private sets, but only the private portion will be scored. This is to ensure the continuity of the dataset. We are going to increase the submission time limit during the forecasting phase, but the size of the test set will also be larger.", "votes": 1 }, { "id": 3068098, "postDate": "2024-12-10T00:49:01.210Z", "content": "Hi Ryan, just want to clarify - do you mean the time limit for forecasting phase will be increased before the competition close?
", "rawMarkdown": "Hi Ryan, just want to clarify - do you mean the time limit for forecasting phase will be increased before the competition close?" }, { "id": 3068122, "postDate": "2024-12-10T01:41:04.777Z", "content": "See the details in Code Requirements on the main page.
", "rawMarkdown": "See the details in [Code Requirements](https://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/overview/code-requirements) on the main page." }, { "id": 3068701, "postDate": "2024-12-10T15:12:51.850Z", "content": "What I want to confirm is that if I skip all the unscored rows, the current submission will predict data for around 200 days, while in the final forecasting phase, I will only need to predict data for around 120 days. So, the timeline is actually tighter right now, correct?
", "rawMarkdown": "What I want to confirm is that if I skip all the unscored rows, the current submission will predict data for around 200 days, while in the final forecasting phase, I will only need to predict data for around 120 days. So, the timeline is actually tighter right now, correct?" } ] } ] }, { "id": 3095959, "postDate": "2025-01-14T00:15:33.720Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3079699, "postDate": "2024-12-24T02:55:34.017Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3106459, "postDate": "2025-01-24T21:33:37.680Z", "content": "Nice thanks.
", "rawMarkdown": "Nice thanks." } ], "comments": [ { "id": 3088359, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2025-01-04T15:35:45.010000", "content": "Got longer than 9 hour submission time (which means that the submission was waiting for resource and did not start) and kaggle error, will this happen also in the last days?
\nupdate: facing this issue for all the recent submissions
", "votes": 14, "replies": [ { "id": 3088403, "author_name": "Evgeniia Grigoreva", "author_url": "", "post_date": "2025-01-04T16:16:59.047000", "content": "If the reason for this is the rerunning of old submissions on the new data, it would be nice to consider delaying it until after the competition ends. @ryanholbrook
", "votes": 2, "replies": [] }, { "id": 3088953, "author_name": "r_machida", "author_url": "", "post_date": "2025-01-05T11:23:27.563000", "content": "I've encountered into the same issue. Yesterday my code took only 4 hours but today takes 9 hours. What happens to the
\napi server?
I have been experiencing the same phenomenon for the past three days. submit, which takes about 2h to finish, did not finish even after 8h, and I got a Notebook Inference Server Error.
\n@ryanholbrook
I've been experiencing the same issue: submissions vastly exceeding the 9-hour limit (some completing after 16 hours) with sporadic successes
", "votes": 0, "replies": [] } ] } ] }, { "id": 3089771, "author_name": "Viji", "author_url": "", "post_date": "2025-01-06T14:10:35.430000", "content": "@ryanholbrook - Can you please check the high submission running time issue as highlighted by others. All 4 notebooks of mine from last evening has gone much over the estimated time and timed out. Thanks
", "votes": 7, "replies": [] }, { "id": 3068211, "author_name": "yb", "author_url": "", "post_date": "2024-12-10T04:51:05.500000", "content": "My submission, which is a very simple notebook (just for testing the end-to-end submission process), has been taking over 5 hours to score. Could this delay be due to backfilling and rerunning existing submissions?
", "votes": 10, "replies": [ { "id": 3068288, "author_name": "Jon", "author_url": "", "post_date": "2024-12-10T05:57:07.123000", "content": "Same here. I thought it was my bug X_X
", "votes": 2, "replies": [] }, { "id": 3068346, "author_name": "Ben Fung", "author_url": "", "post_date": "2024-12-10T07:11:45.343000", "content": "Same here, stuck in scoring.
", "votes": 4, "replies": [] }, { "id": 3068457, "author_name": "Orchid", "author_url": "", "post_date": "2024-12-10T09:34:33.957000", "content": "Same for me as well. Stuck in scoring for the last 10 hours :(
", "votes": 2, "replies": [ { "id": 3068548, "author_name": "policeman2323", "author_url": "", "post_date": "2024-12-10T12:00:54.920000", "content": "are you guys still facing the problem?
", "votes": 2, "replies": [] }, { "id": 3068554, "author_name": "Matias TorresR", "author_url": "", "post_date": "2024-12-10T12:07:08.213000", "content": "My submission has been waiting for a score for 13 hours.
", "votes": 1, "replies": [] } ] }, { "id": 3068572, "author_name": "Ryan Holbrook", "author_url": "", "post_date": "2024-12-10T12:32:14.487000", "content": "Hi @daebak2021 (and others) I will check up on it. It might be that submissions are getting queued due to compute quotas.
", "votes": 4, "replies": [ { "id": 3068776, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2024-12-10T16:42:24.843000", "content": "I have a submission pending for 13 hours due to the update, will it get rerun if it fails?
", "votes": 0, "replies": [] }, { "id": 3068833, "author_name": "Evgeniia Grigoreva", "author_url": "", "post_date": "2024-12-10T17:42:08.340000", "content": "My submissions ran for ~16 hours and then failed with \"Kaggle error\"
", "votes": 2, "replies": [] }, { "id": 3068966, "author_name": "leo", "author_url": "", "post_date": "2024-12-10T22:29:59.523000", "content": "Same here, still facing the problem. Made 5 submissions yesterday, and all are 9h+ now.
", "votes": 1, "replies": [] }, { "id": 3069057, "author_name": "Arash", "author_url": "", "post_date": "2024-12-11T02:35:46.670000", "content": "Same here!
", "votes": 0, "replies": [] }, { "id": 3069538, "author_name": "", "author_url": "", "post_date": "2024-12-11T15:43:04.737000", "content": "Did this work for anyone? I'm still having the same issue as you guys
", "votes": 0, "replies": [] }, { "id": 3069699, "author_name": "Michael Schroeter", "author_url": "", "post_date": "2024-12-11T19:16:59.373000", "content": "Hasn't worked for me yet, I am still having the same issue.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3087593, "author_name": "PlayData", "author_url": "", "post_date": "2025-01-03T16:32:51.757000", "content": "I got a question here. And much thanks to anyone could answer my question:
\nThe rule said that the inference time should be within 1 minute. But since for a successful public leaderboard submission, we got a limit of 8-hours running time. And the validation set in leaderboard has nearly 200 days. So the actual inference time should be completed in 480*60 = 28800s / （900 time id for each date *200） = 0.16s. Am I right? @ryanholbrook @yuanzhezhou
", "votes": 1, "replies": [ { "id": 3087595, "author_name": "PlayData", "author_url": "", "post_date": "2025-01-03T16:36:47.613000", "content": "0.16s to complete all things including feature engineering and inference
", "votes": -1, "replies": [] } ] }, { "id": 3096475, "author_name": "Matthew Willett", "author_url": "", "post_date": "2025-01-14T12:43:28.687000", "content": "How often can we expect leaderboard updates now during the forecasting phase?
", "votes": 2, "replies": [] }, { "id": 3095976, "author_name": "JM", "author_url": "", "post_date": "2025-01-14T00:41:26.783000", "content": "How often can we expect leaderboard updates now in forecasting phase?
", "votes": 2, "replies": [ { "id": 3097497, "author_name": "Shiqiang Lee", "author_url": "", "post_date": "2025-01-15T12:42:59.450000", "content": "seemingly 15d? But I forget where I saw it.
", "votes": 0, "replies": [ { "id": 3099182, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2025-01-17T12:09:13.620000", "content": "I dont see it anywhere. But previous Jane Street competition seems to be updated every 2 weeks: https://www.kaggle.com/competitions/jane-street-market-prediction/discussion/226745
", "votes": 0, "replies": [] }, { "id": 3099184, "author_name": "", "author_url": "", "post_date": "2025-01-17T12:16:09.097000", "content": "", "votes": 3, "replies": [] }, { "id": 3099800, "author_name": "Yirun Zhang", "author_url": "", "post_date": "2025-01-18T09:54:28.747000", "content": "We are planning to update the private leaderboard roughly every month. The second update of public learderboard should happen before it but won't count into the private score.
", "votes": 8, "replies": [] }, { "id": 3100180, "author_name": "John Payne", "author_url": "", "post_date": "2025-01-18T22:16:30.807000", "content": "\n\nWe are planning to update the private leaderboard roughly every month. The second update of public learderboard should happen before it but won't count into the private score.
\n
So if I'm understanding correctly, the public leaderboard will be updated once more with the rest of the data up until the end date of the first phase, then the private leaderboard will be updated periodically with data collected after the second phase starts and will only score the newly collected data.
\nWill we be able to see the private leaderboard? If so, will that replace the public leaderboard that we see now, or will we be able to see both the public and private leaderboards?
", "votes": 0, "replies": [] }, { "id": 3100499, "author_name": "Yimin Hu", "author_url": "", "post_date": "2025-01-19T12:59:56.473000", "content": "Hopefully, a ‘late submission’ feature or some submission API could be made available for lower-ranked participants like me to track our improvement. Many of us are still working on preparing ourselves for future JS competitions. Of course, these submissions wouldn’t affect the existing leaderboard or rankings. Thanks for sharing your solution 4 years ago that I am still learning from, and thanks for hosting this competition.
", "votes": -2, "replies": [] }, { "id": 3102143, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2025-01-21T18:25:50.137000", "content": "@gogo827jz thanks for the information. when is the public LB update?
", "votes": 1, "replies": [] } ] } ] }, { "id": 3070815, "author_name": "Wayne_127", "author_url": "", "post_date": "2024-12-13T04:20:25.677000", "content": "My previous selected submissions have not yet been processed. Will the score remain the same, or are there any issues when rerunning the notebook, such as timeouts or out-of-memory errors?
", "votes": 1, "replies": [ { "id": 3071525, "author_name": "Ryan Holbrook", "author_url": "", "post_date": "2024-12-13T21:20:04.973000", "content": "If they haven't been updated, it's likely they failed the rescore. There was nothing to publish in that case.
", "votes": 1, "replies": [] } ] }, { "id": 3069024, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2024-12-11T01:00:50.890000", "content": "Will the previous submissions' score be updated? It does not seem to be gradually updating.
\nupdate: now updating
", "votes": 2, "replies": [ { "id": 3069046, "author_name": "INF", "author_url": "", "post_date": "2024-12-11T01:50:37.020000", "content": "Same question …
", "votes": 1, "replies": [] }, { "id": 3069263, "author_name": "cm391", "author_url": "", "post_date": "2024-12-11T08:57:04.470000", "content": "you can just resubmit for faster scoring
", "votes": 0, "replies": [ { "id": 3069400, "author_name": "leo", "author_url": "", "post_date": "2024-12-11T12:55:30.727000", "content": "Unfortunately, resubmit didn't work for me.
", "votes": 1, "replies": [] } ] } ] }, { "id": 3089864, "author_name": "Ashutosh Shukla", "author_url": "", "post_date": "2025-01-06T16:07:23.430000", "content": "What does Score mean in leaderboard, can someone please clarify what metric is that?
", "votes": -7, "replies": [] }, { "id": 3085630, "author_name": "atharva bhai pandey", "author_url": "", "post_date": "2025-01-01T11:45:42.070000", "content": "What is the relevance of tags in the features.csv and responder.csv files?
", "votes": -1, "replies": [] }, { "id": 3081423, "author_name": "Max Kazansky", "author_url": "", "post_date": "2024-12-26T17:37:14.113000", "content": "Hi,I am sorry. I am unable to score. The script is paused for several hours. Can you do something about it?
\nThanks.
In the hidden test set will there be responder_6 column also ? We have to only calculate and submit the output.parquet which we get after running the trained model on test.parquet right ?
", "votes": -6, "replies": [ { "id": 3083283, "author_name": "cty_0705", "author_url": "", "post_date": "2024-12-29T08:40:51.670000", "content": "Yes, in the hidden test set, there should be a responder_6 column, which is typically used to indicate whether a customer responded to a particular offer or not. You are correct that your task is to run your trained model on the test.parquet file, which includes the responder_6 column and other relevant features.
", "votes": 0, "replies": [ { "id": 3085626, "author_name": "atharva bhai pandey", "author_url": "", "post_date": "2025-01-01T11:43:39.203000", "content": "Thanks bro .
", "votes": 0, "replies": [] }, { "id": 3090956, "author_name": "Slaviša Milinković", "author_url": "", "post_date": "2025-01-07T20:28:16.270000", "content": "hmm, as far as I understand, test file contains: row_id, date_id, time_id, symbol_id, weight, is_scored and features_xx (00 to 78) columns, while lags file contains: date_id, time_id, symbol_id, responder_x_lag_1 (0 to 8) columns and we should use kaggle_evaluation and predict function …?
", "votes": 0, "replies": [] } ] } ] }, { "id": 3079267, "author_name": "atharva bhai pandey", "author_url": "", "post_date": "2024-12-23T13:17:55.657000", "content": "Why in the test.parquet file all rows are having only values as zero everywhere ?
", "votes": -7, "replies": [] }, { "id": 3079209, "author_name": "atharva bhai pandey", "author_url": "", "post_date": "2024-12-23T11:24:28.567000", "content": "Why does the test.parquet file does not contain any responder features ?
", "votes": -7, "replies": [] }, { "id": 3079170, "author_name": "atharva bhai pandey", "author_url": "", "post_date": "2024-12-23T09:37:14.563000", "content": "The train.parquet file is solely responsible for building the model ? Like do we have to do the train test split on train.parquet only for building the model and then submit ?
", "votes": -6, "replies": [] }, { "id": 3077528, "author_name": "JZ", "author_url": "", "post_date": "2024-12-21T05:03:22.417000", "content": "What about the hidden test set?
", "votes": -2, "replies": [] }, { "id": 3097814, "author_name": "David Johnson 28", "author_url": "", "post_date": "2025-01-15T18:10:44.840000", "content": "When wil it be updated?
", "votes": 0, "replies": [] }, { "id": 3097119, "author_name": "George Zhou", "author_url": "", "post_date": "2025-01-15T04:26:06.843000", "content": "Will everyone's score be updated in the forecasting phase using their selected notebooks, or just a selected amount of people's notebook can be updated?
", "votes": 0, "replies": [] }, { "id": 3097068, "author_name": "ganchan", "author_url": "", "post_date": "2025-01-15T02:32:46.757000", "content": "During the forecast phase, which leaderboard will be updated: the public or the private?
\nI hope the private leaderboard will be updated while the public leaderboard remains unchanged.
The extension of the public test set sounds really helpful. One thing I’m wondering about—is there a chance the additional data will cause any major changes in the nature of the data or the features we’re working with?
", "votes": 0, "replies": [] }, { "id": 3093871, "author_name": "JonnydosSantos", "author_url": "", "post_date": "2025-01-11T12:41:38.997000", "content": "Is there any clarity on the final structure of training data; I assume that the date id's will be updated to the 14th of January with the submission date rolling out from there. This will insure that the date ids are continous? Any thoughts, anyone?
", "votes": 0, "replies": [] }, { "id": 3087816, "author_name": "akamoyo", "author_url": "", "post_date": "2025-01-03T23:22:22.013000", "content": "@ryanholbrook I have a question regarding the 9-hour time limit at the end of the forecasting phase. Is this limit cumulative only for the execution time of each predict() call? I’ve noticed that the time taken between predict() calls (possibly for processing the input data frame) is often longer than the actual execution time of predict() itself. Is anyone clear about how the 9-hour limit is calculated? I believe it would be reasonable to consider only the time spent in predict(). Thanks for any information!
", "votes": 0, "replies": [ { "id": 3091973, "author_name": "Tykmm1", "author_url": "", "post_date": "2025-01-09T03:09:26.007000", "content": "i 've the same question
", "votes": 0, "replies": [] } ] }, { "id": 3083364, "author_name": "leo", "author_url": "", "post_date": "2024-12-29T11:32:19.770000", "content": "\n\nAdditionally, new symbol_id values may appear in future test sets.
\n
This was stated in the data page. Will there be any new symbol_id added to the test set? This is very concerning. It could potentially cause many submission fail.
", "votes": 0, "replies": [ { "id": 3083553, "author_name": "Rubick", "author_url": "", "post_date": "2024-12-29T16:40:44.277000", "content": "The official notebook said it will, I think we need concerning it🤣
", "votes": 0, "replies": [] } ] }, { "id": 3080661, "author_name": "Qiwei Dai", "author_url": "", "post_date": "2024-12-25T14:33:25.457000", "content": "Got it but what about the hidden test set?
", "votes": 0, "replies": [] }, { "id": 3080031, "author_name": "bones-zhu", "author_url": "", "post_date": "2024-12-24T14:25:52.080000", "content": "I have a problem like \"Server did not register a listener for predict\". How could I deal with it ?
\n/kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py in get_all_predictions(self)
\n 51 all_predictions = []
\n 52 for data_batch, validation_batch in self.generate_data_batches():
\n---> 53 predictions = self.predict(*data_batch)
\n 54 self.validate_prediction_batch(predictions, validation_batch)
\n 55 all_predictions.append(predictions)
/kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py in predict(self, *args, **kwargs)
\n 64 return self.client.send('predict', *args, **kwargs)
\n 65 except Exception as e:
\n---> 66 self.handle_server_error(e, 'predict')
\n 67
\n 68 def set_response_timeout_seconds(self, timeout_seconds: float=6_000):
/kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/base_gateway.py in handle_server_error(self, exception, endpoint)
\n 184 raise GatewayRuntimeError(GatewayRuntimeErrorType.SERVER_NEVER_STARTED) from None
\n 185 if f'No listener for {endpoint} was registered' in exception_str:
\n--> 186 raise GatewayRuntimeError(GatewayRuntimeErrorType.SERVER_MISSING_ENDPOINT, f'Server did not register a listener for {endpoint}') from None
\n 187 if 'Exception calling application' in exception_str:
\n 188 # Extract just the exception message raised by the inference server
GatewayRuntimeError: (, 'Server did not register a listener for predict')
", "votes": 0, "replies": [] }, { "id": 3079807, "author_name": "joejeo1", "author_url": "", "post_date": "2024-12-24T06:55:19.220000", "content": "Do you provide time series data in the test set? The example data (from test.parquet) only has one day and one time_id with 39 different symbols. Does each day contains one time_id or multiple time_ids for that day in both the public and private test sets.
", "votes": 0, "replies": [] }, { "id": 3074299, "author_name": "JM", "author_url": "", "post_date": "2024-12-17T14:38:33.547000", "content": "Was 80 actual days or trading days added? / or 80 new date_id added?
", "votes": 0, "replies": [ { "id": 3074640, "author_name": "Ryan Holbrook", "author_url": "", "post_date": "2024-12-17T21:59:31.283000", "content": "It's about 80 new date_ids.
", "votes": 0, "replies": [] } ] }, { "id": 3074022, "author_name": "Blues Hugo", "author_url": "", "post_date": "2024-12-17T08:32:10.450000", "content": "My kaggle kernel crashed from time to time while building training and validation sets, which was a bit distressing, but it was exciting to participate in a competition like this for the first time
", "votes": 0, "replies": [] }, { "id": 3071526, "author_name": "houtadeux", "author_url": "", "post_date": "2024-12-13T21:22:36.567000", "content": "Hi, is it possible to use pretrained model for this competition ?
", "votes": 0, "replies": [ { "id": 3085609, "author_name": "HENRY", "author_url": "", "post_date": "2025-01-01T11:29:28.967000", "content": "Yes, just upload your trained model's result as a .pkl document and create a new dataset to upload it
", "votes": 0, "replies": [ { "id": 3088564, "author_name": "Timothy_brrll", "author_url": "", "post_date": "2025-01-04T20:32:28.947000", "content": "Hey, I am not sure I understand what you mean with \"create a new dataset to upload it\". Doesn't it make more sense to upload the model and load it to make the predictions?
", "votes": 0, "replies": [] } ] } ] }, { "id": 3070504, "author_name": "", "author_url": "", "post_date": "2024-12-12T18:38:39.143000", "content": "", "votes": 0, "replies": [ { "id": 3070533, "author_name": "", "author_url": "", "post_date": "2024-12-12T19:55:39.427000", "content": "", "votes": 6, "replies": [] } ] }, { "id": 3070301, "author_name": "", "author_url": "", "post_date": "2024-12-12T14:26:19.883000", "content": "", "votes": 0, "replies": [] }, { "id": 3069872, "author_name": "", "author_url": "", "post_date": "2024-12-12T01:27:46.703000", "content": "", "votes": 0, "replies": [] }, { "id": 3068287, "author_name": "", "author_url": "", "post_date": "2024-12-10T05:56:27.857000", "content": "", "votes": 0, "replies": [ { "id": 3068682, "author_name": "", "author_url": "", "post_date": "2024-12-10T14:49:27.473000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 3067828, "author_name": "", "author_url": "", "post_date": "2024-12-09T16:38:03.517000", "content": "", "votes": 0, "replies": [ { "id": 3067838, "author_name": "", "author_url": "", "post_date": "2024-12-09T16:52:21.687000", "content": "", "votes": 4, "replies": [ { "id": 3067840, "author_name": "", "author_url": "", "post_date": "2024-12-09T16:57:54.990000", "content": "", "votes": 1, "replies": [] }, { "id": 3067849, "author_name": "", "author_url": "", "post_date": "2024-12-09T17:15:56.677000", "content": "", "votes": 1, "replies": [] }, { "id": 3069227, "author_name": "", "author_url": "", "post_date": "2024-12-11T07:46:18.297000", "content": "", "votes": 1, "replies": [] } ] } ] }, { "id": 3067711, "author_name": "", "author_url": "", "post_date": "2024-12-09T14:44:40.950000", "content": "", "votes": 0, "replies": [ { "id": 3067761, "author_name": "", "author_url": "", "post_date": "2024-12-09T15:23:16.543000", "content": "", "votes": 1, "replies": [ { "id": 3067893, "author_name": "", "author_url": "", "post_date": "2024-12-09T18:12:58.173000", "content": "", "votes": 0, "replies": [] }, { "id": 3067921, "author_name": "", "author_url": "", "post_date": "2024-12-09T18:40:44.347000", "content": "", "votes": 1, "replies": [] }, { "id": 3068098, "author_name": "", "author_url": "", "post_date": "2024-12-10T00:49:01.210000", "content": "", "votes": 0, "replies": [] }, { "id": 3068122, "author_name": "", "author_url": "", "post_date": "2024-12-10T01:41:04.777000", "content": "", "votes": 0, "replies": [] }, { "id": 3068701, "author_name": "", "author_url": "", "post_date": "2024-12-10T15:12:51.850000", "content": "", "votes": 0, "replies": [] } ] } ] }, { "id": 3095959, "author_name": "", "author_url": "", "post_date": "2025-01-14T00:15:33.720000", "content": "", "votes": 0, "replies": [] }, { "id": 3079699, "author_name": "", "author_url": "", "post_date": "2024-12-24T02:55:34.017000", "content": "", "votes": 0, "replies": [] }, { "id": 3106459, "author_name": "", "author_url": "", "post_date": "2025-01-24T21:33:37.680000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3067689": "Hi everyone,\n\nTo help ensure you have the most up-to-date data to test your submissions against, we'll be extending the public test set with an additional ~80 days of data. The extension is continuous with the original set. The symbol ids and the number of time ids per date are both unchanged.\n\nWe'll be rerunning all submissions against this updated data. As we currently have thousands of submissions to rerun, expect this to take at least several days. Once the rerun is complete, the leaderboard will show your updated score.\n\nI hope to kick off the update later today. Please let us know if you have any questions or comments!\n\n**UPDATE 12/09 11:15 EST:** Submissions temporarily paused while data is updated. Will turn on again shortly.\n\n**UPDATE 12/09 13:15 EST:** The data update is complete. Submissions are live again. New submissions will be scored against the updated data. I will be kicking off the rerun of existing submissions soon. Expect this to take a while.\n\n**UPDATE 12/12 09:30 EST:** I'll be releasing the first set of rescores soon. This set only contains those submissions manually chosen as \"selected\" in the Submissions tab, but not those which had been automatically selected at the time of the data update.\n\nI am continuing the rescore with the another set of submissions, which I hope will be completed within a few days. Note that these rescores have to be done in bulk, which is why scores aren't updated incrementally.", "3088359": "Got longer than 9 hour submission time (which means that the submission was waiting for resource and did not start) and kaggle error, will this happen also in the last days?\n\nupdate: facing this issue for all the recent submissions", "3089771": "@ryanholbrook - Can you please check the high submission running time issue as highlighted by others. All 4 notebooks of mine from last evening has gone much over the estimated time and timed out. Thanks", "3068211": "My submission, which is a very simple notebook (just for testing the end-to-end submission process), has been taking over 5 hours to score. Could this delay be due to backfilling and rerunning existing submissions?", "3087593": "I got a question here. And much thanks to anyone could answer my question:\n\nThe rule said that the inference time should be within 1 minute. But since for a successful public leaderboard submission, we got a limit of 8-hours running time. And the validation set in leaderboard has nearly 200 days. So the actual inference time should be completed in 480*60 = 28800s / （900 time id for each date *200） = 0.16s. Am I right? @ryanholbrook @yuanzhezhou ", "3096475": "How often can we expect leaderboard updates now during the forecasting phase?", "3095976": "How often can we expect leaderboard updates now in forecasting phase?", "3070815": "My previous selected submissions have not yet been processed. Will the score remain the same, or are there any issues when rerunning the notebook, such as timeouts or out-of-memory errors? ", "3069024": "Will the previous submissions' score be updated? It does not seem to be gradually updating.\n\nupdate: now updating", "3089864": "What does Score mean in leaderboard, can someone please clarify what metric is that?\n", "3085630": "What is the relevance of tags in the features.csv and responder.csv files?", "3081423": "Hi,I am sorry. I am unable to score. The script is paused for several hours. Can you do something about it?\nThanks.", "3081287": "In the hidden test set will there be responder_6 column also ? We have to only calculate and submit the output.parquet which we get after running the trained model on test.parquet right ?", "3079267": "Why in the test.parquet file all rows are having only values as zero everywhere ?", "3079209": "Why does the test.parquet file does not contain any responder features ?", "3079170": "The train.parquet file is solely responsible for building the model ? Like do we have to do the train test split on train.parquet only for building the model and then submit ?", "3077528": "What about the hidden test set?", "3097814": "When wil it be updated?", "3097119": "Will everyone's score be updated in the forecasting phase using their selected notebooks, or just a selected amount of people's notebook can be updated?", "3097068": "During the forecast phase, which leaderboard will be updated: the public or the private?\nI hope the private leaderboard will be updated while the public leaderboard remains unchanged.", "3095703": "The extension of the public test set sounds really helpful. One thing I’m wondering about—is there a chance the additional data will cause any major changes in the nature of the data or the features we’re working with? ", "3093871": "Is there any clarity on the final structure of training data; I assume that the date id's will be updated to the 14th of January with the submission date rolling out from there. This will insure that the date ids are continous? Any thoughts, anyone?", "3087816": "@ryanholbrook I have a question regarding the 9-hour time limit at the end of the forecasting phase. Is this limit cumulative only for the execution time of each predict() call? I’ve noticed that the time taken between predict() calls (possibly for processing the input data frame) is often longer than the actual execution time of predict() itself. Is anyone clear about how the 9-hour limit is calculated? I believe it would be reasonable to consider only the time spent in predict(). Thanks for any information!", "3083364": ">Additionally, new symbol_id values may appear in future test sets.\n\nThis was stated in the data page. Will there be any new symbol_id added to the test set? This is very concerning. It could potentially cause many submission fail. ", "3080661": "Got it but what about the hidden test set?", "3080031": "I have a problem like \"Server did not register a listener for predict\". How could I deal with it ?\n/kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py in get_all_predictions(self)\n 51 all_predictions = []\n 52 for data_batch, validation_batch in self.generate_data_batches():\n---> 53 predictions = self.predict(*data_batch)\n 54 self.validate_prediction_batch(predictions, validation_batch)\n 55 all_predictions.append(predictions)\n\n/kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/templates.py in predict(self, *args, **kwargs)\n 64 return self.client.send('predict', *args, **kwargs)\n 65 except Exception as e:\n---> 66 self.handle_server_error(e, 'predict')\n 67 \n 68 def set_response_timeout_seconds(self, timeout_seconds: float=6_000):\n\n/kaggle/input/jane-street-real-time-market-data-forecasting/kaggle_evaluation/core/base_gateway.py in handle_server_error(self, exception, endpoint)\n 184 raise GatewayRuntimeError(GatewayRuntimeErrorType.SERVER_NEVER_STARTED) from None\n 185 if f'No listener for {endpoint} was registered' in exception_str:\n--> 186 raise GatewayRuntimeError(GatewayRuntimeErrorType.SERVER_MISSING_ENDPOINT, f'Server did not register a listener for {endpoint}') from None\n 187 if 'Exception calling application' in exception_str:\n 188 # Extract just the exception message raised by the inference server\n\nGatewayRuntimeError: (