all_data = test.join(lags, on=['date_id', 'time_id', 'symbol_id'], how='left')
\nThis line in my predict function is giving me an error during competition submission but works fine during normal run. Any ideas on why this is happening?
all_data = test.join(lags, on=['date_id', 'time_id', 'symbol_id'], how='left')
\nThis line in my predict function is giving me an error during competition submission but works fine during normal run. Any ideas on why this is happening?
@snehalverma10 please peruse the post here, @chumajin has explained this in detail
", "rawMarkdown": "snehalverma10 please peruse the post [here](https://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion/541171), @chumajin has explained this in detail", "votes": null }, { "id": "3020971", "postDate": "10/18/2024 05:13:06", "content": "During \"normal\" run we have only one data portion with time_id equals to 0. However, during the submission, lags variable may be None. And it is not None when time_id equals to 0, again. In this case, lags contains all responders data from the day before, i.e. with different time_id values.
\nSo, simple join of these two variables seems not very useful or even may lead to errors.
", "rawMarkdown": "During \"normal\" run we have only one data portion with time_id equals to 0. However, during the submission, lags variable may be None. And it is not None when time_id equals to 0, again. In this case, lags contains all responders data from the day before, i.e. with different time_id values.\n\nSo, simple join of these two variables seems not very useful or even may lead to errors.", "votes": null }, { "id": "3020973", "postDate": "10/18/2024 05:15:13", "content": "check whether the test set structure or column availability differs during submission. Adding checks and safeguards for missing columns or mismatches between the test and lags DataFrames should help prevent the error.
", "rawMarkdown": "check whether the test set structure or column availability differs during submission. Adding checks and safeguards for missing columns or mismatches between the test and lags DataFrames should help prevent the error.", "votes": null }, { "id": "3021043", "postDate": "10/18/2024 06:54:56", "content": "@ravi20076 Thank you for referring to my notebook.
\n@snehalverma10 Maybe, you can understand from my notebook.
", "rawMarkdown": "ravi20076 Thank you for referring to my notebook.\n\n@snehalverma10 Maybe, you can understand from my notebook.", "votes": null }, { "id": "3021102", "postDate": "10/18/2024 07:37:29", "content": "The given test.parquet is only one batch of the data. To fully test your code, you need to use synthetic test data to simulate all corner cases that can happen in the real submission. Check my work here on how to generate synthetic data: https://www.kaggle.com/code/shiyili/js24-rmf-generate-synthetic-test-data
", "rawMarkdown": "The given test.parquet is only one batch of the data. To fully test your code, you need to use synthetic test data to simulate all corner cases that can happen in the real submission. Check my work here on how to generate synthetic data: https://www.kaggle.com/code/shiyili/js24-rmf-generate-synthetic-test-data", "votes": null }, { "id": "3031370", "postDate": "10/29/2024 16:15:58", "content": "It works fine on normal run because it joins a single batch. During the submission, it keeps joining new batches which leads to creation of duplicate columns and they are named as column_x, column_y and goes on like that.
", "rawMarkdown": "It works fine on normal run because it joins a single batch. During the submission, it keeps joining new batches which leads to creation of duplicate columns and they are named as column_x, column_y and goes on like that.", "votes": null } ], "comments": [ { "id": 3020966, "author_name": "ravi20076", "author_url": "", "post_date": "10/18/2024 05:07:37", "content": "@snehalverma10 please peruse the post here, @chumajin has explained this in detail
", "votes": null, "replies": [ { "id": 3021043, "author_name": "chumajin", "author_url": "", "post_date": "10/18/2024 06:54:56", "content": "@ravi20076 Thank you for referring to my notebook.
\n@snehalverma10 Maybe, you can understand from my notebook.
", "votes": null, "replies": [] } ] }, { "id": 3020971, "author_name": "kdmitrie", "author_url": "", "post_date": "10/18/2024 05:13:06", "content": "During \"normal\" run we have only one data portion with time_id equals to 0. However, during the submission, lags variable may be None. And it is not None when time_id equals to 0, again. In this case, lags contains all responders data from the day before, i.e. with different time_id values.
\nSo, simple join of these two variables seems not very useful or even may lead to errors.
", "votes": null, "replies": [] }, { "id": 3020973, "author_name": "sumit08", "author_url": "", "post_date": "10/18/2024 05:15:13", "content": "check whether the test set structure or column availability differs during submission. Adding checks and safeguards for missing columns or mismatches between the test and lags DataFrames should help prevent the error.
", "votes": null, "replies": [] }, { "id": 3021102, "author_name": "shiyili", "author_url": "", "post_date": "10/18/2024 07:37:29", "content": "The given test.parquet is only one batch of the data. To fully test your code, you need to use synthetic test data to simulate all corner cases that can happen in the real submission. Check my work here on how to generate synthetic data: https://www.kaggle.com/code/shiyili/js24-rmf-generate-synthetic-test-data
", "votes": null, "replies": [] }, { "id": 3031370, "author_name": "gunesevitan", "author_url": "", "post_date": "10/29/2024 16:15:58", "content": "It works fine on normal run because it joins a single batch. During the submission, it keeps joining new batches which leads to creation of duplicate columns and they are named as column_x, column_y and goes on like that.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3020836": "`all_data = test.join(lags, on=['date_id', 'time_id', 'symbol_id'], how='left')` \nThis line in my predict function is giving me an error during competition submission but works fine during normal run. Any ideas on why this is happening?", "3020966": "snehalverma10 please peruse the post [here](https://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion/541171), @chumajin has explained this in detail", "3020971": "During \"normal\" run we have only one data portion with time_id equals to 0. However, during the submission, lags variable may be None. And it is not None when time_id equals to 0, again. In this case, lags contains all responders data from the day before, i.e. with different time_id values.\n\nSo, simple join of these two variables seems not very useful or even may lead to errors.", "3020973": "check whether the test set structure or column availability differs during submission. Adding checks and safeguards for missing columns or mismatches between the test and lags DataFrames should help prevent the error.", "3021043": "ravi20076 Thank you for referring to my notebook.\n\n@snehalverma10 Maybe, you can understand from my notebook.", "3021102": "The given test.parquet is only one batch of the data. To fully test your code, you need to use synthetic test data to simulate all corner cases that can happen in the real submission. Check my work here on how to generate synthetic data: https://www.kaggle.com/code/shiyili/js24-rmf-generate-synthetic-test-data", "3031370": "It works fine on normal run because it joins a single batch. During the submission, it keeps joining new batches which leads to creation of duplicate columns and they are named as column_x, column_y and goes on like that." }, "source": "meta" }