![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F962168%2F336770a8fd5f4d451e7fa9aad91dbc0e%2FScreenshot%202024-10-30%20141225.jpg?generation=1730293996962899&alt=media\")
I'm probably missing something here: what is the 'row_id' found in the test set and not in the training set? How is it obtained, based on training set data?
", "messages": [ { "id": 3031980, "postDate": "2024-10-30T11:45:45.163Z", "content": "row_id is symbol_id?
", "rawMarkdown": "row_id is symbol_id?", "votes": 1, "replies": [ { "id": 3032002, "postDate": "2024-10-30T12:43:24.290Z", "content": "That's the point…
", "rawMarkdown": "That's the point..." } ] }, { "id": 3031093, "postDate": "2024-10-29T10:21:17.390Z", "content": "I'm probably missing something here: what is the 'row_id' found in the test set and not in the training set? How is it obtained, based on training set data?
", "rawMarkdown": "I'm probably missing something here: what is the 'row_id' found in the test set and not in the training set? How is it obtained, based on training set data?", "votes": 1 }, { "id": 3031106, "postDate": "2024-10-29T10:45:04.913Z", "content": "It has nothing to do with the training data. It’s simply a row index to make sure your predictions align correctly with the test batch.
", "rawMarkdown": "It has nothing to do with the training data. It’s simply a row index to make sure your predictions align correctly with the test batch.", "replies": [ { "id": 3031152, "postDate": "2024-10-29T12:23:23.887Z", "content": "Thanks @shiyili. But it does matter A LOT, as every 'symbol_id' is an individual asset (\"identifies a unique financial instrument\").
", "rawMarkdown": "Thanks @shiyili. But it does matter A LOT, as every 'symbol_id' is an individual asset (\"identifies a unique financial instrument\"). ", "votes": -1, "replies": [ { "id": 3031175, "postDate": "2024-10-29T12:54:30.020Z", "content": "Isn't the question about row_id ?
It is. Actually, 'row_id' has to have a meaning. How do you identify an individual asset in the test set? Not by a 'row_id' absent in the training. Each 'symbol_id' contains 1 time series (with dates and times identifiers). The test set should contain the same information for each asset. Unless I'm missing something.
", "rawMarkdown": "It is. Actually, 'row_id' has to have a meaning. How do you identify an individual asset in the test set? Not by a 'row_id' absent in the training. Each 'symbol_id' contains 1 time series (with dates and times identifiers). The test set should contain the same information for each asset. Unless I'm missing something. " }, { "id": 3031982, "postDate": "2024-10-30T11:46:20.977Z", "content": "\n\nHow do you identify an individual asset in the test set?
\n
By symbol_id
", "rawMarkdown": ">How do you identify an individual asset in the test set?\n\nBy symbol_id" }, { "id": 3032003, "postDate": "2024-10-30T12:43:59.210Z", "content": "symbol_id = row_id?
", "rawMarkdown": "symbol_id = row_id?" }, { "id": 3032026, "postDate": "2024-10-30T13:14:46.817Z", "content": "No @thierryme . simply print a test.parquet and you won't be so confused.
\nsymbol_id is symbol_id, data_id is date_id, time_id is time-id. row_id is row index.
No. Look at test.parquet. row_id and symbol_id are clearly different columns. They are only equal because the data is sorted by date_id, time_id, and symbol_id. For a hypothetical row_id=40 (if there are no new symbol_id), symbol_id would theoretically be 0 again.
Can you break this down? Do not think in a supervised tabular learning way, and predict everything as a bulk. Where is the asset identifier? How do you identify the future dates for EACH asset?
\npredictions = test.select(\n ,\n pl.lit().alias(),\n )\n\n \n (predictions, pl.DataFrame | pd.DataFrame)\n \n predictions.columns == [, ]\n \n (predictions) == (test)\nThat's exactly what row_id is for. It's to identify which prediction corresponds to which row of data. The symbol_id is for you to identify which symbol it is.
The idea is that you take the test data, format it into the required formats, make your predictions, and then the row_id is used to identify which prediction corresponds to which observation.
Think about it this way, instead of the function passing in date_id,time_id,symbol_id and responder_6, the function only has to send in row_id and responder_6 to the server.
@abstrphil Thanks, so we assume everything is always sorted
", "rawMarkdown": "@abstrphil Thanks, so we assume everything is always sorted " }, { "id": 3034663, "postDate": "2024-11-02T12:52:51.813Z", "content": "You don't have to assume anything, that's the point of row_id. Any two rows with the same row_id correspond to the same observation.
@sohier Hi, could you please address this? Need to link symbol_id, date_id and time_id in training set to symbol_id, date_id and time_id (historical data for each financial asset) in test set (future data for each financial asset)
", "rawMarkdown": "@sohier Hi, could you please address this? Need to link symbol_id, date_id and time_id in training set to symbol_id, date_id and time_id (historical data for each financial asset) in test set (future data for each financial asset)", "votes": -1 }, { "id": 3031139, "postDate": "2024-10-29T11:59:48.757Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 3031153, "postDate": "2024-10-29T12:26:25.287Z", "content": "Thanks @anthonytherrien! Any idea on how to get 'date_id', 'symbol_id' and 'time_id' (it does matter) from the public leaderboard? It \"usually\" comes in a file called submission_example.
\n\"to help link predictions back to the original rows\": this might be what I'm looking for. Could you please elaborate in this context?
", "rawMarkdown": "Thanks @anthonytherrien! Any idea on how to get 'date_id', 'symbol_id' and 'time_id' (it does matter) from the public leaderboard? It \"usually\" comes in a file called submission_example.\n\n\"to help link predictions back to the original rows\": this might be what I'm looking for. Could you please elaborate in this context?", "replies": [ { "id": 3031766, "postDate": "2024-10-30T04:45:08.007Z", "content": "Actually, 'row_id' has to have a meaning. How do you identify an individual asset in the test set? Not by a 'row_id' absent in the training. Each 'symbol_id' contains 1 time series (with dates and times identifiers). The test set should contain the same information for each asset. Unless I'm missing something.
", "rawMarkdown": "Actually, 'row_id' has to have a meaning. How do you identify an individual asset in the test set? Not by a 'row_id' absent in the training. Each 'symbol_id' contains 1 time series (with dates and times identifiers). The test set should contain the same information for each asset. Unless I'm missing something. " }, { "id": 3075676, "postDate": "2024-12-19T06:05:25.027Z", "content": "Just to be very sure, just like in the sample submission file, does it start from 0 and go all the to N number of predictions?
", "rawMarkdown": "Just to be very sure, just like in the sample submission file, does it start from 0 and go all the to N number of predictions?" } ] } ] } ], "comments": [ { "id": 3031980, "author_name": "JixuanZhang", "author_url": "", "post_date": "2024-10-30T11:45:45.163000", "content": "row_id is symbol_id?
", "votes": 1, "replies": [ { "id": 3032002, "author_name": "T M", "author_url": "", "post_date": "2024-10-30T12:43:24.290000", "content": "That's the point…
", "votes": 0, "replies": [] } ] }, { "id": 3031106, "author_name": "SLi", "author_url": "", "post_date": "2024-10-29T10:45:04.913000", "content": "It has nothing to do with the training data. It’s simply a row index to make sure your predictions align correctly with the test batch.
", "votes": 0, "replies": [ { "id": 3031152, "author_name": "T M", "author_url": "", "post_date": "2024-10-29T12:23:23.887000", "content": "Thanks @shiyili. But it does matter A LOT, as every 'symbol_id' is an individual asset (\"identifies a unique financial instrument\").
", "votes": -1, "replies": [ { "id": 3031175, "author_name": "SLi", "author_url": "", "post_date": "2024-10-29T12:54:30.020000", "content": "Isn't the question about row_id ?
It is. Actually, 'row_id' has to have a meaning. How do you identify an individual asset in the test set? Not by a 'row_id' absent in the training. Each 'symbol_id' contains 1 time series (with dates and times identifiers). The test set should contain the same information for each asset. Unless I'm missing something.
", "votes": 0, "replies": [] }, { "id": 3031982, "author_name": "Evgeniia Grigoreva", "author_url": "", "post_date": "2024-10-30T11:46:20.977000", "content": "\n\nHow do you identify an individual asset in the test set?
\n
By symbol_id
", "votes": 0, "replies": [] }, { "id": 3032003, "author_name": "T M", "author_url": "", "post_date": "2024-10-30T12:43:59.210000", "content": "symbol_id = row_id?
", "votes": 0, "replies": [] }, { "id": 3032026, "author_name": "SLi", "author_url": "", "post_date": "2024-10-30T13:14:46.817000", "content": "No @thierryme . simply print a test.parquet and you won't be so confused.
\nsymbol_id is symbol_id, data_id is date_id, time_id is time-id. row_id is row index.
No. Look at test.parquet. row_id and symbol_id are clearly different columns. They are only equal because the data is sorted by date_id, time_id, and symbol_id. For a hypothetical row_id=40 (if there are no new symbol_id), symbol_id would theoretically be 0 again.
Can you break this down? Do not think in a supervised tabular learning way, and predict everything as a bulk. Where is the asset identifier? How do you identify the future dates for EACH asset?
\npredictions = test.select(\n ,\n pl.lit().alias(),\n )\n\n \n (predictions, pl.DataFrame | pd.DataFrame)\n \n predictions.columns == [, ]\n \n (predictions) == (test)\nThat's exactly what row_id is for. It's to identify which prediction corresponds to which row of data. The symbol_id is for you to identify which symbol it is.
The idea is that you take the test data, format it into the required formats, make your predictions, and then the row_id is used to identify which prediction corresponds to which observation.
Think about it this way, instead of the function passing in date_id,time_id,symbol_id and responder_6, the function only has to send in row_id and responder_6 to the server.
@abstrphil Thanks, so we assume everything is always sorted
", "votes": 0, "replies": [] }, { "id": 3034663, "author_name": "Abstr Phil", "author_url": "", "post_date": "2024-11-02T12:52:51.813000", "content": "You don't have to assume anything, that's the point of row_id. Any two rows with the same row_id correspond to the same observation.
@sohier Hi, could you please address this? Need to link symbol_id, date_id and time_id in training set to symbol_id, date_id and time_id (historical data for each financial asset) in test set (future data for each financial asset)
", "votes": -1, "replies": [] }, { "id": 3031139, "author_name": "", "author_url": "", "post_date": "2024-10-29T11:59:48.757000", "content": "", "votes": 0, "replies": [ { "id": 3031153, "author_name": "T M", "author_url": "", "post_date": "2024-10-29T12:26:25.287000", "content": "Thanks @anthonytherrien! Any idea on how to get 'date_id', 'symbol_id' and 'time_id' (it does matter) from the public leaderboard? It \"usually\" comes in a file called submission_example.
\n\"to help link predictions back to the original rows\": this might be what I'm looking for. Could you please elaborate in this context?
", "votes": 0, "replies": [ { "id": 3031766, "author_name": "T M", "author_url": "", "post_date": "2024-10-30T04:45:08.007000", "content": "Actually, 'row_id' has to have a meaning. How do you identify an individual asset in the test set? Not by a 'row_id' absent in the training. Each 'symbol_id' contains 1 time series (with dates and times identifiers). The test set should contain the same information for each asset. Unless I'm missing something.
", "votes": 0, "replies": [] }, { "id": 3075676, "author_name": "kaso", "author_url": "", "post_date": "2024-12-19T06:05:25.027000", "content": "Just to be very sure, just like in the sample submission file, does it start from 0 and go all the to N number of predictions?
", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3031980": "row_id is symbol_id?", "3031093": "I'm probably missing something here: what is the 'row_id' found in the test set and not in the training set? How is it obtained, based on training set data?", "3031106": "It has nothing to do with the training data. It’s simply a row index to make sure your predictions align correctly with the test batch.", "3032009": "@sohier Hi, could you please address this? Need to link symbol_id, date_id and time_id in training set to symbol_id, date_id and time_id (historical data for each financial asset) in test set (future data for each financial asset)", "3031139": "" } }