![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4858569%2Fa510fba3ef553341b89fcc412d622e42%2F_20241109211055.png?generation=1731157909295208&alt=media\")
for test,I did ffill, bfill
\nafter merge, I did fill by median number, bfill and ffill
\nif there are even one digits after merge I should get a score, but I didn't.
BUT, if I do fill by 0, I will get a score.
\nSO, I suspect that, after merge lags with test, the responder values are all NaN! that why after ffill, bfill, my predictions will result in NaN prediction which will result in \"Score Error\"
\n test_df = test.to_pandas()\n lags_df = lags_.to_pandas()\n test_df = test_df.bfill()\n test_df = test_df.ffill()\n\n\n X_test = pd.merge(test_df, lags_df, on=[, ], how=, suffixes = (,))\n\n \n responder_columns = [ i (, )]\n col responder_columns:\n median_value = X_test[col].median(skipna=) \n X_test[col] = X_test[col].fillna(median_value) \n X_test = X_test.bfill()\n X_test = X_test.ffill()\n
for test,I did ffill, bfill
\nafter merge, I did fill by median number, bfill and ffill
\nif there are even one digits after merge I should get a score, but I didn't.
BUT, if I do fill by 0, I will get a score.
\nSO, I suspect that, after merge lags with test, the responder values are all NaN! that why after ffill, bfill, my predictions will result in NaN prediction which will result in \"Score Error\"
\n test_df = test.to_pandas()\n lags_df = lags_.to_pandas()\n test_df = test_df.bfill()\n test_df = test_df.ffill()\n\n\n X_test = pd.merge(test_df, lags_df, on=[, ], how=, suffixes = (,))\n\n \n responder_columns = [ i (, )]\n col responder_columns:\n median_value = X_test[col].median(skipna=) \n X_test[col] = X_test[col].fillna(median_value) \n X_test = X_test.bfill()\n X_test = X_test.ffill()\n
Forward fill and back fill can result in NaN values if there is a sequence of NaN values at the beginning or end of a dataset, or when there are continuous NaNs without non-NaN values between them.
", "rawMarkdown": "Forward fill and back fill can result in NaN values if there is a sequence of NaN values at the beginning or end of a dataset, or when there are continuous NaNs without non-NaN values between them. " }, { "id": 3041570, "postDate": "2024-11-10T14:24:18.870Z", "content": "how much time does it takes to generate the score ?
\nbecause i am also facing similar issue during submission it runs 9 hours then fails i don't understand why it fails.
", "rawMarkdown": "how much time does it takes to generate the score ?\n\nbecause i am also facing similar issue during submission it runs 9 hours then fails i don't understand why it fails.", "replies": [ { "id": 3042090, "postDate": "2024-11-11T07:21:21.907Z", "content": "it took me a whole night to get the results. I think its probably because im using neural network models. if submitting boosting models, it come back faster, like 2 - 4 hours.
\nmy problem is that NaNs are messing with me.
Is there any conditions like time limitations.because I am using pickle file to load the model and directly try to run predict function but somehow it's not generating score.
", "rawMarkdown": "Is there any conditions like time limitations.because I am using pickle file to load the model and directly try to run predict function but somehow it's not generating score. " }, { "id": 3043409, "postDate": "2024-11-12T12:03:24.120Z", "content": "the way you are doing seems fine. whats the error messsage?
", "rawMarkdown": "the way you are doing seems fine. whats the error messsage?" }, { "id": 3043540, "postDate": "2024-11-12T13:36:29.600Z", "content": "There is no error message came while saving it but when I am trying to run for submission it was not generating score.
", "rawMarkdown": "There is no error message came while saving it but when I am trying to run for submission it was not generating score. " }, { "id": 3044177, "postDate": "2024-11-13T06:14:16.433Z", "content": "likeā¦ come back with 0?
", "rawMarkdown": "like... come back with 0?" } ] } ] }, { "id": 3041495, "postDate": "2024-11-10T12:54:32.777Z", "content": "Not sure but have you checked the data types? I am having a horrid time doing merge and concats with Polars changing the datatype of columns. I am having to explicitly recast everything after each operation.
\nUsing a fill by 0 or by mean could be the difference between a Float32 or Float64 or Int8 etc etc
", "rawMarkdown": "Not sure but have you checked the data types? I am having a horrid time doing merge and concats with Polars changing the datatype of columns. I am having to explicitly recast everything after each operation.\n\nUsing a fill by 0 or by mean could be the difference between a Float32 or Float64 or Int8 etc etc" }, { "id": 3040762, "postDate": "2024-11-09T14:53:07.657Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 3044834, "author_name": "Younes Benalia", "author_url": "", "post_date": "2024-11-13T21:46:50.223000", "content": "Forward fill and back fill can result in NaN values if there is a sequence of NaN values at the beginning or end of a dataset, or when there are continuous NaNs without non-NaN values between them.
", "votes": 0, "replies": [] }, { "id": 3041570, "author_name": "Raviksh Singh Dikola", "author_url": "", "post_date": "2024-11-10T14:24:18.870000", "content": "how much time does it takes to generate the score ?
\nbecause i am also facing similar issue during submission it runs 9 hours then fails i don't understand why it fails.
", "votes": 0, "replies": [ { "id": 3042090, "author_name": "ZT", "author_url": "", "post_date": "2024-11-11T07:21:21.907000", "content": "it took me a whole night to get the results. I think its probably because im using neural network models. if submitting boosting models, it come back faster, like 2 - 4 hours.
\nmy problem is that NaNs are messing with me.
Is there any conditions like time limitations.because I am using pickle file to load the model and directly try to run predict function but somehow it's not generating score.
", "votes": 0, "replies": [] }, { "id": 3043409, "author_name": "ZT", "author_url": "", "post_date": "2024-11-12T12:03:24.120000", "content": "the way you are doing seems fine. whats the error messsage?
", "votes": 0, "replies": [] }, { "id": 3043540, "author_name": "Raviksh Singh Dikola", "author_url": "", "post_date": "2024-11-12T13:36:29.600000", "content": "There is no error message came while saving it but when I am trying to run for submission it was not generating score.
", "votes": 0, "replies": [] }, { "id": 3044177, "author_name": "ZT", "author_url": "", "post_date": "2024-11-13T06:14:16.433000", "content": "likeā¦ come back with 0?
", "votes": 0, "replies": [] } ] } ] }, { "id": 3041495, "author_name": "Michael Timbs", "author_url": "", "post_date": "2024-11-10T12:54:32.777000", "content": "Not sure but have you checked the data types? I am having a horrid time doing merge and concats with Polars changing the datatype of columns. I am having to explicitly recast everything after each operation.
\nUsing a fill by 0 or by mean could be the difference between a Float32 or Float64 or Int8 etc etc
", "votes": 0, "replies": [] }, { "id": 3040762, "author_name": "", "author_url": "", "post_date": "2024-11-09T14:53:07.657000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3040693": "for test,I did ffill, bfill\nafter merge, I did fill by median number, bfill and ffill \nif there are even one digits after merge I should get a score, but I didn't.\n\nBUT, if I do fill by 0, I will get a score.\n\nSO, I suspect that, after merge lags with test, the responder values are all NaN! that why after ffill, bfill, my predictions will result in NaN prediction which will result in \"Score Error\"\n\n\n```python\n test_df = test.to_pandas()\n lags_df = lags_.to_pandas()\n test_df = test_df.bfill()\n test_df = test_df.ffill()\n \n \n X_test = pd.merge(test_df, lags_df, on=['time_id', 'symbol_id'], how='left', suffixes = (\"\",\"_drop\"))\n \n # List of specific columns to fill with median values\n responder_columns = [f'responder_{i}_lag_1' for i in range(1, 9)]\n for col in responder_columns:\n median_value = X_test[col].median(skipna=True) # Calculate the median ignoring NaNs\n X_test[col] = X_test[col].fillna(median_value) # Use assignment to fill NaNs\n X_test = X_test.bfill()\n X_test = X_test.ffill()\n```\n\n\n\n\n\n\n\n\n\n\n", "3044834": "Forward fill and back fill can result in NaN values if there is a sequence of NaN values at the beginning or end of a dataset, or when there are continuous NaNs without non-NaN values between them. ", "3041570": "how much time does it takes to generate the score ?\n\nbecause i am also facing similar issue during submission it runs 9 hours then fails i don't understand why it fails.", "3041495": "Not sure but have you checked the data types? I am having a horrid time doing merge and concats with Polars changing the datatype of columns. I am having to explicitly recast everything after each operation.\n\nUsing a fill by 0 or by mean could be the difference between a Float32 or Float64 or Int8 etc etc", "3040762": "" } }