Say I train a 1st model using train data with PCIAT_Total to fill up blank PCIAT_Total in train data, LGBM.
\nAnd then I use all this data to train a 2nd LGBM model to predict PCIAT_Total in test data, and convert them into sii 0-3 and submit.
\nIs it a valid method or there is data leakage?
Mean Train Cohen's kappa: 1.0000
\nMean Validation Cohen's kappa: 0.3040
\nPublic Score: 0.153 😂
Say I train a 1st model using train data with PCIAT_Total to fill up blank PCIAT_Total in train data, LGBM.
\nAnd then I use all this data to train a 2nd LGBM model to predict PCIAT_Total in test data, and convert them into sii 0-3 and submit.
\nIs it a valid method or there is data leakage?
Mean Train Cohen's kappa: 1.0000
\nMean Validation Cohen's kappa: 0.3040
\nPublic Score: 0.153 😂
Mean Train Cohen's kappa: 1.0000 - this is the main issue @tomyuen
\nI am sure your target column is present in the train data (features)
The target column ('sii') is derived from the total PCIAT questionnaire scores, which is causing the leakage.
", "rawMarkdown": "The target column ('sii') is derived from the total PCIAT questionnaire scores, which is causing the leakage.", "votes": null }, { "id": "3000867", "postDate": "09/28/2024 07:05:35", "content": "Agreed, just ensure none of the PCIAT and sii columns are present in the train dataset while training models @ethanp345 @tomyuen
@ravi20076 @ethanp345
\nThanks both! I've tried it another day but still in vain.
\nI dropped sii and all PCIAT column (except Total) once I load the train_df and I've used the standard X Y split
\nX = train.drop(['PCIAT-PCIAT_Total'], axis=1)
\ny = train['PCIAT-PCIAT_Total']
I guess it's the 1st step, when I trained a LGBM to fill in missed PCIAT_Total in train_df, that there're too many missed value but too few valid sample, that the 1st model memorize the pattern and filled in some toxic PCIAT_Total prediction.
\nSo when I trained the 2nd LGBM, the train score is so high, but of course it doesn't generalize to unseen data.
Most likely I'll drop this idea of 'filling blank PCIAT_Total in train_df' at the moment 😅, and reference how others are doing.
\nThanks again.
You need to drop all PCIAT columns (inclusive of the total column), since that column specifically determines our target ('sii').
\ntarget = \n\ntrain_cols = (train.)\ntest_cols = (test.)\n\n\ncols_to_remove = train_cols - test_cols \n\n\ncols_to_remove(target)\n\n\ntrain = train( = cols_to_remove)\nNow feel free to split to X,y and begin modeling.
\nHope this helps!
", "rawMarkdown": "You need to drop all PCIAT columns (inclusive of the total column), since that column specifically determines our target ('sii'). \n\n```\ntarget = 'sii'\n\ntrain_cols = set(train.columns)\ntest_cols = set(test.columns)\n\n'Finding columns that need to be removed'\ncols_to_remove = train_cols - test_cols \n\n'Removing our target variable from this list'\ncols_to_remove.remove(target)\n\n'Removing columns'\ntrain = train.drop(columns = cols_to_remove)\n\n```\n\nNow feel free to split to X,y and begin modeling. \n\nHope this helps!", "votes": null } ], "comments": [ { "id": 3000595, "author_name": "ravi20076", "author_url": "", "post_date": "09/27/2024 19:44:17", "content": "Mean Train Cohen's kappa: 1.0000 - this is the main issue @tomyuen
\nI am sure your target column is present in the train data (features)
The target column ('sii') is derived from the total PCIAT questionnaire scores, which is causing the leakage.
", "votes": null, "replies": [ { "id": 3000867, "author_name": "ravi20076", "author_url": "", "post_date": "09/28/2024 07:05:35", "content": "Agreed, just ensure none of the PCIAT and sii columns are present in the train dataset while training models @ethanp345 @tomyuen
@ravi20076 @ethanp345
\nThanks both! I've tried it another day but still in vain.
\nI dropped sii and all PCIAT column (except Total) once I load the train_df and I've used the standard X Y split
\nX = train.drop(['PCIAT-PCIAT_Total'], axis=1)
\ny = train['PCIAT-PCIAT_Total']
I guess it's the 1st step, when I trained a LGBM to fill in missed PCIAT_Total in train_df, that there're too many missed value but too few valid sample, that the 1st model memorize the pattern and filled in some toxic PCIAT_Total prediction.
\nSo when I trained the 2nd LGBM, the train score is so high, but of course it doesn't generalize to unseen data.
Most likely I'll drop this idea of 'filling blank PCIAT_Total in train_df' at the moment 😅, and reference how others are doing.
\nThanks again.
You need to drop all PCIAT columns (inclusive of the total column), since that column specifically determines our target ('sii').
\ntarget = \n\ntrain_cols = (train.)\ntest_cols = (test.)\n\n\ncols_to_remove = train_cols - test_cols \n\n\ncols_to_remove(target)\n\n\ntrain = train( = cols_to_remove)\nNow feel free to split to X,y and begin modeling.
\nHope this helps!
", "votes": null, "replies": [] } ] } ] } ] } ], "raw_markdown_by_id": { "3000591": "Say I train a 1st model using train data **with** PCIAT_Total to fill up **blank** PCIAT_Total in train data, LGBM.\nAnd then I use all this data to train a 2nd LGBM model to predict PCIAT_Total in test data, and convert them into sii 0-3 and submit.\nIs it a valid method or there is data leakage? \n\nMean Train Cohen's kappa: 1.0000\nMean Validation Cohen's kappa: 0.3040\nPublic Score: 0.153 😂", "3000595": "`Mean Train Cohen's kappa: 1.0000` - this is the main issue @tomyuen \nI am sure your target column is present in the train data (features)", "3000730": "The target column ('sii') is derived from the total PCIAT questionnaire scores, which is causing the leakage.", "3000867": "Agreed, just ensure none of the `PCIAT and sii columns` are present in the train dataset while training models @ethanp345 @tomyuen", "3001469": "ravi20076 @ethanp345 \nThanks both! I've tried it another day but still in vain. \nI dropped sii and all PCIAT column (except Total) once I load the train_df and I've used the standard X Y split\n` X = train.drop(['PCIAT-PCIAT_Total'], axis=1)`\n` y = train['PCIAT-PCIAT_Total']`\n\nI guess it's the 1st step, when I trained a LGBM to fill in missed PCIAT_Total in train_df, that there're too many missed value but too few valid sample, that the 1st model memorize the pattern and filled in some toxic PCIAT_Total prediction.\nSo when I trained the 2nd LGBM, the train score is so high, but of course it doesn't generalize to unseen data.\n\nMost likely I'll drop this idea of 'filling blank PCIAT_Total in train_df' at the moment 😅, and reference how others are doing.\nThanks again.", "3001485": "You need to drop all PCIAT columns (inclusive of the total column), since that column specifically determines our target ('sii'). \n\n```\ntarget = 'sii'\n\ntrain_cols = set(train.columns)\ntest_cols = set(test.columns)\n\n'Finding columns that need to be removed'\ncols_to_remove = train_cols - test_cols \n\n'Removing our target variable from this list'\ncols_to_remove.remove(target)\n\n'Removing columns'\ntrain = train.drop(columns = cols_to_remove)\n\n```\n\nNow feel free to split to X,y and begin modeling. \n\nHope this helps!" }, "source": "meta" }