Hello everyone,
\nI started working with the Home Credit dataset recently, and I'm still relatively new to Kaggle and data science. I've come across a dilemma regarding the handling of external data sources. The data description mentions that some external data providers might not be available for future evaluations, which is expected. Additionally, it has been confirmed in a forum post that the whole table could be missing after a certain date, although it's also mentioned that empty tables would still be retained if all data are missing in the test sample.
\nWhile exploring public notebooks for guidance on dealing with external data, I've noticed that many include external data in their processing pipelines, such as aggregating data from the static_cb_0 dataset to create the training dataset. Handling missing values seems to be addressed by filling them with column mean/mode.
\nHowever, I'm somewhat concerned about this approach. I'm worried that if a feature from an external data source becomes the dominant predictor for the trained model, and the private test data doesn't include these features, the model's performance could suffer.
\nSo, what's the best approach to tackle this issue? Should we train multiple models and assess feature importance? If features from external data have high importance for a particular model, should we consider dropping that model from the final inference? Alternatively, would it be better to use external data solely to gain insights for better engineering of internal data, without directly incorporating it as a feature for the model or ensemble? Or perhaps my concern is invalid because the features from external data aren't influential enough? Or is it better to consider downweighting features incorporated from external data during model training?
\nI'd appreciate any insights or advice on how to handle this situation effectively. Thank you!
\nAforementioned forum post: https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/476231
", "messages": [ { "id": "2661590", "postDate": "02/21/2024 11:14:46", "content": "Hello everyone,
\nI started working with the Home Credit dataset recently, and I'm still relatively new to Kaggle and data science. I've come across a dilemma regarding the handling of external data sources. The data description mentions that some external data providers might not be available for future evaluations, which is expected. Additionally, it has been confirmed in a forum post that the whole table could be missing after a certain date, although it's also mentioned that empty tables would still be retained if all data are missing in the test sample.
\nWhile exploring public notebooks for guidance on dealing with external data, I've noticed that many include external data in their processing pipelines, such as aggregating data from the static_cb_0 dataset to create the training dataset. Handling missing values seems to be addressed by filling them with column mean/mode.
\nHowever, I'm somewhat concerned about this approach. I'm worried that if a feature from an external data source becomes the dominant predictor for the trained model, and the private test data doesn't include these features, the model's performance could suffer.
\nSo, what's the best approach to tackle this issue? Should we train multiple models and assess feature importance? If features from external data have high importance for a particular model, should we consider dropping that model from the final inference? Alternatively, would it be better to use external data solely to gain insights for better engineering of internal data, without directly incorporating it as a feature for the model or ensemble? Or perhaps my concern is invalid because the features from external data aren't influential enough? Or is it better to consider downweighting features incorporated from external data during model training?
\nI'd appreciate any insights or advice on how to handle this situation effectively. Thank you!
\nAforementioned forum post: https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/476231
", "rawMarkdown": "Hello everyone,\n\nI started working with the Home Credit dataset recently, and I'm still relatively new to Kaggle and data science. I've come across a dilemma regarding the handling of external data sources. The data description mentions that some external data providers might not be available for future evaluations, which is expected. Additionally, it has been confirmed in a forum post that the whole table could be missing after a certain date, although it's also mentioned that empty tables would still be retained if all data are missing in the test sample.\n\nWhile exploring public notebooks for guidance on dealing with external data, I've noticed that many include external data in their processing pipelines, such as aggregating data from the static_cb_0 dataset to create the training dataset. Handling missing values seems to be addressed by filling them with column mean/mode.\n\nHowever, I'm somewhat concerned about this approach. I'm worried that if a feature from an external data source becomes the dominant predictor for the trained model, and the private test data doesn't include these features, the model's performance could suffer.\n\nSo, what's the best approach to tackle this issue? Should we train multiple models and assess feature importance? If features from external data have high importance for a particular model, should we consider dropping that model from the final inference? Alternatively, would it be better to use external data solely to gain insights for better engineering of internal data, without directly incorporating it as a feature for the model or ensemble? Or perhaps my concern is invalid because the features from external data aren't influential enough? Or is it better to consider downweighting features incorporated from external data during model training?\n\nI'd appreciate any insights or advice on how to handle this situation effectively. Thank you!\n\nAforementioned forum post: https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/476231", "votes": null }, { "id": "2661625", "postDate": "02/21/2024 11:29:31", "content": "@pelinkeskin I am reasonably sure that data for certain dates may not be available but I think all columns should be available across the datasets.
", "rawMarkdown": "pelinkeskin I am reasonably sure that data for certain dates may not be available but I think all columns should be available across the datasets.", "votes": null } ], "comments": [ { "id": 2661625, "author_name": "ravi20076", "author_url": "", "post_date": "02/21/2024 11:29:31", "content": "@pelinkeskin I am reasonably sure that data for certain dates may not be available but I think all columns should be available across the datasets.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2661590": "Hello everyone,\n\nI started working with the Home Credit dataset recently, and I'm still relatively new to Kaggle and data science. I've come across a dilemma regarding the handling of external data sources. The data description mentions that some external data providers might not be available for future evaluations, which is expected. Additionally, it has been confirmed in a forum post that the whole table could be missing after a certain date, although it's also mentioned that empty tables would still be retained if all data are missing in the test sample.\n\nWhile exploring public notebooks for guidance on dealing with external data, I've noticed that many include external data in their processing pipelines, such as aggregating data from the static_cb_0 dataset to create the training dataset. Handling missing values seems to be addressed by filling them with column mean/mode.\n\nHowever, I'm somewhat concerned about this approach. I'm worried that if a feature from an external data source becomes the dominant predictor for the trained model, and the private test data doesn't include these features, the model's performance could suffer.\n\nSo, what's the best approach to tackle this issue? Should we train multiple models and assess feature importance? If features from external data have high importance for a particular model, should we consider dropping that model from the final inference? Alternatively, would it be better to use external data solely to gain insights for better engineering of internal data, without directly incorporating it as a feature for the model or ensemble? Or perhaps my concern is invalid because the features from external data aren't influential enough? Or is it better to consider downweighting features incorporated from external data during model training?\n\nI'd appreciate any insights or advice on how to handle this situation effectively. Thank you!\n\nAforementioned forum post: https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/476231", "2661625": "pelinkeskin I am reasonably sure that data for certain dates may not be available but I think all columns should be available across the datasets." }, "source": "meta" }