![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4746009%2Fe5e6cbe3eca0f8222489261d600bc576%2F1656511567090.jpg?generation=1656511604640284&alt=media\")
from sklearn.impute import SimpleImputer\nmy_imputer = SimpleImputer()\ndata_with_imputed_values = my_imputer.fit_transform(original_data)\n
from sklearn.impute import SimpleImputer\nmy_imputer = SimpleImputer()\ndata_with_imputed_values = my_imputer.fit_transform(original_data)\nHere are some good notebooks for handling missing values:
\nThanks for sharing :)
", "rawMarkdown": "Thanks for sharing :)" } ] }, { "id": 1838681, "postDate": "2022-06-30T18:01:46.387Z", "content": "Hi @shanggangli , there are many ways you can do missing value imputation:
\nHope this helps!
", "rawMarkdown": "Hi @shanggangli , there are many ways you can do missing value imputation:\n- Imputing with any statistical measure like mean, median, model depending upon the column.\n- Using any tree based learning method to find suitable replacement for missing values in data.\n- Also you can do time series based imputation since this is a time series problem thus forward filling/ back filling can be done or some other imputation by applying a transformation is also a feasible option.\n\nHope this helps! ", "votes": 3, "replies": [ { "id": 1839629, "postDate": "2022-07-01T15:20:09.053Z", "content": "Wow, thanks so much. l get more knowledge~
", "rawMarkdown": "Wow, thanks so much. l get more knowledge~", "votes": 1 }, { "id": 1840565, "postDate": "2022-07-02T11:15:23.330Z", "content": "\n\nAlso you can do time series based imputation since this is a time series problem thus forward filling/ back filling can be done or some other imputation by applying a transformation is also a feasible option.
\n
nice methods @shanggangli, So we fill nan values of let say monday with tuesday for a given customer in forward filling.
\nNow what can be done when we have nan in last row of a customer. (i.e. there is no next time-stamp)
This was a good question, which what can be done when we have nan in last row of a customer. (i.e. there is no next time-stamp)
In my knowledge, we can use the most frequent values to fill it or base some models(ARIMA,MLP, LSTM and so on) to predict it.
\nBut, if there are quite a lot missing data needed to fill, that still will be a problem.
", "rawMarkdown": "This was a good question, which ` what can be done when we have nan in last row of a customer. (i.e. there is no next time-stamp)`\n\nIn my knowledge, we can use the most frequent values to fill it or base some models(ARIMA,MLP, LSTM and so on) to predict it.\n\nBut, if there are quite a lot missing data needed to fill, that still will be a problem.", "votes": 1 } ] }, { "id": 1842361, "postDate": "2022-07-04T00:03:22.050Z", "content": "I am wondering if those missing values aren't a clue for the kind of transactions that the person have done at a particular statement.
", "rawMarkdown": "I am wondering if those missing values aren't a clue for the kind of transactions that the person have done at a particular statement.", "votes": 1 }, { "id": 1838914, "postDate": "2022-07-01T02:10:31.687Z", "content": "I would suggest using XGBoost or Lightgbm and let the algorithm take care of the missing values.
\nHow to deal with missing values
\nMissing Value Handle
these were good insights. Thanks a lot.❤️
", "rawMarkdown": "these were good insights. Thanks a lot.❤️", "votes": 1 }, { "id": 1839662, "postDate": "2022-07-01T15:48:57.913Z", "content": "you are more than welcome! Good luck!
", "rawMarkdown": "you are more than welcome! Good luck!", "votes": 1 } ] }, { "id": 1893253, "postDate": "2022-08-10T16:50:57.797Z", "content": "I was also interested in this and was thinking of starting a thread .. Specially how are folks dealing with missing values specially for Neural Network models which may be more sensitive to this , A simple FillNA -1 or likewise solution I think will not cut it for the model to score high . For GBDT though still it shall be ok .
", "rawMarkdown": "I was also interested in this and was thinking of starting a thread .. Specially how are folks dealing with missing values specially for Neural Network models which may be more sensitive to this , A simple FillNA -1 or likewise solution I think will not cut it for the model to score high . For GBDT though still it shall be ok . " }, { "id": 1838424, "postDate": "2022-06-30T14:04:53.637Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1839679, "author_name": "The Devastator", "author_url": "", "post_date": "2022-07-01T16:12:24.677000", "content": "Here are some good notebooks for handling missing values:
\nThanks for sharing :)
", "votes": 0, "replies": [] } ] }, { "id": 1838681, "author_name": "Ishaan Jain", "author_url": "", "post_date": "2022-06-30T18:01:46.387000", "content": "Hi @shanggangli , there are many ways you can do missing value imputation:
\nHope this helps!
", "votes": 3, "replies": [ { "id": 1839629, "author_name": "SgangX", "author_url": "", "post_date": "2022-07-01T15:20:09.053000", "content": "Wow, thanks so much. l get more knowledge~
", "votes": 1, "replies": [] }, { "id": 1840565, "author_name": "AKR", "author_url": "", "post_date": "2022-07-02T11:15:23.330000", "content": "\n\nAlso you can do time series based imputation since this is a time series problem thus forward filling/ back filling can be done or some other imputation by applying a transformation is also a feasible option.
\n
nice methods @shanggangli, So we fill nan values of let say monday with tuesday for a given customer in forward filling.
\nNow what can be done when we have nan in last row of a customer. (i.e. there is no next time-stamp)
This was a good question, which what can be done when we have nan in last row of a customer. (i.e. there is no next time-stamp)
In my knowledge, we can use the most frequent values to fill it or base some models(ARIMA,MLP, LSTM and so on) to predict it.
\nBut, if there are quite a lot missing data needed to fill, that still will be a problem.
", "votes": 1, "replies": [] } ] }, { "id": 1842361, "author_name": "HABP", "author_url": "", "post_date": "2022-07-04T00:03:22.050000", "content": "I am wondering if those missing values aren't a clue for the kind of transactions that the person have done at a particular statement.
", "votes": 1, "replies": [] }, { "id": 1838914, "author_name": "1110Ra", "author_url": "", "post_date": "2022-07-01T02:10:31.687000", "content": "I would suggest using XGBoost or Lightgbm and let the algorithm take care of the missing values.
\nHow to deal with missing values
\nMissing Value Handle
these were good insights. Thanks a lot.❤️
", "votes": 1, "replies": [] }, { "id": 1839662, "author_name": "1110Ra", "author_url": "", "post_date": "2022-07-01T15:48:57.913000", "content": "you are more than welcome! Good luck!
", "votes": 1, "replies": [] } ] }, { "id": 1893253, "author_name": "Gaurav Rawat", "author_url": "", "post_date": "2022-08-10T16:50:57.797000", "content": "I was also interested in this and was thinking of starting a thread .. Specially how are folks dealing with missing values specially for Neural Network models which may be more sensitive to this , A simple FillNA -1 or likewise solution I think will not cut it for the model to score high . For GBDT though still it shall be ok .
", "votes": 0, "replies": [] }, { "id": 1838424, "author_name": "", "author_url": "", "post_date": "2022-06-30T14:04:53.637000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1837362": "\n#### You can see, there are some features missing rate of over 90%.\n#### And you guys, how to deal with these features?\n\n\n\n\n#### l also see some information *[here](https://www.kaggle.com/code/dansbecker/handling-missing-values/notebook)*\n#### there are some tips in the URL:\n##### 1. drop the features\n##### 2. Imputation\n```\nfrom sklearn.impute import SimpleImputer\nmy_imputer = SimpleImputer()\ndata_with_imputed_values = my_imputer.fit_transform(original_data)\n```\n\n## So, what about you?\n", "1839679": "Here are some good notebooks for handling missing values:\n\n- [Handling missing data MCAR, MAR and MNAR (Part I)](https://www.kaggle.com/yassirarezki/handling-missing-data-mcar-mar-and-mnar-part-i)\n- [Handling Missing Values](https://www.kaggle.com/dansbecker/handling-missing-values)\n- [Data Cleaning Challenge: Handling missing values](https://www.kaggle.com/rtatman/data-cleaning-challenge-handling-missing-values)\n- [Handling Missing Values](https://www.kaggle.com/alexisbcook/handling-missing-values)\n- [A Guide to Handling Missing values in Python](https://www.kaggle.com/parulpandey/a-guide-to-handling-missing-values-in-python)\n- [Handling Missing Values](https://www.kaggle.com/debarshichanda/handling-missing-values)\n- [Handling Missing Values](https://www.kaggle.com/sorzhe/handling-missing-values)\n- [EDA, Handling missing values using Regression](https://www.kaggle.com/modojj/eda-handling-missing-values-using-regression)\n- [Data Cleaning Challenge: Imputing missing values For R](https://www.kaggle.com/rtatman/data-cleaning-challenge-imputing-missing-values)\n- [Data Cleaning Challenge: Handling missing values](https://www.kaggle.com/grjasewe/data-cleaning-challenge-handling-missing-values)\n- [Missing Values Handling](https://www.kaggle.com/mansipriya/missing-values-handling)\n- [Tutorial - Handling missing values For R](https://www.kaggle.com/itacdonev/tutorial-handling-missing-values)\n- [Simple techniques for missing data imputation](https://www.kaggle.com/residentmario/simple-techniques-for-missing-data-imputation)\n- [Missing Data Imputation using Regression](https://www.kaggle.com/shashankasubrahmanya/missing-data-imputation-using-regression)\n- [Handling Missing Values FIFA 19](https://www.kaggle.com/rk2802/handling-missing-values-fifa-19)", "1838681": "Hi @shanggangli , there are many ways you can do missing value imputation:\n- Imputing with any statistical measure like mean, median, model depending upon the column.\n- Using any tree based learning method to find suitable replacement for missing values in data.\n- Also you can do time series based imputation since this is a time series problem thus forward filling/ back filling can be done or some other imputation by applying a transformation is also a feasible option.\n\nHope this helps! ", "1842361": "I am wondering if those missing values aren't a clue for the kind of transactions that the person have done at a particular statement.", "1838914": "I would suggest using XGBoost or Lightgbm and let the algorithm take care of the missing values.\n\n[How to deal with missing values](https://xgboost.readthedocs.io/en/stable/faq.html#how-to-deal-with-missing-values)\n[Missing Value Handle](https://lightgbm.readthedocs.io/en/latest/Advanced-Topics.html#missing-value-handle)", "1893253": "I was also interested in this and was thinking of starting a thread .. Specially how are folks dealing with missing values specially for Neural Network models which may be more sensitive to this , A simple FillNA -1 or likewise solution I think will not cut it for the model to score high . For GBDT though still it shall be ok . ", "1838424": "" } }