amtinstpaidbefduel24m_4187115A (Number of instalments paid before due date in the last 24 months.)\nhas 100% null rate in the first 10 weeks! and the null ratio is gradually decreasing over weeks:
\n
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F690886%2F9b3c50646386aad6511871bcb8ff40d7%2FScreenshot%20from%202024-02-14%2023-28-56.png?generation=1707924557438357&alt=media\")
I am looking at the null count per week. There are quite a lot of features
\n1) have 100% null rate in the first week
\n2) have 100% null rate in the last week
\n3) have a consecutive 100% null rate across multiple weeks.
I only looked into the 2 static tables. Here are some features might have a trendy null rate:
\n[', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ']\nFor example amtinstpaidbefduel24m_4187115A (Number of instalments paid before due date in the last 24 months.)
\nhas 100% null rate in the first 10 weeks! and the null ratio is gradually decreasing over weeks:
\n
clientscnt_136L is opposite:
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F690886%2F8ef1c742aa3f5384d59c8fedbb5da159%2FScreenshot%20from%202024-02-14%2023-31-17.png?generation=1707924686407129&alt=media\")
Some ideas:
\n1) for features that have consecutive null in the last few weeks, we should drop them completely.
\n2) for features that have consecutive null in the first few weeks, we can either drop them or find some ways to impute them (might be from other tables)
\n3) Only train on the latest few weeks that have fewer null issues…
I am looking at the null count per week. There are quite a lot of features
\n1) have 100% null rate in the first week
\n2) have 100% null rate in the last week
\n3) have a consecutive 100% null rate across multiple weeks.
I only looked into the 2 static tables. Here are some features might have a trendy null rate:
\n[', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ']\nFor example amtinstpaidbefduel24m_4187115A (Number of instalments paid before due date in the last 24 months.)
\nhas 100% null rate in the first 10 weeks! and the null ratio is gradually decreasing over weeks:
\n
clientscnt_136L is opposite:
\n
Some ideas:
\n1) for features that have consecutive null in the last few weeks, we should drop them completely.
\n2) for features that have consecutive null in the first few weeks, we can either drop them or find some ways to impute them (might be from other tables)
\n3) Only train on the latest few weeks that have fewer null issues…
I tried to remove features with 100% null rate in the last 2 weeks here
\nhttps://www.kaggle.com/code/kingychiu/dropping-deprecated-features
but the lb changed from 0.559 (original) to 0.558 (dropped)…
\nI guess the null rate is something we need to guess or prompt in the test set…
\nThe data quality is far lower than what a usual Kaggle competition is. Need to do some more EDA.
", "rawMarkdown": "I tried to remove features with 100% null rate in the last 2 weeks here\nhttps://www.kaggle.com/code/kingychiu/dropping-deprecated-features\n\nbut the lb changed from 0.559 (original) to 0.558 (dropped)...\n\nI guess the null rate is something we need to guess or prompt in the test set...\n\nThe data quality is far lower than what a usual Kaggle competition is. Need to do some more EDA.", "votes": null }, { "id": "2653103", "postDate": "02/15/2024 07:51:02", "content": "@kingychiu are those trends there in depth=0 tables as well ?
", "rawMarkdown": "kingychiu are those trends there in depth=0 tables as well ?", "votes": null }, { "id": "2653105", "postDate": "02/15/2024 07:58:52", "content": "Yes. I am only working with the 0-depth table. Don't want to be too overwhelmed haha…
", "rawMarkdown": "Yes. I am only working with the 0-depth table. Don't want to be too overwhelmed haha...", "votes": null }, { "id": "2654801", "postDate": "02/16/2024 12:45:07", "content": "Interesting! thanks for sharing!
\nIn case it helps you, I have published a nb showing missing values from all tables link .
\neg. in clientscnt_136L only 417 rows out of 1.5m contain num values..
\nhowever rest of clientscnt_* seem better.
Any suggestions on better visualization are welcome!
", "rawMarkdown": "Interesting! thanks for sharing!\n\nIn case it helps you, I have published a nb showing missing values from all tables [link](https://www.kaggle.com/code/imeintanis/home-credit-2024-eda-missing-values) . \n\neg. in `clientscnt_136L` only 417 rows out of 1.5m contain num values.. \nhowever rest of `clientscnt_*` seem better. \n\nAny suggestions on better visualization are welcome!", "votes": null }, { "id": "2654823", "postDate": "02/16/2024 13:06:49", "content": "Why not just average the missing values from a reasonable window. The approach might can be susceptible to variance. But I do think on certain reasonable features that tend to have low variance and follows a trend, itwould make it beneficial
", "rawMarkdown": "Why not just average the missing values from a reasonable window. The approach might can be susceptible to variance. But I do think on certain reasonable features that tend to have low variance and follows a trend, itwould make it beneficial", "votes": null } ], "comments": [ { "id": 2653085, "author_name": "kingychiu", "author_url": "", "post_date": "02/15/2024 07:25:13", "content": "I tried to remove features with 100% null rate in the last 2 weeks here
\nhttps://www.kaggle.com/code/kingychiu/dropping-deprecated-features
but the lb changed from 0.559 (original) to 0.558 (dropped)…
\nI guess the null rate is something we need to guess or prompt in the test set…
\nThe data quality is far lower than what a usual Kaggle competition is. Need to do some more EDA.
", "votes": null, "replies": [] }, { "id": 2653103, "author_name": "davutpolat", "author_url": "", "post_date": "02/15/2024 07:51:02", "content": "@kingychiu are those trends there in depth=0 tables as well ?
", "votes": null, "replies": [ { "id": 2653105, "author_name": "kingychiu", "author_url": "", "post_date": "02/15/2024 07:58:52", "content": "Yes. I am only working with the 0-depth table. Don't want to be too overwhelmed haha…
", "votes": null, "replies": [] } ] }, { "id": 2654801, "author_name": "imeintanis", "author_url": "", "post_date": "02/16/2024 12:45:07", "content": "Interesting! thanks for sharing!
\nIn case it helps you, I have published a nb showing missing values from all tables link .
\neg. in clientscnt_136L only 417 rows out of 1.5m contain num values..
\nhowever rest of clientscnt_* seem better.
Any suggestions on better visualization are welcome!
", "votes": null, "replies": [] }, { "id": 2654823, "author_name": "skaarface", "author_url": "", "post_date": "02/16/2024 13:06:49", "content": "Why not just average the missing values from a reasonable window. The approach might can be susceptible to variance. But I do think on certain reasonable features that tend to have low variance and follows a trend, itwould make it beneficial
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2652154": "I am looking at the null count per week. There are quite a lot of features \n1) have 100% null rate in the first week\n2) have 100% null rate in the last week\n3) have a consecutive 100% null rate across multiple weeks.\n\nI only looked into the 2 static tables. Here are some features might have a trendy null rate:\n```\n['amtinstpaidbefduel24m_4187115A', 'avgdbddpdlast3m_4187120P', 'avgdbdtollast24m_4525197P', 'avglnamtstart24m_4525187A', 'avgoutstandbalancel6m_4187114A', 'avgpmtlast12m_4525200A', 'clientscnt_136L', 'datelastinstal40dpd_247D', 'dtlastpmtallstes_4499206D', 'interestrategrace_34L', 'maxannuity_4075009A', 'maxdbddpdtollast6m_4187119P', 'maxlnamtstart6m_4525199A', 'maxoutstandbalancel12m_4187113A', 'maxpmtlast3m_4525190A', 'mindbdtollast24m_4525191P', 'numinstlswithdpd5_4187116L', 'numinstmatpaidtearly2d_4499204L', 'numinstpaid_4499208L', 'numinstpaidearly3dest_4493216L', 'numinstpaidearly5dest_4493211L', 'numinstpaidearly5dobd_4499205L', 'numinstpaidearlyest_4493214L', 'numinstpaidlastcontr_4325080L', 'numinstregularpaidest_4493210L', 'numinsttopaygrest_4493213L', 'numinstunpaidmaxest_4493212L', 'payvacationpostpone_4187118D', 'sumoutstandtotalest_4493215A', 'totinstallast1m_4525188A', 'validfrom_1069D', 'assignmentdate_238D', 'assignmentdate_4527235D', 'assignmentdate_4955616D', 'birthdate_574D', 'contractssum_5085716L', 'pmtaverage_3A', 'pmtaverage_4527227A', 'pmtaverage_4955615A', 'pmtcount_4527229L', 'pmtcount_4955617L', 'pmtcount_693L', 'pmtscount_423L', 'pmtssum_45A', 'requesttype_4525192L', 'responsedate_1012D', 'responsedate_4527233D', 'responsedate_4917613D']\n```\n\nFor example `amtinstpaidbefduel24m_4187115A` (Number of instalments paid before due date in the last 24 months.)\nhas 100% null rate in the first 10 weeks! and the null ratio is gradually decreasing over weeks:\n\n\n`clientscnt_136L` is opposite:\n\n\n\nSome ideas:\n1) for features that have consecutive null in the last few weeks, we should drop them completely.\n2) for features that have consecutive null in the first few weeks, we can either drop them or find some ways to impute them (might be from other tables)\n3) Only train on the latest few weeks that have fewer null issues...", "2653085": "I tried to remove features with 100% null rate in the last 2 weeks here\nhttps://www.kaggle.com/code/kingychiu/dropping-deprecated-features\n\nbut the lb changed from 0.559 (original) to 0.558 (dropped)...\n\nI guess the null rate is something we need to guess or prompt in the test set...\n\nThe data quality is far lower than what a usual Kaggle competition is. Need to do some more EDA.", "2653103": "kingychiu are those trends there in depth=0 tables as well ?", "2653105": "Yes. I am only working with the 0-depth table. Don't want to be too overwhelmed haha...", "2654801": "Interesting! thanks for sharing!\n\nIn case it helps you, I have published a nb showing missing values from all tables [link](https://www.kaggle.com/code/imeintanis/home-credit-2024-eda-missing-values) . \n\neg. in `clientscnt_136L` only 417 rows out of 1.5m contain num values.. \nhowever rest of `clientscnt_*` seem better. \n\nAny suggestions on better visualization are welcome!", "2654823": "Why not just average the missing values from a reasonable window. The approach might can be susceptible to variance. But I do think on certain reasonable features that tend to have low variance and follows a trend, itwould make it beneficial" }, "source": "meta" }