![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F20132679%2F39af4a59c6c1e4282d46aa159ce63cae%2FScreenshot%202024-05-14%20171739.png?generation=1715682156871611&alt=media\")
I noticed that some popular notebooks are dropping columns with some 3000 categories, but I think that we should just collapse them
\nHere is a quick function:
\ndef collapseData(X, , originalDf, threshold=, replaceVal=\"NOTPRESENTVALUE\"):\n dataValCounts = originalDf[].value_counts()/len(originalDf)\n replaceDict = {i:replaceVal i dataValCounts[dataValCounts<=threshold].}\n\n XUnique = X[].()\n replaceDict.({i:replaceVal i XUnique[~np.isin(XUnique, dataValCounts.)]})\n\n X[].map(lambda x: replaceDict.(x, x))\nI used this for the person dataset thing and significantly reduced categories from like 7,500 to just 500 (improving cv by about 0.004)
", "messages": [ { "id": "2747159", "postDate": "04/11/2024 17:48:25", "content": "I noticed that some popular notebooks are dropping columns with some 3000 categories, but I think that we should just collapse them
\nHere is a quick function:
\ndef collapseData(X, , originalDf, threshold=, replaceVal=\"NOTPRESENTVALUE\"):\n dataValCounts = originalDf[].value_counts()/len(originalDf)\n replaceDict = {i:replaceVal i dataValCounts[dataValCounts<=threshold].}\n\n XUnique = X[].()\n replaceDict.({i:replaceVal i XUnique[~np.isin(XUnique, dataValCounts.)]})\n\n X[].map(lambda x: replaceDict.(x, x))\nI used this for the person dataset thing and significantly reduced categories from like 7,500 to just 500 (improving cv by about 0.004)
", "rawMarkdown": "I noticed that some popular notebooks are dropping columns with some 3000 categories, but I think that we should just collapse them\n\nHere is a quick function:\n```\ndef collapseData(X, column, originalDf, threshold=0.001, replaceVal=\"NOTPRESENTVALUE\"):\n dataValCounts = originalDf[column].value_counts()/len(originalDf)\n replaceDict = {i:replaceVal for i in dataValCounts[dataValCounts<=threshold].index}\n \n XUnique = X[column].unique()\n replaceDict.update({i:replaceVal for i in XUnique[~np.isin(XUnique, dataValCounts.index)]})\n \n return X[column].map(lambda x: replaceDict.get(x, x))\n```\n\nI used this for the person dataset thing and significantly reduced categories from like 7,500 to just 500 (improving cv by about 0.004)", "votes": null }, { "id": "2748071", "postDate": "04/12/2024 08:28:26", "content": "Thanks for your suggestion. I will try it
", "rawMarkdown": "Thanks for your suggestion. I will try it", "votes": null }, { "id": "2748406", "postDate": "04/12/2024 12:28:41", "content": "I'm back again. It's unfortunate that I didn't see any improvement in scores before exhausting today's DAILY SUBMISSIONS. Hopefully, adjusting the threshold tomorrow will change this situation. Good luck to you!
", "rawMarkdown": "I'm back again. It's unfortunate that I didn't see any improvement in scores before exhausting today's DAILY SUBMISSIONS. Hopefully, adjusting the threshold tomorrow will change this situation. Good luck to you!", "votes": null }, { "id": "2748701", "postDate": "04/12/2024 14:41:00", "content": "very nice and interesting, but I have a doubt, doesn't the internal binning of lgbm take care of this large number of categories in a column? I would be very much interested in comparing this approach with the internal binning strategy of lgbm.
", "rawMarkdown": "very nice and interesting, but I have a doubt, doesn't the internal binning of lgbm take care of this large number of categories in a column? I would be very much interested in comparing this approach with the internal binning strategy of lgbm.", "votes": null }, { "id": "2748760", "postDate": "04/12/2024 15:37:01", "content": "Maybe, it may be the case but I actually also use this to make count encoding columns which I believe some people are talking about. Also, it's just nice to reduce the column count. 7,500 columns is just a little much.
", "rawMarkdown": "Maybe, it may be the case but I actually also use this to make count encoding columns which I believe some people are talking about. Also, it's just nice to reduce the column count. 7,500 columns is just a little much.", "votes": null }, { "id": "2748956", "postDate": "04/12/2024 18:31:06", "content": "This approach can also be used instead of removing columns with a number of categories greater than a certain threshold. Therefore, some columns can be kept.
", "rawMarkdown": "This approach can also be used instead of removing columns with a number of categories greater than a certain threshold. Therefore, some columns can be kept.", "votes": null }, { "id": "2750179", "postDate": "04/13/2024 13:51:48", "content": "Thanks for sharing will try this & update if CV improves !!
", "rawMarkdown": "Thanks for sharing will try this & update if CV improves !!", "votes": null }, { "id": "2809721", "postDate": "05/13/2024 00:28:50", "content": "Thanks for sharing.
", "rawMarkdown": "Thanks for sharing.", "votes": null }, { "id": "2811860", "postDate": "05/14/2024 00:57:50", "content": "Which threshold did you choose?
", "rawMarkdown": "Which threshold did you choose?", "votes": null }, { "id": "2811984", "postDate": "05/14/2024 03:48:44", "content": "Thanks, I'll give it a try.
", "rawMarkdown": "Thanks, I'll give it a try.", "votes": null }, { "id": "2812603", "postDate": "05/14/2024 10:23:13", "content": "I have tried but catched this bug.It said that str div str. Can anyone help me solve it? Thanks.
Here you have an encoding depending on the cardinality that covers all the categorical columns https://www.kaggle.com/code/eu1234/creditrisk-sklearn-pipeline-integrated-metric
", "rawMarkdown": "Here you have an encoding depending on the cardinality that covers all the categorical columns https://www.kaggle.com/code/eu1234/creditrisk-sklearn-pipeline-integrated-metric", "votes": null } ], "comments": [ { "id": 2748071, "author_name": "tianya55", "author_url": "", "post_date": "04/12/2024 08:28:26", "content": "Thanks for your suggestion. I will try it
", "votes": null, "replies": [ { "id": 2748406, "author_name": "tianya55", "author_url": "", "post_date": "04/12/2024 12:28:41", "content": "I'm back again. It's unfortunate that I didn't see any improvement in scores before exhausting today's DAILY SUBMISSIONS. Hopefully, adjusting the threshold tomorrow will change this situation. Good luck to you!
", "votes": null, "replies": [ { "id": 2811860, "author_name": "phamhoanglenguyen", "author_url": "", "post_date": "05/14/2024 00:57:50", "content": "Which threshold did you choose?
", "votes": null, "replies": [] } ] } ] }, { "id": 2748701, "author_name": "shreyas9181", "author_url": "", "post_date": "04/12/2024 14:41:00", "content": "very nice and interesting, but I have a doubt, doesn't the internal binning of lgbm take care of this large number of categories in a column? I would be very much interested in comparing this approach with the internal binning strategy of lgbm.
", "votes": null, "replies": [ { "id": 2748760, "author_name": "kawaiicoderuwu", "author_url": "", "post_date": "04/12/2024 15:37:01", "content": "Maybe, it may be the case but I actually also use this to make count encoding columns which I believe some people are talking about. Also, it's just nice to reduce the column count. 7,500 columns is just a little much.
", "votes": null, "replies": [] }, { "id": 2748956, "author_name": "andreynesterov", "author_url": "", "post_date": "04/12/2024 18:31:06", "content": "This approach can also be used instead of removing columns with a number of categories greater than a certain threshold. Therefore, some columns can be kept.
", "votes": null, "replies": [] } ] }, { "id": 2750179, "author_name": "kunduruanil", "author_url": "", "post_date": "04/13/2024 13:51:48", "content": "Thanks for sharing will try this & update if CV improves !!
", "votes": null, "replies": [] }, { "id": 2809721, "author_name": "dannykan", "author_url": "", "post_date": "05/13/2024 00:28:50", "content": "Thanks for sharing.
", "votes": null, "replies": [] }, { "id": 2811984, "author_name": "kamonabeyy", "author_url": "", "post_date": "05/14/2024 03:48:44", "content": "Thanks, I'll give it a try.
", "votes": null, "replies": [] }, { "id": 2812603, "author_name": "nguynphmhongl", "author_url": "", "post_date": "05/14/2024 10:23:13", "content": "I have tried but catched this bug.It said that str div str. Can anyone help me solve it? Thanks.
Here you have an encoding depending on the cardinality that covers all the categorical columns https://www.kaggle.com/code/eu1234/creditrisk-sklearn-pipeline-integrated-metric
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2747159": "I noticed that some popular notebooks are dropping columns with some 3000 categories, but I think that we should just collapse them\n\nHere is a quick function:\n```\ndef collapseData(X, column, originalDf, threshold=0.001, replaceVal=\"NOTPRESENTVALUE\"):\n dataValCounts = originalDf[column].value_counts()/len(originalDf)\n replaceDict = {i:replaceVal for i in dataValCounts[dataValCounts<=threshold].index}\n \n XUnique = X[column].unique()\n replaceDict.update({i:replaceVal for i in XUnique[~np.isin(XUnique, dataValCounts.index)]})\n \n return X[column].map(lambda x: replaceDict.get(x, x))\n```\n\nI used this for the person dataset thing and significantly reduced categories from like 7,500 to just 500 (improving cv by about 0.004)", "2748071": "Thanks for your suggestion. I will try it", "2748406": "I'm back again. It's unfortunate that I didn't see any improvement in scores before exhausting today's DAILY SUBMISSIONS. Hopefully, adjusting the threshold tomorrow will change this situation. Good luck to you!", "2748701": "very nice and interesting, but I have a doubt, doesn't the internal binning of lgbm take care of this large number of categories in a column? I would be very much interested in comparing this approach with the internal binning strategy of lgbm.", "2748760": "Maybe, it may be the case but I actually also use this to make count encoding columns which I believe some people are talking about. Also, it's just nice to reduce the column count. 7,500 columns is just a little much.", "2748956": "This approach can also be used instead of removing columns with a number of categories greater than a certain threshold. Therefore, some columns can be kept.", "2750179": "Thanks for sharing will try this & update if CV improves !!", "2809721": "Thanks for sharing.", "2811860": "Which threshold did you choose?", "2811984": "Thanks, I'll give it a try.", "2812603": "I have tried but catched this bug.It said that str div str. Can anyone help me solve it? Thanks.", "2813540": "Here you have an encoding depending on the cardinality that covers all the categorical columns https://www.kaggle.com/code/eu1234/creditrisk-sklearn-pipeline-integrated-metric" }, "source": "meta" }