Has anyone successfully implemented one hot encoding, because many popular notebooks don't use it and it takes more than 22 GB of data for an 8 GB dataset that I am fitting into it.
\nIt overflows on memory each time and remainder=passthrough is the main cause, it's fine if I do drop, but I need the other data. I don't understand why an 8GB dataset uses that much memory.
Has anyone successfully implemented one hot encoding, because many popular notebooks don't use it and it takes more than 22 GB of data for an 8 GB dataset that I am fitting into it.
\nIt overflows on memory each time and remainder=passthrough is the main cause, it's fine if I do drop, but I need the other data. I don't understand why an 8GB dataset uses that much memory.
You should look at the number of categories first and then decide if one-hot is needed. If you have too many categories, then one-hot will do more harm than good @kawaiicoderuwu
", "rawMarkdown": "You should look at the number of categories first and then decide if one-hot is needed. If you have too many categories, then one-hot will do more harm than good @kawaiicoderuwu", "votes": null }, { "id": "2750374", "postDate": "04/13/2024 16:12:45", "content": "I tried one-hot encoding, but without significant gain (about +0.001). Moreover, many popular packages support preprocessing of categorical data, e.g. \"LightGBM offers good accuracy with integer-coded categorical features\", AutoML libs also provide processing of categories using label encoding, one-hot encoding, etc. So I decided to reject the idea of preprocessing categorical data using one-hot encoding (at least for now).
", "rawMarkdown": "I tried one-hot encoding, but without significant gain (about +0.001). Moreover, many popular packages support preprocessing of categorical data, e.g. \"LightGBM offers good accuracy with integer-coded categorical features\", AutoML libs also provide processing of categories using label encoding, one-hot encoding, etc. So I decided to reject the idea of preprocessing categorical data using one-hot encoding (at least for now).", "votes": null }, { "id": "2750991", "postDate": "04/14/2024 00:41:10", "content": "Oh I thought onehot or label encoding was always needed with categorical data. I always learn something new in these competitions :)
", "rawMarkdown": "Oh I thought onehot or label encoding was always needed with categorical data. I always learn something new in these competitions :)", "votes": null } ], "comments": [ { "id": 2750178, "author_name": "ravi20076", "author_url": "", "post_date": "04/13/2024 13:51:40", "content": "You should look at the number of categories first and then decide if one-hot is needed. If you have too many categories, then one-hot will do more harm than good @kawaiicoderuwu
", "votes": null, "replies": [] }, { "id": 2750374, "author_name": "andreynesterov", "author_url": "", "post_date": "04/13/2024 16:12:45", "content": "I tried one-hot encoding, but without significant gain (about +0.001). Moreover, many popular packages support preprocessing of categorical data, e.g. \"LightGBM offers good accuracy with integer-coded categorical features\", AutoML libs also provide processing of categories using label encoding, one-hot encoding, etc. So I decided to reject the idea of preprocessing categorical data using one-hot encoding (at least for now).
", "votes": null, "replies": [ { "id": 2750991, "author_name": "kawaiicoderuwu", "author_url": "", "post_date": "04/14/2024 00:41:10", "content": "Oh I thought onehot or label encoding was always needed with categorical data. I always learn something new in these competitions :)
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2750124": "Has anyone successfully implemented one hot encoding, because many popular notebooks don't use it and it takes more than 22 GB of data for an 8 GB dataset that I am fitting into it. \nIt overflows on memory each time and remainder=passthrough is the main cause, it's fine if I do drop, but I need the other data. I don't understand why an 8GB dataset uses that much memory.", "2750178": "You should look at the number of categories first and then decide if one-hot is needed. If you have too many categories, then one-hot will do more harm than good @kawaiicoderuwu", "2750374": "I tried one-hot encoding, but without significant gain (about +0.001). Moreover, many popular packages support preprocessing of categorical data, e.g. \"LightGBM offers good accuracy with integer-coded categorical features\", AutoML libs also provide processing of categories using label encoding, one-hot encoding, etc. So I decided to reject the idea of preprocessing categorical data using one-hot encoding (at least for now).", "2750991": "Oh I thought onehot or label encoding was always needed with categorical data. I always learn something new in these competitions :)" }, "source": "meta" }