After looking into the data, the binds column is highly imbalanced:
\n0: 293656924 samples
\n1: 1589906 samples
\nconstituting to ~ 99.46% and ~ 0.53850062% of the total samples respectively.
\nI am opening this thread to learn how everybody is tackling data imbalance at this level. All the suggestions are welcome!
After looking into the data, the binds column is highly imbalanced:
\n0: 293656924 samples
\n1: 1589906 samples
\nconstituting to ~ 99.46% and ~ 0.53850062% of the total samples respectively.
\nI am opening this thread to learn how everybody is tackling data imbalance at this level. All the suggestions are welcome!
you should start to do experiments.
\ne.g. just use the simplest xgboot + fingerprint or chemberta (3-layer trasnformer),
\ntry with different pos to neg ratio in sampling your batch.
maybe there is no difference? who's know?
\nin datascience, decision is based on data.
\nno experiments results, no plan
example of experiments:
\n2.https://www.tensorflow.org/tutorials/structured_data/imbalanced_data
", "rawMarkdown": "you should start to do experiments.\ne.g. just use the simplest xgboot + fingerprint or chemberta (3-layer trasnformer),\ntry with different pos to neg ratio in sampling your batch.\n\nmaybe there is no difference? who's know?\n\n---\nin datascience, decision is based on data.\nno experiments results, no plan\n\nexample of experiments:\n1. Tuning gradient boosting for imbalanced bioassay modelling with custom loss functions\nhttps://github.com/dahvida/gradient_boosting_CLF\n\n2.https://www.tensorflow.org/tutorials/structured_data/imbalanced_data", "votes": 6, "replies": [ { "id": 2791117, "postDate": "2024-05-03T13:38:48.903Z", "content": "Oh! Captain！My Captain~😙
", "rawMarkdown": "Oh! Captain！My Captain~😙" } ] }, { "id": 2863562, "postDate": "2024-06-09T14:00:18.153Z", "content": "Trying to deal with the label imbalance by using SMOTE. I'm reading in the training data as a pyspark dataframe, but I can't find a way to apply SMOTE directly on it. It's also too big to convert to a pandas dataframe (my searches are telling me to convert it to a pandas dataframe to apply smote), is there a way to either:
\n1) apply SMOTE on the full dataset?
\n2) a good way to break up the dataset into batches and perform oversampling on each batch?
For me, I tried using SMOTE on a chunk of dataset and it's giving me OOM error. Still looking for the way to deal with it.
", "rawMarkdown": "For me, I tried using SMOTE on a chunk of dataset and it's giving me OOM error. Still looking for the way to deal with it. ", "votes": 1, "replies": [ { "id": 2893689, "postDate": "2024-06-28T02:31:35.137Z", "content": "In regards to this, I split the dataset into binds == 0 and binds == 1, and then I took binds == 1 set and randomly sampled it to a size that was manageable, and then I used SMOTE on that set and saved it as a data shard. I repeated this by continuing to take different samples from binds == 1 and used SMOTE and saved it as different shards.
", "rawMarkdown": "In regards to this, I split the dataset into binds == 0 and binds == 1, and then I took binds == 1 set and randomly sampled it to a size that was manageable, and then I used SMOTE on that set and saved it as a data shard. I repeated this by continuing to take different samples from binds == 1 and used SMOTE and saved it as different shards." } ] } ] }, { "id": 2786172, "postDate": "2024-05-01T07:42:34.390Z", "content": "Take a look at the tutorial notebook. They sample a 1:1 ratio (with an upper limit of 30k samples for each to reduce training times). But it will, most likely, be a well-kept secret on how to sample it best. At least at the start of the competition.
", "rawMarkdown": "Take a look at the [tutorial notebook](https://www.kaggle.com/code/andrewdblevins/leash-tutorial-ecfps-and-random-forest). They sample a 1:1 ratio (with an upper limit of 30k samples for each to reduce training times). But it will, most likely, be a well-kept secret on how to sample it best. At least at the start of the competition.", "votes": 2 }, { "id": 2780567, "postDate": "2024-04-28T09:56:17.293Z", "content": "Data imbalance needs to be addressed to ensure that the model has even data distribution to learn from and use. SMOTE is one of the techniques to address data imbalance
", "rawMarkdown": "Data imbalance needs to be addressed to ensure that the model has even data distribution to learn from and use. SMOTE is one of the techniques to address data imbalance" }, { "id": 2780390, "postDate": "2024-04-28T07:01:20.500Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2780394, "author_name": "hengck23", "author_url": "", "post_date": "2024-04-28T07:08:37.057000", "content": "you should start to do experiments.
\ne.g. just use the simplest xgboot + fingerprint or chemberta (3-layer trasnformer),
\ntry with different pos to neg ratio in sampling your batch.
maybe there is no difference? who's know?
\nin datascience, decision is based on data.
\nno experiments results, no plan
example of experiments:
\n2.https://www.tensorflow.org/tutorials/structured_data/imbalanced_data
", "votes": 6, "replies": [ { "id": 2791117, "author_name": "cheese", "author_url": "", "post_date": "2024-05-03T13:38:48.903000", "content": "Oh! Captain！My Captain~😙
", "votes": 0, "replies": [] } ] }, { "id": 2863562, "author_name": "MARVINMENG", "author_url": "", "post_date": "2024-06-09T14:00:18.153000", "content": "Trying to deal with the label imbalance by using SMOTE. I'm reading in the training data as a pyspark dataframe, but I can't find a way to apply SMOTE directly on it. It's also too big to convert to a pandas dataframe (my searches are telling me to convert it to a pandas dataframe to apply smote), is there a way to either:
\n1) apply SMOTE on the full dataset?
\n2) a good way to break up the dataset into batches and perform oversampling on each batch?
For me, I tried using SMOTE on a chunk of dataset and it's giving me OOM error. Still looking for the way to deal with it.
", "votes": 1, "replies": [ { "id": 2893689, "author_name": "MARVINMENG", "author_url": "", "post_date": "2024-06-28T02:31:35.137000", "content": "In regards to this, I split the dataset into binds == 0 and binds == 1, and then I took binds == 1 set and randomly sampled it to a size that was manageable, and then I used SMOTE on that set and saved it as a data shard. I repeated this by continuing to take different samples from binds == 1 and used SMOTE and saved it as different shards.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2786172, "author_name": "Ariel Ebersberger", "author_url": "", "post_date": "2024-05-01T07:42:34.390000", "content": "Take a look at the tutorial notebook. They sample a 1:1 ratio (with an upper limit of 30k samples for each to reduce training times). But it will, most likely, be a well-kept secret on how to sample it best. At least at the start of the competition.
", "votes": 2, "replies": [] }, { "id": 2780567, "author_name": "Soumitri Kadambi", "author_url": "", "post_date": "2024-04-28T09:56:17.293000", "content": "Data imbalance needs to be addressed to ensure that the model has even data distribution to learn from and use. SMOTE is one of the techniques to address data imbalance
", "votes": 0, "replies": [] }, { "id": 2780390, "author_name": "", "author_url": "", "post_date": "2024-04-28T07:01:20.500000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2780281": "After looking into the data, the binds column is highly imbalanced:\n**0:** 293656924 samples\n**1:** 1589906 samples\nconstituting to ~ 99.46% and ~ 0.53850062% of the total samples respectively. \nI am opening this thread to learn how everybody is tackling data imbalance at this level. All the suggestions are welcome! \n\n\n\n", "2780394": "you should start to do experiments.\ne.g. just use the simplest xgboot + fingerprint or chemberta (3-layer trasnformer),\ntry with different pos to neg ratio in sampling your batch.\n\nmaybe there is no difference? who's know?\n\n---\nin datascience, decision is based on data.\nno experiments results, no plan\n\nexample of experiments:\n1. Tuning gradient boosting for imbalanced bioassay modelling with custom loss functions\nhttps://github.com/dahvida/gradient_boosting_CLF\n\n2.https://www.tensorflow.org/tutorials/structured_data/imbalanced_data", "2863562": "Trying to deal with the label imbalance by using SMOTE. I'm reading in the training data as a pyspark dataframe, but I can't find a way to apply SMOTE directly on it. It's also too big to convert to a pandas dataframe (my searches are telling me to convert it to a pandas dataframe to apply smote), is there a way to either:\n1) apply SMOTE on the full dataset?\n2) a good way to break up the dataset into batches and perform oversampling on each batch?", "2786172": "Take a look at the [tutorial notebook](https://www.kaggle.com/code/andrewdblevins/leash-tutorial-ecfps-and-random-forest). They sample a 1:1 ratio (with an upper limit of 30k samples for each to reduce training times). But it will, most likely, be a well-kept secret on how to sample it best. At least at the start of the competition.", "2780567": "Data imbalance needs to be addressed to ensure that the model has even data distribution to learn from and use. SMOTE is one of the techniques to address data imbalance", "2780390": "" } }