I'm new to this competition, and I'm confused by the scores on the leaderboard. It seems like most people are using ensemble models, and I wanted to ask what everyone's single models (including tree models or neural networks) are scoring. For me, my tree model is currently at 0.612.
", "messages": [ { "id": "2519376", "postDate": "11/10/2023 00:51:55", "content": "I'm new to this competition, and I'm confused by the scores on the leaderboard. It seems like most people are using ensemble models, and I wanted to ask what everyone's single models (including tree models or neural networks) are scoring. For me, my tree model is currently at 0.612.
", "rawMarkdown": "I'm new to this competition, and I'm confused by the scores on the leaderboard. It seems like most people are using ensemble models, and I wanted to ask what everyone's single models (including tree models or neural networks) are scoring. For me, my tree model is currently at 0.612\u0000.", "votes": null }, { "id": "2519416", "postDate": "11/10/2023 02:42:13", "content": "0.568 for a tensorflow nn
\n0.602 with catboost
\n0.602 with Pyboost
updated:
\n0.567 for tensorflow nn with lots of augmentation and 4 days training time.
0.600 for a tensorflow nn
", "rawMarkdown": "0.600 for a tensorflow nn", "votes": null }, { "id": "2526010", "postDate": "11/15/2023 14:35:13", "content": "Hi @phoenixzero77 ! In this Playground type of competition, margins are very tight in the Leaderboard, so usually the top scoring solutions are made by ensembles. Some people tweak the ensemble mix and a marginally better score. However, as the Public Leaderboard is calculated with c. 39% of the data, when final scores are computed you may see differences. In the last playground, I went from 43 to 21 and many people went down. Hope it helps. Have fun!
", "rawMarkdown": "Hi @phoenixzero77 ! In this Playground type of competition, margins are very tight in the Leaderboard, so usually the top scoring solutions are made by ensembles. Some people tweak the ensemble mix and a marginally better score. However, as the Public Leaderboard is calculated with c. 39% of the data, when final scores are computed you may see differences. In the last playground, I went from 43 to 21 and many people went down. Hope it helps. Have fun!", "votes": null }, { "id": "2531756", "postDate": "11/20/2023 13:55:04", "content": "Thats pretty amazing! my nn cant break 0.59 and val los diverges after that. Have tried every regularization/dropout technique I could think of.
\nDid you face the same problem?
Think I faced similar issues, regularization and dropout along with blends of many tensorflow models all liked the .59x range. Current model being used is big at 97,517,459 parameters. After one hot of my features I have 210 of them.
\nIf memory serves there are only 34 data rows in training for the cell types that are in test. That's a pretty crappy number. The multiple shared ensembles suggested to me that the mean values for these 34 are different from the mean of the predicted test. I got a 0.001 LB improvement two weeks ago by a simple -0.02775 adjustment to every row/column of one of my model submissions.
\nThe literature all suggests that this single cell method is pretty noisy. Batch effects, donor effects, etc all present and happy to make you confused.
\nUsing a mix of augmentation methods I have taken the 610 rows up to 29, 878 rows. The 0.568 result was about 1/3 of that size augmentation. The model training with 29K rows and 210 features has been running for 2.5 days on dual GPU local machine, hopefully training will be done in the next day - if you see me make a big jump in the LB in the next day or two than heavy augmentation caused the leap.
", "rawMarkdown": "Think I faced similar issues, regularization and dropout along with blends of many tensorflow models all liked the .59x range. Current model being used is big at 97,517,459 parameters. After one hot of my features I have 210 of them.\n\nIf memory serves there are only 34 data rows in training for the cell types that are in test. That's a pretty crappy number. The multiple shared ensembles suggested to me that the mean values for these 34 are different from the mean of the predicted test. I got a 0.001 LB improvement two weeks ago by a simple -0.02775 adjustment to every row/column of one of my model submissions.\n\nThe literature all suggests that this single cell method is pretty noisy. Batch effects, donor effects, etc all present and happy to make you confused.\n\nUsing a mix of augmentation methods I have taken the 610 rows up to 29, 878 rows. The 0.568 result was about 1/3 of that size augmentation. The model training with 29K rows and 210 features has been running for 2.5 days on dual GPU local machine, hopefully training will be done in the next day - if you see me make a big jump in the LB in the next day or two than heavy augmentation caused the leap.", "votes": null }, { "id": "2532722", "postDate": "11/21/2023 09:40:16", "content": "May I ask what kind of augmentation you used? By the way, I recently achieved 0.58 score by nn.
", "rawMarkdown": "May I ask what kind of augmentation you used? By the way, I recently achieved 0.58 score by nn.", "votes": null }, { "id": "2532953", "postDate": "11/21/2023 13:24:43", "content": "Thanks a ton ! Yupp instead of stacking ensembles this feature engineering to be the correct approach
\nWhat I've observed is that for different lables the correlation for features is extremely varied and a lot of(different)features have 0 correlation for different lables so I've been trying to get better features
\nYour approach of augmentation and adding tons of data should also solve the same issue
\nThank you, will try augmentation
Four different methods to augment.
\nIt does look like I might have at least 24 more hours of training time - so around 4 days total is pretty expensive - sure hoping it works :)
", "rawMarkdown": "Four different methods to augment.\n1. Pretty standard - use some good LB submissions. This one's a bit scary - it does improve LB score but IMO many of the good scoring ensembles are going to drop to the bottom of the barrel with private test results.\n2. Add to the mean - i do positive/negative \n df0 = de_train\n columns_to_update = df0.columns[5:] # Exclude the string columns\n df0[columns_to_update] += -0.0280\n df1 = de_train\n columns_to_update = df1.columns[5:] # Exclude the string columns\n df1[columns_to_update] += +0.0280\n3. Add random noise - this mostly aimed at expression values around 0, so my upper limit for the random value is pretty small.\n4. Add random noise based on the mean for the column. As a excellent general rule for most labels the standard deviation increases with larger values. The upper limit for the random value is a fraction of the mean.\n\nIt does look like I might have at least 24 more hours of training time - so around 4 days total is pretty expensive - sure hoping it works :)", "votes": null }, { "id": "2533316", "postDate": "11/21/2023 19:15:06", "content": "Thanks a ton! yupp agreed using lb submissions should lead to overfitting.
\nIm going to try and use LINCS data for augmentation https://www.kaggle.com/code/laurasisson/leveraging-lincs-for-dataset-augmentation. A lot of work is required to make it compatible but is a good avenue.
\nNot sure if it can be done in the time.
Good luck to you, hope your full dataset works !
\nP.S. If you can still train, adding SMILE features gives a good quick improvement.
0.589 tensorflow nn
", "rawMarkdown": "0.589 tensorflow nn", "votes": null }, { "id": "2534845", "postDate": "11/23/2023 00:07:47", "content": "0.583 TF NN Model
", "rawMarkdown": "0.583 TF NN Model", "votes": null }, { "id": "2534864", "postDate": "11/23/2023 00:46:55", "content": "Hi,@jainam213.What do you mean by adding SMILE features gives a good quick improvement.
", "rawMarkdown": "Hi,@jainam213.What do you mean by adding SMILE features gives a good quick improvement.", "votes": null }, { "id": "2535510", "postDate": "11/23/2023 11:45:17", "content": "0.557 for single nn
", "rawMarkdown": "0.557 for single nn", "votes": null }, { "id": "2537076", "postDate": "11/24/2023 18:20:33", "content": "Hey!
\nFeautre engineering and adding smile as features as done in
\nhttps://www.kaggle.com/code/mehrankazeminia/3-op2-feature-augment-fragments-of-smiles
\nimproved the model a bit :)
\nIs your score due a single model or ensemble btw? not sure if it improves ensembles
Wow! that's insanely awesome
\ndidn't think any1 would get so good scores with a single model
ensembles,and my best solo model scored 0.557 which used pseudo label(from some submissions).Currently i am not sure will it lead to overfitting…
", "rawMarkdown": "ensembles,and my best solo model scored 0.557 which used pseudo label(from some submissions).Currently i am not sure will it lead to overfitting...", "votes": null }, { "id": "2537325", "postDate": "11/25/2023 04:08:59", "content": "Thanks!
\nYou should split test/val for cv before adding pseudo lables. Since you are using them lb will naturally increase but if the cv is consistent you are probably fine.( remember dont split the dataset after adding the pseudo- labels and only use them for train, otherwise cv will also be misleading)
\nif cv dosent follow you should be overfitting
Cool,i'll check it
", "rawMarkdown": "Cool,i'll check it", "votes": null }, { "id": "2537737", "postDate": "11/25/2023 12:51:15", "content": "So, i did some quick tests using pseudo labels: Test
\nMean Absolute Error (MAE): 0.8498710950373457 Mean Rowwise Root Mean Squared Error (MRRMSE): 1.3408820786496103
Full dataset CV
\nnot doing
\n\nYou should split test/val for cv before adding pseudo lables
\n
Mean Absolute Error (MAE): 0.47567107134329567 Mean Row wise Root Mean Squared Error (MRRMSE): 0.7658528435466305
\nso yeah, pseudo labels should be leading to overfitting… Did you observe similar trends? hmmm cant figure out how to this prevent overfitting/ improving in some other way without overfitting
i dont know which method you have implemented and how you add pseudo labels…
\nsame for me that after adding pseudo lables local cv(MAE and MRRMSE) truly fall,i think it could be overfitting or,maybe the testset(private and public) has lower variance than trainset since normally local cv is higher than lb.I guess perhaps these pseudo labels are easier for model to fit so cv drops.However,i dont know which one is true.
\nAs to aviod overfitting,i try to train more different models(like mlp,transformer,gnn…),use weight decay,K-fold,set low epoches e.t.c.Still not confident about them…
and emmm a trick,sometimes i would exclude some rows,like id 0 and 2,so every time after training i compare the prediction and,you know,those public high-scored submission in terms of certain id…hope it will help you(laugh)
", "rawMarkdown": "and emmm a trick,sometimes i would exclude some rows,like id 0 and 2,so every time after training i compare the prediction and,you know,those public high-scored submission in terms of certain id...hope it will help you(laugh)", "votes": null }, { "id": "2537994", "postDate": "11/25/2023 17:02:50", "content": "Ahh thats great idea! thanks will definately try it out
\n\n\nAs to aviod overfitting,i try to train more different models(like mlp,transformer,gnn…),use weight decay,K-fold,set low epoches e.t.c.Still not confident about them…
\n
This/feature augmentation seems to be the way, i haven't been able to get more than 0.57 from diffrent types of features but i still haven't given up on getting more out of features(SMILES).. This completely avoids overfitting since were just using a single model k-fold.
\nI'll try and quickly train diffrent models and ensemble noww😅
\n\nI think it could be overfitting or,maybe the testset(private and public) has lower variance than trainset since normally local cv is higher than lb.
\n
Whatever the case idts its the worth the risk and its better to completely avoid pseudo labels for the final submissions
\nFor everything, Im currently using a simple nn with dense layers, batchnorm and dropout btw
", "rawMarkdown": "Ahh thats great idea! thanks will definately try it out\n\n>As to aviod overfitting,i try to train more different models(like mlp,transformer,gnn…),use weight decay,K-fold,set low epoches e.t.c.Still not confident about them…\n\nThis/feature augmentation seems to be the way, i haven't been able to get more than 0.57 from diffrent types of features but i still haven't given up on getting more out of features(SMILES).. This completely avoids overfitting since were just using a single model k-fold.\nI'll try and quickly train diffrent models and ensemble noww😅\n\n>I think it could be overfitting or,maybe the testset(private and public) has lower variance than trainset since normally local cv is higher than lb.\n\nWhatever the case idts its the worth the risk and its better to completely avoid pseudo labels for the final submissions\n\nFor everything, Im currently using a simple nn with dense layers, batchnorm and dropout btw", "votes": null }, { "id": "2543291", "postDate": "11/30/2023 00:29:32", "content": "0.579 TF NN Model - 30 minutes training
", "rawMarkdown": "0.579 TF NN Model - 30 minutes training", "votes": null }, { "id": "2544043", "postDate": "11/30/2023 15:05:35", "content": "What do you use for training? Kaggle or another cloud platform? Thank you sharing your results :)
", "rawMarkdown": "What do you use for training? Kaggle or another cloud platform? Thank you sharing your results :)", "votes": null }, { "id": "2544066", "postDate": "11/30/2023 15:19:50", "content": "I'm using Kaggle only
", "rawMarkdown": "I'm using Kaggle only", "votes": null } ], "comments": [ { "id": 2519416, "author_name": "pcjimmmy", "author_url": "", "post_date": "11/10/2023 02:42:13", "content": "0.568 for a tensorflow nn
\n0.602 with catboost
\n0.602 with Pyboost
updated:
\n0.567 for tensorflow nn with lots of augmentation and 4 days training time.
Thats pretty amazing! my nn cant break 0.59 and val los diverges after that. Have tried every regularization/dropout technique I could think of.
\nDid you face the same problem?
Think I faced similar issues, regularization and dropout along with blends of many tensorflow models all liked the .59x range. Current model being used is big at 97,517,459 parameters. After one hot of my features I have 210 of them.
\nIf memory serves there are only 34 data rows in training for the cell types that are in test. That's a pretty crappy number. The multiple shared ensembles suggested to me that the mean values for these 34 are different from the mean of the predicted test. I got a 0.001 LB improvement two weeks ago by a simple -0.02775 adjustment to every row/column of one of my model submissions.
\nThe literature all suggests that this single cell method is pretty noisy. Batch effects, donor effects, etc all present and happy to make you confused.
\nUsing a mix of augmentation methods I have taken the 610 rows up to 29, 878 rows. The 0.568 result was about 1/3 of that size augmentation. The model training with 29K rows and 210 features has been running for 2.5 days on dual GPU local machine, hopefully training will be done in the next day - if you see me make a big jump in the LB in the next day or two than heavy augmentation caused the leap.
", "votes": null, "replies": [ { "id": 2532722, "author_name": "phoenixzero77", "author_url": "", "post_date": "11/21/2023 09:40:16", "content": "May I ask what kind of augmentation you used? By the way, I recently achieved 0.58 score by nn.
", "votes": null, "replies": [ { "id": 2533015, "author_name": "pcjimmmy", "author_url": "", "post_date": "11/21/2023 14:22:09", "content": "Four different methods to augment.
\nIt does look like I might have at least 24 more hours of training time - so around 4 days total is pretty expensive - sure hoping it works :)
", "votes": null, "replies": [ { "id": 2533316, "author_name": "jainam213", "author_url": "", "post_date": "11/21/2023 19:15:06", "content": "Thanks a ton! yupp agreed using lb submissions should lead to overfitting.
\nIm going to try and use LINCS data for augmentation https://www.kaggle.com/code/laurasisson/leveraging-lincs-for-dataset-augmentation. A lot of work is required to make it compatible but is a good avenue.
\nNot sure if it can be done in the time.
Good luck to you, hope your full dataset works !
\nP.S. If you can still train, adding SMILE features gives a good quick improvement.
Hi,@jainam213.What do you mean by adding SMILE features gives a good quick improvement.
", "votes": null, "replies": [ { "id": 2537076, "author_name": "jainam213", "author_url": "", "post_date": "11/24/2023 18:20:33", "content": "Hey!
\nFeautre engineering and adding smile as features as done in
\nhttps://www.kaggle.com/code/mehrankazeminia/3-op2-feature-augment-fragments-of-smiles
\nimproved the model a bit :)
\nIs your score due a single model or ensemble btw? not sure if it improves ensembles
ensembles,and my best solo model scored 0.557 which used pseudo label(from some submissions).Currently i am not sure will it lead to overfitting…
", "votes": null, "replies": [ { "id": 2537325, "author_name": "jainam213", "author_url": "", "post_date": "11/25/2023 04:08:59", "content": "Thanks!
\nYou should split test/val for cv before adding pseudo lables. Since you are using them lb will naturally increase but if the cv is consistent you are probably fine.( remember dont split the dataset after adding the pseudo- labels and only use them for train, otherwise cv will also be misleading)
\nif cv dosent follow you should be overfitting
Cool,i'll check it
", "votes": null, "replies": [ { "id": 2537737, "author_name": "jainam213", "author_url": "", "post_date": "11/25/2023 12:51:15", "content": "So, i did some quick tests using pseudo labels: Test
\nMean Absolute Error (MAE): 0.8498710950373457 Mean Rowwise Root Mean Squared Error (MRRMSE): 1.3408820786496103
Full dataset CV
\nnot doing
\n\nYou should split test/val for cv before adding pseudo lables
\n
Mean Absolute Error (MAE): 0.47567107134329567 Mean Row wise Root Mean Squared Error (MRRMSE): 0.7658528435466305
\nso yeah, pseudo labels should be leading to overfitting… Did you observe similar trends? hmmm cant figure out how to this prevent overfitting/ improving in some other way without overfitting
i dont know which method you have implemented and how you add pseudo labels…
\nsame for me that after adding pseudo lables local cv(MAE and MRRMSE) truly fall,i think it could be overfitting or,maybe the testset(private and public) has lower variance than trainset since normally local cv is higher than lb.I guess perhaps these pseudo labels are easier for model to fit so cv drops.However,i dont know which one is true.
\nAs to aviod overfitting,i try to train more different models(like mlp,transformer,gnn…),use weight decay,K-fold,set low epoches e.t.c.Still not confident about them…
and emmm a trick,sometimes i would exclude some rows,like id 0 and 2,so every time after training i compare the prediction and,you know,those public high-scored submission in terms of certain id…hope it will help you(laugh)
", "votes": null, "replies": [ { "id": 2537994, "author_name": "jainam213", "author_url": "", "post_date": "11/25/2023 17:02:50", "content": "Ahh thats great idea! thanks will definately try it out
\n\n\nAs to aviod overfitting,i try to train more different models(like mlp,transformer,gnn…),use weight decay,K-fold,set low epoches e.t.c.Still not confident about them…
\n
This/feature augmentation seems to be the way, i haven't been able to get more than 0.57 from diffrent types of features but i still haven't given up on getting more out of features(SMILES).. This completely avoids overfitting since were just using a single model k-fold.
\nI'll try and quickly train diffrent models and ensemble noww😅
\n\nI think it could be overfitting or,maybe the testset(private and public) has lower variance than trainset since normally local cv is higher than lb.
\n
Whatever the case idts its the worth the risk and its better to completely avoid pseudo labels for the final submissions
\nFor everything, Im currently using a simple nn with dense layers, batchnorm and dropout btw
", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ] } ] } ] } ] }, { "id": 2544043, "author_name": "wguesdon", "author_url": "", "post_date": "11/30/2023 15:05:35", "content": "What do you use for training? Kaggle or another cloud platform? Thank you sharing your results :)
", "votes": null, "replies": [ { "id": 2544066, "author_name": "jainam213", "author_url": "", "post_date": "11/30/2023 15:19:50", "content": "I'm using Kaggle only
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2532953, "author_name": "jainam213", "author_url": "", "post_date": "11/21/2023 13:24:43", "content": "Thanks a ton ! Yupp instead of stacking ensembles this feature engineering to be the correct approach
\nWhat I've observed is that for different lables the correlation for features is extremely varied and a lot of(different)features have 0 correlation for different lables so I've been trying to get better features
\nYour approach of augmentation and adding tons of data should also solve the same issue
\nThank you, will try augmentation
0.600 for a tensorflow nn
", "votes": null, "replies": [] }, { "id": 2526010, "author_name": "octaviograu", "author_url": "", "post_date": "11/15/2023 14:35:13", "content": "Hi @phoenixzero77 ! In this Playground type of competition, margins are very tight in the Leaderboard, so usually the top scoring solutions are made by ensembles. Some people tweak the ensemble mix and a marginally better score. However, as the Public Leaderboard is calculated with c. 39% of the data, when final scores are computed you may see differences. In the last playground, I went from 43 to 21 and many people went down. Hope it helps. Have fun!
", "votes": null, "replies": [] }, { "id": 2533626, "author_name": "jjleesunny", "author_url": "", "post_date": "11/22/2023 05:07:46", "content": "0.589 tensorflow nn
", "votes": null, "replies": [ { "id": 2534845, "author_name": "jjleesunny", "author_url": "", "post_date": "11/23/2023 00:07:47", "content": "0.583 TF NN Model
", "votes": null, "replies": [ { "id": 2543291, "author_name": "jjleesunny", "author_url": "", "post_date": "11/30/2023 00:29:32", "content": "0.579 TF NN Model - 30 minutes training
", "votes": null, "replies": [] } ] } ] }, { "id": 2535510, "author_name": "jankowalski2000", "author_url": "", "post_date": "11/23/2023 11:45:17", "content": "0.557 for single nn
", "votes": null, "replies": [ { "id": 2537109, "author_name": "jainam213", "author_url": "", "post_date": "11/24/2023 18:49:08", "content": "Wow! that's insanely awesome
\ndidn't think any1 would get so good scores with a single model