As I have seen and tried lot of notebook public on this Competition. I am just trying to conclude what all i can try to improve my scores.
\nAlgorithums- XGB,LGB, Catboost, Tabnet, RNN( GRU/LSTM), Pyboost , AutoMl models
\nFeature Engineering- Use Only given 188 features, ADD mean max min std type features, add A-B type features, add A/B or A*B type features or build some new features around that.
\nHP tuning- Specific to Models.
\nDifferent seed
\nEnsemble multiple models
\nTill now I have tried more than 20-30 iterations of above techniques and getting a score of 0.797 ensembling 2 of my best models. and planning to try more combination of above things.
Hope this will help others in this competition and Please let me know if a have missed any important thing to try to improve my results. and Don't forget to Upvote if you like this discussion.
", "messages": [ { "id": 1836833, "postDate": "2022-06-29T06:12:16.293Z", "content": "As I have seen and tried lot of notebook public on this Competition. I am just trying to conclude what all i can try to improve my scores.
\nAlgorithums- XGB,LGB, Catboost, Tabnet, RNN( GRU/LSTM), Pyboost , AutoMl models
\nFeature Engineering- Use Only given 188 features, ADD mean max min std type features, add A-B type features, add A/B or A*B type features or build some new features around that.
\nHP tuning- Specific to Models.
\nDifferent seed
\nEnsemble multiple models
\nTill now I have tried more than 20-30 iterations of above techniques and getting a score of 0.797 ensembling 2 of my best models. and planning to try more combination of above things.
Hope this will help others in this competition and Please let me know if a have missed any important thing to try to improve my results. and Don't forget to Upvote if you like this discussion.
", "rawMarkdown": "As I have seen and tried lot of notebook public on this Competition. I am just trying to conclude what all i can try to improve my scores.\n**Algorithums**- XGB,LGB, Catboost, Tabnet, RNN( GRU/LSTM), Pyboost , AutoMl models\n**Feature Engineering**- Use Only given 188 features, ADD mean max min std type features, add A-B type features, add A/B or A*B type features or build some new features around that.\n**HP tuning**- Specific to Models.\n**Different seed**\n**Ensemble multiple models** \nTill now I have tried more than 20-30 iterations of above techniques and getting a score of 0.797 ensembling 2 of my best models. and planning to try more combination of above things.\n\nHope this will help others in this competition and Please let me know if a have missed any important thing to try to improve my results. and Don't forget to Upvote if you like this discussion.\n", "votes": 34 }, { "id": 1836937, "postDate": "2022-06-29T08:17:21.763Z", "content": "also include lgbm DART boosting strategy. It is slow but improves the accuracy of your model.
", "rawMarkdown": "also include lgbm DART boosting strategy. It is slow but improves the accuracy of your model.", "votes": 7, "replies": [ { "id": 1836955, "postDate": "2022-06-29T08:31:50.130Z", "content": "Thanks for Idea @mohammadrahmati . But both of my current best models are LGB with dart. That's give a significant positive impact to scores.
", "rawMarkdown": "Thanks for Idea @mohammadrahmati . But both of my current best models are LGB with dart. That's give a significant positive impact to scores. ", "votes": 3 }, { "id": 1837638, "postDate": "2022-06-29T18:44:07.063Z", "content": "@mohammadrahmati If you have a non-DART lgbm model, can you expect an immediate improvement by changing to DART? Or would you also spend a lot of time readjusting the other hyperparameters?
", "rawMarkdown": "@mohammadrahmati If you have a non-DART lgbm model, can you expect an immediate improvement by changing to DART? Or would you also spend a lot of time readjusting the other hyperparameters?", "votes": 2 }, { "id": 1837733, "postDate": "2022-06-29T21:01:14.593Z", "content": "@datahobbit I would say yes. At least it worked for one of my models. I would suggest choosing the default parameters and let it run for a few thousands of rounds.
", "rawMarkdown": "@datahobbit I would say yes. At least it worked for one of my models. I would suggest choosing the default parameters and let it run for a few thousands of rounds. ", "votes": 1 }, { "id": 1837745, "postDate": "2022-06-29T21:09:23.437Z", "content": "OK great thank you. At least with lgbm the slowdown from 'gbdt' is noticeable but not totally awful. I tried it first with xgboost but that switches from GPU to CPU and it became totally impractical to run enough rounds.
", "rawMarkdown": "OK great thank you. At least with lgbm the slowdown from 'gbdt' is noticeable but not totally awful. I tried it first with xgboost but that switches from GPU to CPU and it became totally impractical to run enough rounds. ", "votes": 3 }, { "id": 1837751, "postDate": "2022-06-29T21:29:37.493Z", "content": "My system takes around one hour to finish 15000 rounds per fold on CPU (around 5 hours to finish all folds).
\nyou may find this useful: https://www.kaggle.com/competitions/amex-default-prediction/discussion/332575
\n\nI tried it first with xgboost but that switches from GPU to CPU and it became totally impractical to run enough rounds.
\n
@datahobbit How do you know that it switched to CPU? When i added the parameter 'booster': 'dart' to my XGB model and then ran nvidia-smi, the GPU was being used and the CV improved.
Additionally, i use parameters:
\n 'tree_method':'gpu_hist',\n 'predictor':'gpu_predictor',\n@mohammadrahmati and @cdeotte thanks for the advice. Chris, I don't know for certain it switched to GPU but the speed dropped by close to 100x so I just assumed. I will try the settings you provided - thanks again.
", "rawMarkdown": "@mohammadrahmati and @cdeotte thanks for the advice. Chris, I don't know for certain it switched to GPU but the speed dropped by close to 100x so I just assumed. I will try the settings you provided - thanks again.", "votes": 1 }, { "id": 1837965, "postDate": "2022-06-30T04:38:13.487Z", "content": "It was much slower for me too. Maybe 10x or 20x slower, but i checked and it was using GPU at 85%. I'm not sure why DART is so slow.
", "rawMarkdown": "It was much slower for me too. Maybe 10x or 20x slower, but i checked and it was using GPU at 85%. I'm not sure why DART is so slow.", "votes": 5 }, { "id": 1837981, "postDate": "2022-06-30T05:03:43.293Z", "content": "OK, thanks that's helpful.
", "rawMarkdown": "OK, thanks that's helpful. ", "votes": 1 }, { "id": 1854176, "postDate": "2022-07-13T13:31:21.523Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1839836, "postDate": "2022-07-01T18:06:05.093Z", "content": "I would also add that trying different subsets of features can also be helpful (e.g. removing correlated features, or features that don't have much predictive power, etc.). Additionally, feature engineering is an important part of any machine learning competition, so spending time brainstorming new features to add can also be very beneficial.
", "rawMarkdown": "I would also add that trying different subsets of features can also be helpful (e.g. removing correlated features, or features that don't have much predictive power, etc.). Additionally, feature engineering is an important part of any machine learning competition, so spending time brainstorming new features to add can also be very beneficial. \n\n\n", "votes": 3 }, { "id": 1866531, "postDate": "2022-07-22T14:56:16.070Z", "content": "hey~ what do you think about how much influence to the model s given by the seed, try different is really cost time.
", "rawMarkdown": "hey~ what do you think about how much influence to the model s given by the seed, try different is really cost time. " }, { "id": 1840951, "postDate": "2022-07-02T17:07:14.160Z", "content": "I have got the best results with an ANN model using Keras/TensorFlow. Also I have done oversampling since data is unbalanced.
", "rawMarkdown": "I have got the best results with an ANN model using Keras/TensorFlow. Also I have done oversampling since data is unbalanced." }, { "id": 1837884, "postDate": "2022-06-30T02:59:46.140Z", "content": "Hello, thanks for sharing.
", "rawMarkdown": "Hello, thanks for sharing.\n", "votes": 1 }, { "id": 1840005, "postDate": "2022-07-01T21:03:30.613Z", "content": "Thanks for such a concise explanation.
", "rawMarkdown": "Thanks for such a concise explanation." } ], "comments": [ { "id": 1836937, "author_name": "1110Ra", "author_url": "", "post_date": "2022-06-29T08:17:21.763000", "content": "also include lgbm DART boosting strategy. It is slow but improves the accuracy of your model.
", "votes": 7, "replies": [ { "id": 1836955, "author_name": "Jaswinder Singh", "author_url": "", "post_date": "2022-06-29T08:31:50.130000", "content": "Thanks for Idea @mohammadrahmati . But both of my current best models are LGB with dart. That's give a significant positive impact to scores.
", "votes": 3, "replies": [] }, { "id": 1837638, "author_name": "MichaelP", "author_url": "", "post_date": "2022-06-29T18:44:07.063000", "content": "@mohammadrahmati If you have a non-DART lgbm model, can you expect an immediate improvement by changing to DART? Or would you also spend a lot of time readjusting the other hyperparameters?
", "votes": 2, "replies": [] }, { "id": 1837733, "author_name": "1110Ra", "author_url": "", "post_date": "2022-06-29T21:01:14.593000", "content": "@datahobbit I would say yes. At least it worked for one of my models. I would suggest choosing the default parameters and let it run for a few thousands of rounds.
", "votes": 1, "replies": [] }, { "id": 1837745, "author_name": "MichaelP", "author_url": "", "post_date": "2022-06-29T21:09:23.437000", "content": "OK great thank you. At least with lgbm the slowdown from 'gbdt' is noticeable but not totally awful. I tried it first with xgboost but that switches from GPU to CPU and it became totally impractical to run enough rounds.
", "votes": 3, "replies": [] }, { "id": 1837751, "author_name": "1110Ra", "author_url": "", "post_date": "2022-06-29T21:29:37.493000", "content": "My system takes around one hour to finish 15000 rounds per fold on CPU (around 5 hours to finish all folds).
\nyou may find this useful: https://www.kaggle.com/competitions/amex-default-prediction/discussion/332575
\n\nI tried it first with xgboost but that switches from GPU to CPU and it became totally impractical to run enough rounds.
\n
@datahobbit How do you know that it switched to CPU? When i added the parameter 'booster': 'dart' to my XGB model and then ran nvidia-smi, the GPU was being used and the CV improved.
Additionally, i use parameters:
\n 'tree_method':'gpu_hist',\n 'predictor':'gpu_predictor',\n@mohammadrahmati and @cdeotte thanks for the advice. Chris, I don't know for certain it switched to GPU but the speed dropped by close to 100x so I just assumed. I will try the settings you provided - thanks again.
", "votes": 1, "replies": [] }, { "id": 1837965, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-06-30T04:38:13.487000", "content": "It was much slower for me too. Maybe 10x or 20x slower, but i checked and it was using GPU at 85%. I'm not sure why DART is so slow.
", "votes": 5, "replies": [] }, { "id": 1837981, "author_name": "MichaelP", "author_url": "", "post_date": "2022-06-30T05:03:43.293000", "content": "OK, thanks that's helpful.
", "votes": 1, "replies": [] }, { "id": 1854176, "author_name": "", "author_url": "", "post_date": "2022-07-13T13:31:21.523000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1839836, "author_name": "The Devastator", "author_url": "", "post_date": "2022-07-01T18:06:05.093000", "content": "I would also add that trying different subsets of features can also be helpful (e.g. removing correlated features, or features that don't have much predictive power, etc.). Additionally, feature engineering is an important part of any machine learning competition, so spending time brainstorming new features to add can also be very beneficial.
", "votes": 3, "replies": [] }, { "id": 1866531, "author_name": "xianZ_waikato", "author_url": "", "post_date": "2022-07-22T14:56:16.070000", "content": "hey~ what do you think about how much influence to the model s given by the seed, try different is really cost time.
", "votes": 0, "replies": [] }, { "id": 1840951, "author_name": "Johnny Torres", "author_url": "", "post_date": "2022-07-02T17:07:14.160000", "content": "I have got the best results with an ANN model using Keras/TensorFlow. Also I have done oversampling since data is unbalanced.
", "votes": 0, "replies": [] }, { "id": 1837884, "author_name": "Praise the Lord Jesus", "author_url": "", "post_date": "2022-06-30T02:59:46.140000", "content": "Hello, thanks for sharing.
", "votes": 1, "replies": [] }, { "id": 1840005, "author_name": "Sohail Ahmed", "author_url": "", "post_date": "2022-07-01T21:03:30.613000", "content": "Thanks for such a concise explanation.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1836833": "As I have seen and tried lot of notebook public on this Competition. I am just trying to conclude what all i can try to improve my scores.\n**Algorithums**- XGB,LGB, Catboost, Tabnet, RNN( GRU/LSTM), Pyboost , AutoMl models\n**Feature Engineering**- Use Only given 188 features, ADD mean max min std type features, add A-B type features, add A/B or A*B type features or build some new features around that.\n**HP tuning**- Specific to Models.\n**Different seed**\n**Ensemble multiple models** \nTill now I have tried more than 20-30 iterations of above techniques and getting a score of 0.797 ensembling 2 of my best models. and planning to try more combination of above things.\n\nHope this will help others in this competition and Please let me know if a have missed any important thing to try to improve my results. and Don't forget to Upvote if you like this discussion.\n", "1836937": "also include lgbm DART boosting strategy. It is slow but improves the accuracy of your model.", "1839836": "I would also add that trying different subsets of features can also be helpful (e.g. removing correlated features, or features that don't have much predictive power, etc.). Additionally, feature engineering is an important part of any machine learning competition, so spending time brainstorming new features to add can also be very beneficial. \n\n\n", "1866531": "hey~ what do you think about how much influence to the model s given by the seed, try different is really cost time. ", "1840951": "I have got the best results with an ANN model using Keras/TensorFlow. Also I have done oversampling since data is unbalanced.", "1837884": "Hello, thanks for sharing.\n", "1840005": "Thanks for such a concise explanation." } }