Hi experts, I am new here and participating in this competition for learning only. I am stuck at 0.795 and was wondering if experts on this could give some direction on what more can be done to move beyond? Specifically would love to understand how leaders in this competition are thinking about: 1) Feature Engineering (not so obvious ones), 2) Hyper-parameter tuning for the models you are using.
", "messages": [ { "id": "1823411", "postDate": "06/17/2022 11:15:38", "content": "Hi experts, I am new here and participating in this competition for learning only. I am stuck at 0.795 and was wondering if experts on this could give some direction on what more can be done to move beyond? Specifically would love to understand how leaders in this competition are thinking about: 1) Feature Engineering (not so obvious ones), 2) Hyper-parameter tuning for the models you are using.
", "rawMarkdown": "Hi experts, I am new here and participating in this competition for learning only. I am stuck at 0.795 and was wondering if experts on this could give some direction on what more can be done to move beyond? Specifically would love to understand how leaders in this competition are thinking about: 1) Feature Engineering (not so obvious ones), 2) Hyper-parameter tuning for the models you are using.", "votes": null }, { "id": "1823473", "postDate": "06/17/2022 12:11:14", "content": "Easiest way is just ensemble, but to break even further you'll need some good FE or more
", "rawMarkdown": "Easiest way is just ensemble, but to break even further you'll need some good FE or more", "votes": null }, { "id": "1823529", "postDate": "06/17/2022 13:10:01", "content": "Model ensembling works very well. Try different model types (xgboost, catboost, lightgbm). For each model can experiment with changing feature sets (i.e. use mean features in xgboost but dont include that in catboost, etc.).
\n0.798 is achievable using an ensemble of a fine tuned public kernels. Speaking from experience here :)
\nmy best solo model is 0.796 on LB. It is really hard to push that third digit here with a single model
", "rawMarkdown": "Model ensembling works very well. Try different model types (xgboost, catboost, lightgbm). For each model can experiment with changing feature sets (i.e. use mean features in xgboost but dont include that in catboost, etc.). \n\n0.798 is achievable using an ensemble of a fine tuned public kernels. Speaking from experience here :)\n\nmy best solo model is 0.796 on LB. It is really hard to push that third digit here with a single model", "votes": null }, { "id": "1823608", "postDate": "06/17/2022 14:15:25", "content": "That is a really great answer, wow! 🙂 Thank you for sharing this!
\nI have not started to build an ensemble, more of trying out a couple of techniques. But one thing I noticed is that feature selection is next to impossible 🙂 I wonder if you had a similar experience?
\nI played with null hypotheses, various types of feature importance, forward feature selection, but it is all proving to be quite useless 🙂 I guess one thing is the data itself and the other the fact that these gradient boosting techniques are really good in dealing with noise.
\nAnyhow, really appreciate everything that you have shared so far, really good food for thought 🙂
\nBest of luck in the competition @raddar!
", "rawMarkdown": "That is a really great answer, wow! 🙂 Thank you for sharing this!\n\nI have not started to build an ensemble, more of trying out a couple of techniques. But one thing I noticed is that feature selection is next to impossible 🙂 I wonder if you had a similar experience?\n\nI played with null hypotheses, various types of feature importance, forward feature selection, but it is all proving to be quite useless 🙂 I guess one thing is the data itself and the other the fact that these gradient boosting techniques are really good in dealing with noise.\n\nAnyhow, really appreciate everything that you have shared so far, really good food for thought 🙂\n\nBest of luck in the competition @raddar!", "votes": null }, { "id": "1823642", "postDate": "06/17/2022 14:43:47", "content": "Thanks for the kind words.
\nMy experience is the same - feature selection not too useful and feature engineering extremely hard - but not impossible.
\nI think to reach 0.799+ you need to somehow blend in one or two neural networks in the ensemble. Struggling a bit so far in that direction :D
", "rawMarkdown": "Thanks for the kind words.\n\nMy experience is the same - feature selection not too useful and feature engineering extremely hard - but not impossible. \n\nI think to reach 0.799+ you need to somehow blend in one or two neural networks in the ensemble. Struggling a bit so far in that direction :D", "votes": null }, { "id": "1823711", "postDate": "06/17/2022 15:54:43", "content": "I think ensembling will be enough
", "rawMarkdown": "I think ensembling will be enough", "votes": null }, { "id": "1823907", "postDate": "06/17/2022 18:24:03", "content": "You can definitely get 0.795+ with the NNs :) Though they require a lot of preprocessing etc.
", "rawMarkdown": "You can definitely get 0.795+ with the NNs :) Though they require a lot of preprocessing etc.", "votes": null }, { "id": "1824456", "postDate": "06/18/2022 10:54:45", "content": "sometimes, changing seeds and averaging the result can improve the score. You can take a look at this paper https://arxiv.org/pdf/2109.08203.pdf and also this discussion
", "rawMarkdown": "sometimes, changing seeds and averaging the result can improve the score. You can take a look at this paper [https://arxiv.org/pdf/2109.08203.pdf](https://arxiv.org/pdf/2109.08203.pdf) and also this [discussion](https://www.kaggle.com/c/petfinder-pawpularity-score/discussion/296706)", "votes": null }, { "id": "1825188", "postDate": "06/19/2022 04:35:08", "content": "Ensembling as well as fine search using GridSearch CV and cross validation gives me better results as a beginner
", "rawMarkdown": "Ensembling as well as fine search using GridSearch CV and cross validation gives me better results as a beginner", "votes": null }, { "id": "1828484", "postDate": "06/21/2022 18:24:13", "content": "Dear @bmkhandu
\nIndeed, a bit like the Titanic; it is fairly easy to score 78% but really quite challenging to get 80%.
\nAll the best,
\ncarl
May be ANN with hyper parameter tuning could result in better accuracy as they would go on with fined parameters
", "rawMarkdown": "May be ANN with hyper parameter tuning could result in better accuracy as they would go on with fined parameters", "votes": null } ], "comments": [ { "id": 1823473, "author_name": "julianmukaj", "author_url": "", "post_date": "06/17/2022 12:11:14", "content": "Easiest way is just ensemble, but to break even further you'll need some good FE or more
", "votes": null, "replies": [] }, { "id": 1823529, "author_name": "raddar", "author_url": "", "post_date": "06/17/2022 13:10:01", "content": "Model ensembling works very well. Try different model types (xgboost, catboost, lightgbm). For each model can experiment with changing feature sets (i.e. use mean features in xgboost but dont include that in catboost, etc.).
\n0.798 is achievable using an ensemble of a fine tuned public kernels. Speaking from experience here :)
\nmy best solo model is 0.796 on LB. It is really hard to push that third digit here with a single model
", "votes": null, "replies": [ { "id": 1823608, "author_name": "radek1", "author_url": "", "post_date": "06/17/2022 14:15:25", "content": "That is a really great answer, wow! 🙂 Thank you for sharing this!
\nI have not started to build an ensemble, more of trying out a couple of techniques. But one thing I noticed is that feature selection is next to impossible 🙂 I wonder if you had a similar experience?
\nI played with null hypotheses, various types of feature importance, forward feature selection, but it is all proving to be quite useless 🙂 I guess one thing is the data itself and the other the fact that these gradient boosting techniques are really good in dealing with noise.
\nAnyhow, really appreciate everything that you have shared so far, really good food for thought 🙂
\nBest of luck in the competition @raddar!
", "votes": null, "replies": [] }, { "id": 1823642, "author_name": "raddar", "author_url": "", "post_date": "06/17/2022 14:43:47", "content": "Thanks for the kind words.
\nMy experience is the same - feature selection not too useful and feature engineering extremely hard - but not impossible.
\nI think to reach 0.799+ you need to somehow blend in one or two neural networks in the ensemble. Struggling a bit so far in that direction :D
", "votes": null, "replies": [] }, { "id": 1823907, "author_name": "bacicnikola", "author_url": "", "post_date": "06/17/2022 18:24:03", "content": "You can definitely get 0.795+ with the NNs :) Though they require a lot of preprocessing etc.
", "votes": null, "replies": [] } ] }, { "id": 1823711, "author_name": "kadirgirne", "author_url": "", "post_date": "06/17/2022 15:54:43", "content": "I think ensembling will be enough
", "votes": null, "replies": [] }, { "id": 1824456, "author_name": "naiborhujosua", "author_url": "", "post_date": "06/18/2022 10:54:45", "content": "sometimes, changing seeds and averaging the result can improve the score. You can take a look at this paper https://arxiv.org/pdf/2109.08203.pdf and also this discussion
", "votes": null, "replies": [] }, { "id": 1825188, "author_name": "shivamagarwal29", "author_url": "", "post_date": "06/19/2022 04:35:08", "content": "Ensembling as well as fine search using GridSearch CV and cross validation gives me better results as a beginner
", "votes": null, "replies": [] }, { "id": 1828484, "author_name": "carlmcbrideellis", "author_url": "", "post_date": "06/21/2022 18:24:13", "content": "Dear @bmkhandu
\nIndeed, a bit like the Titanic; it is fairly easy to score 78% but really quite challenging to get 80%.
\nAll the best,
\ncarl
May be ANN with hyper parameter tuning could result in better accuracy as they would go on with fined parameters
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1823411": "Hi experts, I am new here and participating in this competition for learning only. I am stuck at 0.795 and was wondering if experts on this could give some direction on what more can be done to move beyond? Specifically would love to understand how leaders in this competition are thinking about: 1) Feature Engineering (not so obvious ones), 2) Hyper-parameter tuning for the models you are using.", "1823473": "Easiest way is just ensemble, but to break even further you'll need some good FE or more", "1823529": "Model ensembling works very well. Try different model types (xgboost, catboost, lightgbm). For each model can experiment with changing feature sets (i.e. use mean features in xgboost but dont include that in catboost, etc.). \n\n0.798 is achievable using an ensemble of a fine tuned public kernels. Speaking from experience here :)\n\nmy best solo model is 0.796 on LB. It is really hard to push that third digit here with a single model", "1823608": "That is a really great answer, wow! 🙂 Thank you for sharing this!\n\nI have not started to build an ensemble, more of trying out a couple of techniques. But one thing I noticed is that feature selection is next to impossible 🙂 I wonder if you had a similar experience?\n\nI played with null hypotheses, various types of feature importance, forward feature selection, but it is all proving to be quite useless 🙂 I guess one thing is the data itself and the other the fact that these gradient boosting techniques are really good in dealing with noise.\n\nAnyhow, really appreciate everything that you have shared so far, really good food for thought 🙂\n\nBest of luck in the competition @raddar!", "1823642": "Thanks for the kind words.\n\nMy experience is the same - feature selection not too useful and feature engineering extremely hard - but not impossible. \n\nI think to reach 0.799+ you need to somehow blend in one or two neural networks in the ensemble. Struggling a bit so far in that direction :D", "1823711": "I think ensembling will be enough", "1823907": "You can definitely get 0.795+ with the NNs :) Though they require a lot of preprocessing etc.", "1824456": "sometimes, changing seeds and averaging the result can improve the score. You can take a look at this paper [https://arxiv.org/pdf/2109.08203.pdf](https://arxiv.org/pdf/2109.08203.pdf) and also this [discussion](https://www.kaggle.com/c/petfinder-pawpularity-score/discussion/296706)", "1825188": "Ensembling as well as fine search using GridSearch CV and cross validation gives me better results as a beginner", "1828484": "Dear @bmkhandu \n\nIndeed, a bit like the Titanic; [it is fairly easy to score 78%](https://www.kaggle.com/code/carlmcbrideellis/kiss-small-and-simple-titanic-models/) but really quite challenging to get 80%. \n\nAll the best,\ncarl", "1828493": "May be ANN with hyper parameter tuning could result in better accuracy as they would go on with fined parameters" }, "source": "meta" }