I will not bore everyone with a complete overview of my 35th place solo solution. I can't wait to read over the actual top places solutions, especially Little Boat's who I have learned so much from!! Instead, I will go over how I squeezed the last -0.0011 out of my RMSE score, -0.0001 to -0.0003 at a time. It took a lot of experimenting to find what worked.
\n\nFirst a quick overview of my basic approach. I did a lot of feature engineering ending up with 170 numerical features derived from text properties, image properties, mining of test_active/train_active, price encodings, categorical stats, polynomial features, etc.
\n\nI had a team of GBT models (XGBoost and LightGBM) and neural net models (Keras). My word embedding was created entirely from Avito text. My GBT's included Tfidf and Countv on text, as well as TfidfEmbeddingVectorizer features. My NN's were BiLSTM/Conv1D and Conv1D/Conv1D similar to those in my DonorsChoose.org playground competition 1st place solution.
\n\nAt first I linearly stacked these models using out-of-fold predictions. I was at something like RMSE 0.2178 on the PLB, and searching for how to further reduce my RMSE. What worked:
\n\n1.) -0.0002: I did an DataFrame corr() on all of my numerical features, and pruned out features with correlations above 0.92.
\n2.) -0.0003: For diversity I added into my stacker LR, GBT and NN models that worked exclusively on just numerical features, on just categorical features, on just text features and on just images. Individually these models scored poorly, but they helped the stack.
\n3.) -0.0002: Non-linear stacking with XGBoost
\n4.) -0.0002: Add in categorical features and price into the stacker, which turn out to be helpful segmenting the predictions.
\n5.) -0.0001: Using CatBoost as non-linear stacker! CatBoost didn't work well for me as a base model, but it's target encoding of categoricals helped with stacking.
\n6.) -0.0001: Stacker of stackers! I stacked slight variations of my best XGBoost and CatBoost stackers.
Thank you to Kaggle & Avito for a fun competition, and fellow Kagglers for pushing me to squeeze out those last -0.0001 RMSE's!
", "messages": [ { "id": "349222", "postDate": "06/28/2018 00:09:07", "content": "I will not bore everyone with a complete overview of my 35th place solo solution. I can't wait to read over the actual top places solutions, especially Little Boat's who I have learned so much from!! Instead, I will go over how I squeezed the last -0.0011 out of my RMSE score, -0.0001 to -0.0003 at a time. It took a lot of experimenting to find what worked.
\n\nFirst a quick overview of my basic approach. I did a lot of feature engineering ending up with 170 numerical features derived from text properties, image properties, mining of test_active/train_active, price encodings, categorical stats, polynomial features, etc.
\n\nI had a team of GBT models (XGBoost and LightGBM) and neural net models (Keras). My word embedding was created entirely from Avito text. My GBT's included Tfidf and Countv on text, as well as TfidfEmbeddingVectorizer features. My NN's were BiLSTM/Conv1D and Conv1D/Conv1D similar to those in my DonorsChoose.org playground competition 1st place solution.
\n\nAt first I linearly stacked these models using out-of-fold predictions. I was at something like RMSE 0.2178 on the PLB, and searching for how to further reduce my RMSE. What worked:
\n\n1.) -0.0002: I did an DataFrame corr() on all of my numerical features, and pruned out features with correlations above 0.92.
\n2.) -0.0003: For diversity I added into my stacker LR, GBT and NN models that worked exclusively on just numerical features, on just categorical features, on just text features and on just images. Individually these models scored poorly, but they helped the stack.
\n3.) -0.0002: Non-linear stacking with XGBoost
\n4.) -0.0002: Add in categorical features and price into the stacker, which turn out to be helpful segmenting the predictions.
\n5.) -0.0001: Using CatBoost as non-linear stacker! CatBoost didn't work well for me as a base model, but it's target encoding of categoricals helped with stacking.
\n6.) -0.0001: Stacker of stackers! I stacked slight variations of my best XGBoost and CatBoost stackers.
Thank you to Kaggle & Avito for a fun competition, and fellow Kagglers for pushing me to squeeze out those last -0.0001 RMSE's!
", "rawMarkdown": "I will not bore everyone with a complete overview of my 35th place solo solution. I can't wait to read over the actual top places solutions, especially [Little Boat][1]'s who I have learned so much from!! Instead, I will go over how I squeezed the last -0.0011 out of my RMSE score, -0.0001 to -0.0003 at a time. It took a lot of experimenting to find what worked.\n\nFirst a quick overview of my basic approach. I did a lot of feature engineering ending up with 170 numerical features derived from text properties, image properties, mining of test_active/train_active, price encodings, categorical stats, polynomial features, etc. \n\nI had a team of GBT models (XGBoost and LightGBM) and neural net models (Keras). My word embedding was created entirely from Avito text. My GBT's included Tfidf and Countv on text, as well as [TfidfEmbeddingVectorizer][2] features. My NN's were BiLSTM/Conv1D and Conv1D/Conv1D similar to those in my DonorsChoose.org playground competition [1st place solution][3]. \n\nAt first I linearly stacked these models using out-of-fold predictions. I was at something like RMSE 0.2178 on the PLB, and searching for how to further reduce my RMSE. What worked:\n\n1.) -0.0002: I did an DataFrame corr() on all of my numerical features, and pruned out features with correlations above 0.92.Thanks @Harlen for sharing and congratulations on a strong finish.
", "rawMarkdown": "Thanks @Harlen for sharing and congratulations on a strong finish.", "votes": null }, { "id": "349237", "postDate": "06/28/2018 00:31:00", "content": "Thanks @Harlen
\n\nLovely to see you compete here after the donorsChoose competition and looking forward to seeing you in other competitions.
\n\nHappy Kaggling.
", "rawMarkdown": "Thanks @Harlen\n\nLovely to see you compete here after the donorsChoose competition and looking forward to seeing you in other competitions.\n\nHappy Kaggling.", "votes": null }, { "id": "349248", "postDate": "06/28/2018 00:47:10", "content": "Very step reduce rsme is very detail. Thanks sharing your excellent forward technology.
", "rawMarkdown": "Very step reduce rsme is very detail. Thanks sharing your excellent forward technology.", "votes": null }, { "id": "349255", "postDate": "06/28/2018 00:59:12", "content": "Thanks for sharing.
", "rawMarkdown": "Thanks for sharing.", "votes": null }, { "id": "349258", "postDate": "06/28/2018 01:02:40", "content": "Congratulations! I need to learn many of the techniques you've outlined here.
", "rawMarkdown": "Congratulations! I need to learn many of the techniques you've outlined here.", "votes": null }, { "id": "349264", "postDate": "06/28/2018 01:11:37", "content": "Thanks Harlan. You did very impressively for a soloist and I think a lot of us (including our team) eagerly followed https://www.kaggle.com/shadowwarrior/1st-place-solution when plotting our plan of attack for this competition.
", "rawMarkdown": "Thanks Harlan. You did very impressively for a soloist and I think a lot of us (including our team) eagerly followed https://www.kaggle.com/shadowwarrior/1st-place-solution when plotting our plan of attack for this competition.", "votes": null }, { "id": "349270", "postDate": "06/28/2018 01:18:55", "content": "Much appreciated and congrats on 14th place, Peter!
", "rawMarkdown": "Much appreciated and congrats on 14th place, Peter!", "votes": null }, { "id": "349272", "postDate": "06/28/2018 01:20:28", "content": "Thanks! Stacking was really powerful in this competition.
", "rawMarkdown": "Thanks! Stacking was really powerful in this competition.", "votes": null }, { "id": "349274", "postDate": "06/28/2018 01:21:26", "content": "Thanks! It's a true bummer to be one short of a gold medal, but I guess someone has to be there. :) We did way better and pushed way farther than I ever expected, but the rest of the competition was very fierce.
", "rawMarkdown": "Thanks! It's a true bummer to be one short of a gold medal, but I guess someone has to be there. :) We did way better and pushed way farther than I ever expected, but the rest of the competition was very fierce.", "votes": null }, { "id": "349275", "postDate": "06/28/2018 01:22:41", "content": "I used the category hashing you introduced in DonorsChoose.org in my NN! It works better for me than entity embedding.
", "rawMarkdown": "I used the category hashing you introduced in [DonorsChoose.org][1] in my NN! It works better for me than entity embedding.\n\n [1]: https://www.kaggle.com/c/donorschoose-application-screening", "votes": null }, { "id": "349277", "postDate": "06/28/2018 01:23:23", "content": "Thanks DUO. I enjoyed reading your many contributions in this competition.
", "rawMarkdown": "Thanks DUO. I enjoyed reading your many contributions in this competition.", "votes": null }, { "id": "349279", "postDate": "06/28/2018 01:25:08", "content": "Thanks Ahmed. I thought ml-team was destined for gold. You came very close.
", "rawMarkdown": "Thanks Ahmed. I thought ml-team was destined for gold. You came very close.", "votes": null }, { "id": "349281", "postDate": "06/28/2018 01:26:13", "content": "Thanks YaGana!
", "rawMarkdown": "Thanks YaGana!", "votes": null }, { "id": "349288", "postDate": "06/28/2018 01:44:03", "content": "Thanks.
\n\nI really wanted to have fun and learn in the competition and thanks to everyone, i did.
", "rawMarkdown": "Thanks. \n\nI really wanted to have fun and learn in the competition and thanks to everyone, i did.", "votes": null }, { "id": "349289", "postDate": "06/28/2018 01:44:39", "content": "Congrats Harlan and thanks for sharing !
\n\nThis competition was hard and has probably set a record of the least solo competitors in the top 50.
\n\nYou did a really impressive work.
", "rawMarkdown": "Congrats Harlan and thanks for sharing ! \n\nThis competition was hard and has probably set a record of the least solo competitors in the top 50.\n\nYou did a really impressive work.", "votes": null }, { "id": "349319", "postDate": "06/28/2018 02:26:49", "content": "Thank you, Serigne, and congrats to you for continuing your string of top finishes!
", "rawMarkdown": "Thank you, Serigne, and congrats to you for continuing your string of top finishes!", "votes": null }, { "id": "349343", "postDate": "06/28/2018 02:44:57", "content": "Congrats Harlan....
", "rawMarkdown": "Congrats Harlan....", "votes": null }, { "id": "349691", "postDate": "06/28/2018 13:44:05", "content": "I struggled a lot trying to reduce my RMSE score. With this post I have now learnt that the improvement in score will not be drastic. It is slow and cumulative.
\n\nCheers Harlan!
", "rawMarkdown": "I struggled a lot trying to reduce my RMSE score. With this post I have now learnt that the improvement in score will not be drastic. It is slow and cumulative.\n\nCheers Harlan!", "votes": null }, { "id": "349695", "postDate": "06/28/2018 13:56:46", "content": "@Abdul: Slow, but educational. Always cool when something unexpected works, like for me CatBoost vs XGBoost as a stacker.
", "rawMarkdown": "Abdul: Slow, but educational. Always cool when something unexpected works, like for me CatBoost vs XGBoost as a stacker.", "votes": null }, { "id": "349698", "postDate": "06/28/2018 14:03:30", "content": "Sure Harlan. :)
\n\n+1 to the list of items I have learnt from this competition.
", "rawMarkdown": "Sure Harlan. :) \n\n+1 to the list of items I have learnt from this competition.", "votes": null }, { "id": "349766", "postDate": "06/28/2018 15:57:40", "content": "Congrats Harlan. I was eagerly looking for your solutions!!!
", "rawMarkdown": "Congrats Harlan. I was eagerly looking for your solutions!!!", "votes": null } ], "comments": [ { "id": 349232, "author_name": "sheriytm", "author_url": "", "post_date": "06/28/2018 00:23:15", "content": "Thanks @Harlen for sharing and congratulations on a strong finish.
", "votes": null, "replies": [ { "id": 349281, "author_name": "shadowwarrior", "author_url": "", "post_date": "06/28/2018 01:26:13", "content": "Thanks YaGana!
", "votes": null, "replies": [] } ] }, { "id": 349237, "author_name": "ahmedalesh", "author_url": "", "post_date": "06/28/2018 00:31:00", "content": "Thanks @Harlen
\n\nLovely to see you compete here after the donorsChoose competition and looking forward to seeing you in other competitions.
\n\nHappy Kaggling.
", "votes": null, "replies": [ { "id": 349279, "author_name": "shadowwarrior", "author_url": "", "post_date": "06/28/2018 01:25:08", "content": "Thanks Ahmed. I thought ml-team was destined for gold. You came very close.
", "votes": null, "replies": [] }, { "id": 349288, "author_name": "ahmedalesh", "author_url": "", "post_date": "06/28/2018 01:44:03", "content": "Thanks.
\n\nI really wanted to have fun and learn in the competition and thanks to everyone, i did.
", "votes": null, "replies": [] } ] }, { "id": 349248, "author_name": "classtag", "author_url": "", "post_date": "06/28/2018 00:47:10", "content": "Very step reduce rsme is very detail. Thanks sharing your excellent forward technology.
", "votes": null, "replies": [ { "id": 349277, "author_name": "shadowwarrior", "author_url": "", "post_date": "06/28/2018 01:23:23", "content": "Thanks DUO. I enjoyed reading your many contributions in this competition.
", "votes": null, "replies": [] } ] }, { "id": 349255, "author_name": "qinhui1999", "author_url": "", "post_date": "06/28/2018 00:59:12", "content": "Thanks for sharing.
", "votes": null, "replies": [ { "id": 349275, "author_name": "shadowwarrior", "author_url": "", "post_date": "06/28/2018 01:22:41", "content": "I used the category hashing you introduced in DonorsChoose.org in my NN! It works better for me than entity embedding.
", "votes": null, "replies": [] } ] }, { "id": 349258, "author_name": "eigenvector", "author_url": "", "post_date": "06/28/2018 01:02:40", "content": "Congratulations! I need to learn many of the techniques you've outlined here.
", "votes": null, "replies": [ { "id": 349272, "author_name": "shadowwarrior", "author_url": "", "post_date": "06/28/2018 01:20:28", "content": "Thanks! Stacking was really powerful in this competition.
", "votes": null, "replies": [] } ] }, { "id": 349264, "author_name": "peterhurford", "author_url": "", "post_date": "06/28/2018 01:11:37", "content": "Thanks Harlan. You did very impressively for a soloist and I think a lot of us (including our team) eagerly followed https://www.kaggle.com/shadowwarrior/1st-place-solution when plotting our plan of attack for this competition.
", "votes": null, "replies": [ { "id": 349270, "author_name": "shadowwarrior", "author_url": "", "post_date": "06/28/2018 01:18:55", "content": "Much appreciated and congrats on 14th place, Peter!
", "votes": null, "replies": [] }, { "id": 349274, "author_name": "peterhurford", "author_url": "", "post_date": "06/28/2018 01:21:26", "content": "Thanks! It's a true bummer to be one short of a gold medal, but I guess someone has to be there. :) We did way better and pushed way farther than I ever expected, but the rest of the competition was very fierce.
", "votes": null, "replies": [] } ] }, { "id": 349289, "author_name": "serigne", "author_url": "", "post_date": "06/28/2018 01:44:39", "content": "Congrats Harlan and thanks for sharing !
\n\nThis competition was hard and has probably set a record of the least solo competitors in the top 50.
\n\nYou did a really impressive work.
", "votes": null, "replies": [ { "id": 349319, "author_name": "shadowwarrior", "author_url": "", "post_date": "06/28/2018 02:26:49", "content": "Thank you, Serigne, and congrats to you for continuing your string of top finishes!
", "votes": null, "replies": [] } ] }, { "id": 349343, "author_name": "samratp", "author_url": "", "post_date": "06/28/2018 02:44:57", "content": "Congrats Harlan....
", "votes": null, "replies": [] }, { "id": 349691, "author_name": "abdul0807", "author_url": "", "post_date": "06/28/2018 13:44:05", "content": "I struggled a lot trying to reduce my RMSE score. With this post I have now learnt that the improvement in score will not be drastic. It is slow and cumulative.
\n\nCheers Harlan!
", "votes": null, "replies": [ { "id": 349695, "author_name": "shadowwarrior", "author_url": "", "post_date": "06/28/2018 13:56:46", "content": "@Abdul: Slow, but educational. Always cool when something unexpected works, like for me CatBoost vs XGBoost as a stacker.
", "votes": null, "replies": [] }, { "id": 349698, "author_name": "abdul0807", "author_url": "", "post_date": "06/28/2018 14:03:30", "content": "Sure Harlan. :)
\n\n+1 to the list of items I have learnt from this competition.
", "votes": null, "replies": [] } ] }, { "id": 349766, "author_name": "subikashpal", "author_url": "", "post_date": "06/28/2018 15:57:40", "content": "Congrats Harlan. I was eagerly looking for your solutions!!!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "349222": "I will not bore everyone with a complete overview of my 35th place solo solution. I can't wait to read over the actual top places solutions, especially [Little Boat][1]'s who I have learned so much from!! Instead, I will go over how I squeezed the last -0.0011 out of my RMSE score, -0.0001 to -0.0003 at a time. It took a lot of experimenting to find what worked.\n\nFirst a quick overview of my basic approach. I did a lot of feature engineering ending up with 170 numerical features derived from text properties, image properties, mining of test_active/train_active, price encodings, categorical stats, polynomial features, etc. \n\nI had a team of GBT models (XGBoost and LightGBM) and neural net models (Keras). My word embedding was created entirely from Avito text. My GBT's included Tfidf and Countv on text, as well as [TfidfEmbeddingVectorizer][2] features. My NN's were BiLSTM/Conv1D and Conv1D/Conv1D similar to those in my DonorsChoose.org playground competition [1st place solution][3]. \n\nAt first I linearly stacked these models using out-of-fold predictions. I was at something like RMSE 0.2178 on the PLB, and searching for how to further reduce my RMSE. What worked:\n\n1.) -0.0002: I did an DataFrame corr() on all of my numerical features, and pruned out features with correlations above 0.92.