First of all, thanks to Avito and Kaggle for providing this great competition. No leakage, very representative testing set, countless solutions. I believe every one enjoyed the game.
\n\nBesides, thanks to all community contributors, like Dieter and Peter Hurford, who published helpful kernels and discussions.
\n\nFeatures:\nMost of the ideas came from public kernels and discussions: tf-idf, CountVectorizor (char level), image meta features, self-trained W2V, ads time from Benjamin Minixhofer's aggregated features.
\n\nWhat I did additionally is clustering ads titles based on sub-categories. For example:
\n\nsub_category = df[(df['category_name'] == 'something') & (df['param_1'] == 'something') & (df['param_2'] == 'something') & (df['param_3'] == 'something')]\ntfidf = TfidfVectorizor()\ntfidf_vec = tfidf.fit_transform(sub_category['title'])\nkmean_cluster = KMmean()\nsub_category['title_cluster'] = kmean_cluster.fit_predict(tfidf_vec)\nI did such calculation for all sub-catgories. After doing this, in the subcategory of iphone, all iphone 5s 32gb were in one group and iphone 7 128 GB were in another group. Then it was a good time to compare the prices and other features. So I groupby this 'title_cluster' and calculated aggregated features like price rank (which gave a big improvement), mean price, count and so on.
\n\nI also used regex to manually extract numbers out of the title for properties (area, room numbers, floor) and automobile (vehicle years) ads. Then calculated features such as price per room, price per sqm.
\n\nOther than that, since traditional ML models such as LightGBM are not good at handling unstructured data like text and images, I used NN (biLSTM for texts and a simple 4 layers CNN for image) to generate vector representations. Image vectors directly came from a NN that were used to predict the deal_probability. But when I tried the same strategy for texts, the generated text vector dramatically made my LightGBM overfitted. Then I tried another method: only using title and description W2V in a biLSTM NN model with MSE loss and predicting everything else: price, item_seq_number, city, region, user_id, parent_category_name, category_name, param_1, param_2, param_3. All categorical features are onehot encoded after removing low frequent entities. Therefore, X = title W2V + description W2V. y = a few hundred columns table.
\n\nModels:\nThree layer of stacking. first layer: 11 Lightgbms, 6NNs. second layer: 3 LightGBM, 2 Ridge. third layer is just a Ridge and a LighGBM with linear average. I am a newbie of NN (my oof NN scores were very unstable) and hope to learn more NN strategies from other teams.
\n\nThanks
", "messages": [ { "id": "349311", "postDate": "06/28/2018 02:17:21", "content": "First of all, thanks to Avito and Kaggle for providing this great competition. No leakage, very representative testing set, countless solutions. I believe every one enjoyed the game.
\n\nBesides, thanks to all community contributors, like Dieter and Peter Hurford, who published helpful kernels and discussions.
\n\nFeatures:\nMost of the ideas came from public kernels and discussions: tf-idf, CountVectorizor (char level), image meta features, self-trained W2V, ads time from Benjamin Minixhofer's aggregated features.
\n\nWhat I did additionally is clustering ads titles based on sub-categories. For example:
\n\nsub_category = df[(df['category_name'] == 'something') & (df['param_1'] == 'something') & (df['param_2'] == 'something') & (df['param_3'] == 'something')]\ntfidf = TfidfVectorizor()\ntfidf_vec = tfidf.fit_transform(sub_category['title'])\nkmean_cluster = KMmean()\nsub_category['title_cluster'] = kmean_cluster.fit_predict(tfidf_vec)\nI did such calculation for all sub-catgories. After doing this, in the subcategory of iphone, all iphone 5s 32gb were in one group and iphone 7 128 GB were in another group. Then it was a good time to compare the prices and other features. So I groupby this 'title_cluster' and calculated aggregated features like price rank (which gave a big improvement), mean price, count and so on.
\n\nI also used regex to manually extract numbers out of the title for properties (area, room numbers, floor) and automobile (vehicle years) ads. Then calculated features such as price per room, price per sqm.
\n\nOther than that, since traditional ML models such as LightGBM are not good at handling unstructured data like text and images, I used NN (biLSTM for texts and a simple 4 layers CNN for image) to generate vector representations. Image vectors directly came from a NN that were used to predict the deal_probability. But when I tried the same strategy for texts, the generated text vector dramatically made my LightGBM overfitted. Then I tried another method: only using title and description W2V in a biLSTM NN model with MSE loss and predicting everything else: price, item_seq_number, city, region, user_id, parent_category_name, category_name, param_1, param_2, param_3. All categorical features are onehot encoded after removing low frequent entities. Therefore, X = title W2V + description W2V. y = a few hundred columns table.
\n\nModels:\nThree layer of stacking. first layer: 11 Lightgbms, 6NNs. second layer: 3 LightGBM, 2 Ridge. third layer is just a Ridge and a LighGBM with linear average. I am a newbie of NN (my oof NN scores were very unstable) and hope to learn more NN strategies from other teams.
\n\nThanks
", "rawMarkdown": "First of all, thanks to Avito and Kaggle for providing this great competition. No leakage, very representative testing set, countless solutions. I believe every one enjoyed the game.\n\nBesides, thanks to all community contributors, like Dieter and Peter Hurford, who published helpful kernels and discussions.\n\nFeatures:\nMost of the ideas came from public kernels and discussions: tf-idf, CountVectorizor (char level), image meta features, self-trained W2V, ads time from Benjamin Minixhofer's aggregated features.\n\nWhat I did additionally is clustering ads titles based on sub-categories. For example:\n\n sub_category = df[(df['category_name'] == 'something') & (df['param_1'] == 'something') & (df['param_2'] == 'something') & (df['param_3'] == 'something')]\n tfidf = TfidfVectorizor()\n tfidf_vec = tfidf.fit_transform(sub_category['title'])\n kmean_cluster = KMmean()\n sub_category['title_cluster'] = kmean_cluster.fit_predict(tfidf_vec)\n\nI did such calculation for all sub-catgories. After doing this, in the subcategory of iphone, all iphone 5s 32gb were in one group and iphone 7 128 GB were in another group. Then it was a good time to compare the prices and other features. So I groupby this 'title_cluster' and calculated aggregated features like price rank (which gave a big improvement), mean price, count and so on.\n\nI also used regex to manually extract numbers out of the title for properties (area, room numbers, floor) and automobile (vehicle years) ads. Then calculated features such as price per room, price per sqm.\n\nOther than that, since traditional ML models such as LightGBM are not good at handling unstructured data like text and images, I used NN (biLSTM for texts and a simple 4 layers CNN for image) to generate vector representations. Image vectors directly came from a NN that were used to predict the deal_probability. But when I tried the same strategy for texts, the generated text vector dramatically made my LightGBM overfitted. Then I tried another method: only using title and description W2V in a biLSTM NN model with MSE loss and predicting everything else: price, item_seq_number, city, region, user_id, parent_category_name, category_name, param_1, param_2, param_3. All categorical features are onehot encoded after removing low frequent entities. Therefore, X = title W2V + description W2V. y = a few hundred columns table.\n\nModels:\nThree layer of stacking. first layer: 11 Lightgbms, 6NNs. second layer: 3 LightGBM, 2 Ridge. third layer is just a Ridge and a LighGBM with linear average. I am a newbie of NN (my oof NN scores were very unstable) and hope to learn more NN strategies from other teams.\n\nThanks", "votes": null }, { "id": "349315", "postDate": "06/28/2018 02:23:54", "content": "Congratulations for winning a solo gold medal in this competition!!!
", "rawMarkdown": "Congratulations for winning a solo gold medal in this competition!!!", "votes": null }, { "id": "349320", "postDate": "06/28/2018 02:27:23", "content": "Congratulations for solo gold medal ! Your team name is interesting at the last week of competition : )
", "rawMarkdown": "Congratulations for solo gold medal ! Your team name is interesting at the last week of competition : )", "votes": null }, { "id": "349324", "postDate": "06/28/2018 02:29:12", "content": "Bummer for us to lose out on gold, but I'm really glad you were able to solo gold. That impressive accomplishment makes me feel better about our loss. I was secretly rooting for you throughout your quest up the LB.
", "rawMarkdown": "Bummer for us to lose out on gold, but I'm really glad you were able to solo gold. That impressive accomplishment makes me feel better about our loss. I was secretly rooting for you throughout your quest up the LB.", "votes": null }, { "id": "349325", "postDate": "06/28/2018 02:29:17", "content": "Congrats for your Gold... So, finally you showed everyone that solo gold is not very difficult :-)
", "rawMarkdown": "Congrats for your Gold... So, finally you showed everyone that solo gold is not very difficult :-)", "votes": null }, { "id": "349333", "postDate": "06/28/2018 02:36:12", "content": "Congrats for your impressive solo gold ! Your Grandma will be proud :)
", "rawMarkdown": "Congrats for your impressive solo gold ! Your Grandma will be proud :)", "votes": null }, { "id": "349337", "postDate": "06/28/2018 02:38:52", "content": "Congratulations for the solo gold medal!
", "rawMarkdown": "Congratulations for the solo gold medal!", "votes": null }, { "id": "349365", "postDate": "06/28/2018 03:18:34", "content": "Ayyy! Congrats on the solo gold. Truly an impressive feat 🍻
", "rawMarkdown": "Ayyy! Congrats on the solo gold. Truly an impressive feat 🍻", "votes": null }, { "id": "349422", "postDate": "06/28/2018 04:59:33", "content": "Congratulations @Weber on an amazing solo gold. Your solution is inspiring as well.
", "rawMarkdown": "Congratulations @Weber on an amazing solo gold. Your solution is inspiring as well.", "votes": null }, { "id": "349438", "postDate": "06/28/2018 05:46:01", "content": "Congratulations for your solo gold medal. Great job!
", "rawMarkdown": "Congratulations for your solo gold medal. Great job!", "votes": null }, { "id": "349471", "postDate": "06/28/2018 06:56:31", "content": "Congratulations!
", "rawMarkdown": "Congratulations!", "votes": null }, { "id": "349475", "postDate": "06/28/2018 07:03:23", "content": "\"Win a gold for my grandma\", congratz. Hands down, it's the best name among all teams.\nYour nanny must be so happy.
", "rawMarkdown": "\"Win a gold for my grandma\", congratz. Hands down, it's the best name among all teams.\nYour nanny must be so happy.", "votes": null }, { "id": "349842", "postDate": "06/28/2018 18:07:57", "content": "Thanks! It's effort and luck making it easy.
", "rawMarkdown": "Thanks! It's effort and luck making it easy.", "votes": null }, { "id": "349843", "postDate": "06/28/2018 18:08:34", "content": "Haha, thanks. Just had something fun in the game
", "rawMarkdown": "Haha, thanks. Just had something fun in the game", "votes": null }, { "id": "349844", "postDate": "06/28/2018 18:10:11", "content": "Your team also did a great job. I was just a little bit more lucky at the end. You will be the lucky guy next time I believe.
", "rawMarkdown": "Your team also did a great job. I was just a little bit more lucky at the end. You will be the lucky guy next time I believe.", "votes": null }, { "id": "349937", "postDate": "06/28/2018 22:14:22", "content": "@Peter, what were the image feature you used - I think you mentioned earlier you got a .001 or .002 uplift from them ? Well done on your position, would love to see a write up
", "rawMarkdown": "Peter, what were the image feature you used - I think you mentioned earlier you got a .001 or .002 uplift from them ? Well done on your position, would love to see a write up", "votes": null }, { "id": "349938", "postDate": "06/28/2018 22:31:54", "content": "@Darragh, yeah the final boost from images was ~0.0015 on LightGBM. They also seemed to help our NNs significantly too. We're working on a write-up -- we're just not as fast as everyone else, haha.
", "rawMarkdown": "Darragh, yeah the final boost from images was ~0.0015 on LightGBM. They also seemed to help our NNs significantly too. We're working on a write-up -- we're just not as fast as everyone else, haha.", "votes": null }, { "id": "349940", "postDate": "06/28/2018 22:36:36", "content": "Take your time and make it good :) looking forward to it, good job!
", "rawMarkdown": "Take your time and make it good :) looking forward to it, good job!", "votes": null }, { "id": "349985", "postDate": "06/29/2018 01:30:08", "content": "I'm so glad you got a solo gold that was \"so hard to get\"! :)
", "rawMarkdown": "I'm so glad you got a solo gold that was \"so hard to get\"! :)", "votes": null }, { "id": "421546", "postDate": "11/15/2018 05:37:57", "content": "学长好强！！学习学习！
", "rawMarkdown": "学长好强！！学习学习！", "votes": null } ], "comments": [ { "id": 349315, "author_name": "cczaixian", "author_url": "", "post_date": "06/28/2018 02:23:54", "content": "Congratulations for winning a solo gold medal in this competition!!!
", "votes": null, "replies": [] }, { "id": 349320, "author_name": "mzr2017", "author_url": "", "post_date": "06/28/2018 02:27:23", "content": "Congratulations for solo gold medal ! Your team name is interesting at the last week of competition : )
", "votes": null, "replies": [ { "id": 349843, "author_name": "wenbozhao", "author_url": "", "post_date": "06/28/2018 18:08:34", "content": "Haha, thanks. Just had something fun in the game
", "votes": null, "replies": [] } ] }, { "id": 349324, "author_name": "peterhurford", "author_url": "", "post_date": "06/28/2018 02:29:12", "content": "Bummer for us to lose out on gold, but I'm really glad you were able to solo gold. That impressive accomplishment makes me feel better about our loss. I was secretly rooting for you throughout your quest up the LB.
", "votes": null, "replies": [ { "id": 349844, "author_name": "wenbozhao", "author_url": "", "post_date": "06/28/2018 18:10:11", "content": "Your team also did a great job. I was just a little bit more lucky at the end. You will be the lucky guy next time I believe.
", "votes": null, "replies": [] }, { "id": 349937, "author_name": "darraghdog", "author_url": "", "post_date": "06/28/2018 22:14:22", "content": "@Peter, what were the image feature you used - I think you mentioned earlier you got a .001 or .002 uplift from them ? Well done on your position, would love to see a write up
", "votes": null, "replies": [] }, { "id": 349938, "author_name": "peterhurford", "author_url": "", "post_date": "06/28/2018 22:31:54", "content": "@Darragh, yeah the final boost from images was ~0.0015 on LightGBM. They also seemed to help our NNs significantly too. We're working on a write-up -- we're just not as fast as everyone else, haha.
", "votes": null, "replies": [] }, { "id": 349940, "author_name": "darraghdog", "author_url": "", "post_date": "06/28/2018 22:36:36", "content": "Take your time and make it good :) looking forward to it, good job!
", "votes": null, "replies": [] } ] }, { "id": 349325, "author_name": "samratp", "author_url": "", "post_date": "06/28/2018 02:29:17", "content": "Congrats for your Gold... So, finally you showed everyone that solo gold is not very difficult :-)
", "votes": null, "replies": [ { "id": 349842, "author_name": "wenbozhao", "author_url": "", "post_date": "06/28/2018 18:07:57", "content": "Thanks! It's effort and luck making it easy.
", "votes": null, "replies": [] } ] }, { "id": 349333, "author_name": "serigne", "author_url": "", "post_date": "06/28/2018 02:36:12", "content": "Congrats for your impressive solo gold ! Your Grandma will be proud :)
", "votes": null, "replies": [] }, { "id": 349337, "author_name": "naivelamb", "author_url": "", "post_date": "06/28/2018 02:38:52", "content": "Congratulations for the solo gold medal!
", "votes": null, "replies": [] }, { "id": 349365, "author_name": "yk1598", "author_url": "", "post_date": "06/28/2018 03:18:34", "content": "Ayyy! Congrats on the solo gold. Truly an impressive feat 🍻
", "votes": null, "replies": [] }, { "id": 349422, "author_name": "sheriytm", "author_url": "", "post_date": "06/28/2018 04:59:33", "content": "Congratulations @Weber on an amazing solo gold. Your solution is inspiring as well.
", "votes": null, "replies": [] }, { "id": 349438, "author_name": "qinhui1999", "author_url": "", "post_date": "06/28/2018 05:46:01", "content": "Congratulations for your solo gold medal. Great job!
", "votes": null, "replies": [] }, { "id": 349471, "author_name": "greener", "author_url": "", "post_date": "06/28/2018 06:56:31", "content": "Congratulations!
", "votes": null, "replies": [] }, { "id": 349475, "author_name": "tmhung", "author_url": "", "post_date": "06/28/2018 07:03:23", "content": "\"Win a gold for my grandma\", congratz. Hands down, it's the best name among all teams.\nYour nanny must be so happy.
", "votes": null, "replies": [] }, { "id": 349985, "author_name": "u39kun", "author_url": "", "post_date": "06/29/2018 01:30:08", "content": "I'm so glad you got a solo gold that was \"so hard to get\"! :)
", "votes": null, "replies": [] }, { "id": 421546, "author_name": "louishgy", "author_url": "", "post_date": "11/15/2018 05:37:57", "content": "学长好强！！学习学习！
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "349311": "First of all, thanks to Avito and Kaggle for providing this great competition. No leakage, very representative testing set, countless solutions. I believe every one enjoyed the game.\n\nBesides, thanks to all community contributors, like Dieter and Peter Hurford, who published helpful kernels and discussions.\n\nFeatures:\nMost of the ideas came from public kernels and discussions: tf-idf, CountVectorizor (char level), image meta features, self-trained W2V, ads time from Benjamin Minixhofer's aggregated features.\n\nWhat I did additionally is clustering ads titles based on sub-categories. For example:\n\n sub_category = df[(df['category_name'] == 'something') & (df['param_1'] == 'something') & (df['param_2'] == 'something') & (df['param_3'] == 'something')]\n tfidf = TfidfVectorizor()\n tfidf_vec = tfidf.fit_transform(sub_category['title'])\n kmean_cluster = KMmean()\n sub_category['title_cluster'] = kmean_cluster.fit_predict(tfidf_vec)\n\nI did such calculation for all sub-catgories. After doing this, in the subcategory of iphone, all iphone 5s 32gb were in one group and iphone 7 128 GB were in another group. Then it was a good time to compare the prices and other features. So I groupby this 'title_cluster' and calculated aggregated features like price rank (which gave a big improvement), mean price, count and so on.\n\nI also used regex to manually extract numbers out of the title for properties (area, room numbers, floor) and automobile (vehicle years) ads. Then calculated features such as price per room, price per sqm.\n\nOther than that, since traditional ML models such as LightGBM are not good at handling unstructured data like text and images, I used NN (biLSTM for texts and a simple 4 layers CNN for image) to generate vector representations. Image vectors directly came from a NN that were used to predict the deal_probability. But when I tried the same strategy for texts, the generated text vector dramatically made my LightGBM overfitted. Then I tried another method: only using title and description W2V in a biLSTM NN model with MSE loss and predicting everything else: price, item_seq_number, city, region, user_id, parent_category_name, category_name, param_1, param_2, param_3. All categorical features are onehot encoded after removing low frequent entities. Therefore, X = title W2V + description W2V. y = a few hundred columns table.\n\nModels:\nThree layer of stacking. first layer: 11 Lightgbms, 6NNs. second layer: 3 LightGBM, 2 Ridge. third layer is just a Ridge and a LighGBM with linear average. I am a newbie of NN (my oof NN scores were very unstable) and hope to learn more NN strategies from other teams.\n\nThanks", "349315": "Congratulations for winning a solo gold medal in this competition!!!", "349320": "Congratulations for solo gold medal ! Your team name is interesting at the last week of competition : )", "349324": "Bummer for us to lose out on gold, but I'm really glad you were able to solo gold. That impressive accomplishment makes me feel better about our loss. I was secretly rooting for you throughout your quest up the LB.", "349325": "Congrats for your Gold... So, finally you showed everyone that solo gold is not very difficult :-)", "349333": "Congrats for your impressive solo gold ! Your Grandma will be proud :)", "349337": "Congratulations for the solo gold medal!", "349365": "Ayyy! Congrats on the solo gold. Truly an impressive feat 🍻", "349422": "Congratulations @Weber on an amazing solo gold. Your solution is inspiring as well.", "349438": "Congratulations for your solo gold medal. Great job!", "349471": "Congratulations!", "349475": "\"Win a gold for my grandma\", congratz. Hands down, it's the best name among all teams.\nYour nanny must be so happy.", "349842": "Thanks! It's effort and luck making it easy.", "349843": "Haha, thanks. Just had something fun in the game", "349844": "Your team also did a great job. I was just a little bit more lucky at the end. You will be the lucky guy next time I believe.", "349937": "Peter, what were the image feature you used - I think you mentioned earlier you got a .001 or .002 uplift from them ? Well done on your position, would love to see a write up", "349938": "Darragh, yeah the final boost from images was ~0.0015 on LightGBM. They also seemed to help our NNs significantly too. We're working on a write-up -- we're just not as fast as everyone else, haha.", "349940": "Take your time and make it good :) looking forward to it, good job!", "349985": "I'm so glad you got a solo gold that was \"so hard to get\"! :)", "421546": "学长好强！！学习学习！" }, "source": "meta" }